Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-34713

Rebalance failed with pre_rebalance_janitor_run_failed

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 6.5.0
    • 6.5.0
    • ns_server
    • Untriaged
    • Unknown
    • KV-Engine MH Beta part 2, KV-Engine MH 2nd Beta

    Description

      build: 6.5.0-3573

      Test: longevity (https://github.com/couchbaselabs/sequoia/blob/master/tests/integration/test_allFeatures_alice_timers.yml )
      http://qa.sc.couchbase.com/job/centos-systest-launcher-2/412/consoleFull

      Rebalance failed with :

      [ns_server:error,2019-06-23T14:01:32.943-07:00,ns_1@172.23.104.113:ns_doctor<0.9960.0>:ns_doctor:update_status:316]The following buckets became not ready on node 'ns_1@172.23.96.215': ["CUSTOMER",
                                                                            "DISTRICT",
                                                                            "HISTORY",
                                                                            "ITEM",
                                                                            "NEW_ORDER",
                                                                            "ORDERS",
                                                                            "ORDER_LINE",
                                                                            "STOCK",
                                                                            "WAREHOUSE",
                                                                            "default"], those of them are active []
      [user:error,2019-06-23T14:05:20.427-07:00,ns_1@172.23.104.113:<0.5272.331>:ns_orchestrator:log_rebalance_completion:1426]Rebalance exited with reason {pre_rebalance_janitor_run_failed,"default",
                                       ['ns_1@172.23.96.215']}.
      Rebalance Operation Id = ae4db9c7282a5a43e0792f80701d366a
      

      From the test console its running:

      [2019-06-23T14:05:07-07:00, sequoiatools/couchbase-cli:6.5:e0c50e] rebalance -c 172.23.97.74:8091 -u Administrator -p password
       
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.97.74:8091 -u Administrator -p password]
       
      docker logs e0c50e
      docker start e0c50e
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      [2019-06-23T14:05:29-07:00, sequoiatools/cmd:c82cbf] 60
      

      Logs:

      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.104.113.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.104.114.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.104.148.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.104.151.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.104.68.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.104.78.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.96.14.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.96.18.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.96.183.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.96.190.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.96.191.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.96.207.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.96.209.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.96.210.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.96.212.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.96.215.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.96.216.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.96.219.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.96.220.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.96.221.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.96.223.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.96.254.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.96.48.zip
      https://s3.amazonaws.com/bugdb/jira/systestmon-1561399932/collectinfo-2019-06-24T181214-ns_1%40172.23.97.74.zip

      Attachments

        1. screenshot-1.png
          screenshot-1.png
          151 kB
        2. screenshot-2.png
          screenshot-2.png
          195 kB
        3. screenshot-3.png
          screenshot-3.png
          357 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ajit.yagaty Ajit Yagaty [X] (Inactive)
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty