Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35790

Rebalance failed due to not_all_nodes_are_ready_yet

    XMLWordPrintable

Details

    Description

      Build: 6.5.0-4169

      Test: Durability component test 

      Cycle: 5th

      Test step:

      • Create 6 node cluster all kv
      • Create 3 buckets with replica 1,1,2
      • load data as with no majority and with majority (bucket having replica 1 and 2)
      • Rebalance in kv node
      • Rebalance out kv node
      • Swap kv nodes
      • Update replica of bucket as 2 where we load data as majority. Important point to note that default bucket is still have replica as 1 and data loading as Durability=None
      • Failover -> full recovery 
      • Failover -> delta recovery (failed)

      [2019-09-03T17:12:03-07:00, sequoiatools/couchbase-cli:6.5:f5e39c] failover -c 172.23.97.74:8091 --server-failover 172.23.96.207:8091 -u Administrator -p password
      		 
      [2019-09-03T17:23:38-07:00, sequoiatools/couchbase-cli:6.5:ccb87e] recovery -c 172.23.97.74:8091 --server-recovery 172.23.96.207:8091 --recovery-type delta -u Administrator -p password
      		 
      [2019-09-03T17:23:45-07:00, sequoiatools/couchbase-cli:6.5:475190] rebalance -c 172.23.97.74:8091 -u Administrator -p password
      		 
      → 
      		 
       
      		 
       
      		 
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.97.74:8091 -u Administrator -p password]
      		 
       
      		 
       
      		 
      docker logs 475190
      		 
      docker start 475190
      		 
       
      		 
       
      		 
      *Unable to display progress bar on this os
      		 
      JERROR: Rebalance failed. See logs for detailed reason. You can try again. 

      [user:error,2019-09-03T17:24:07.316-07:00,ns_1@172.23.97.74:<0.14096.0>:ns_orchestrator:log_rebalance_completion:1472]Rebalance exited with reason {not_all_nodes_are_ready_yet,
      		 
                                       ['ns_1@172.23.96.207']}.
      		 
      Rebalance Operation Id = 8f8d58eb407180078e340e2f4000dc76

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-35631 [Jepsen] Crash due to uncaught pre-condition failure in EPBucket::flushVBucket

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              owend Daniel Owen
              vikas.chaudhary Vikas Chaudhary
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty