Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-17853

rebalance failed due to set vbuckets failed

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 3.1.4
    • 3.1.4
    • couchbase-bucket
    • Security Level: Public
    • None
    • centos 6.5 64-bit

    Description

      Install cb server 3.1.4 on 7 nodes cluster.
      Create 3 buckets with 2 replicas on each bucket.
      Load 1 M keys to each bucket.
      Hard failover 2 nodes and rebalane. Rebalance failed with errors 

      Rebalance exited with reason {unexpected_exit,
      		 
      {'EXIT',<0.16413.20>,
      		 
      {bulk_set_vbucket_state_failed,
      		 
      [{'ns_1@172.23.105.147',
      		 
      {'EXIT',
      		 
      {{badmatch,
      		 
      {error,
      		 
      {setup_replications_failed,
      		 
      [{'ns_1@172.23.107.79',
      		 
      {errors,[{2,630}]}}]}}},
      		 
      {gen_server,call,
      		 
      [{'janitor_agent-bucket-1',
      		 
      'ns_1@172.23.105.147'},
      		 
      {if_rebalance,<0.2056.20>,
      		 
      {update_vbucket_state,630,replica,
      		 
      undefined,undefined}},
      		 
      infinity]}}}}]}}}
      		 
      	ns_orchestrator002 	ns_1@172.23.105.147 	13:05:41 - Tue Feb 2, 2016
      		 
      <0.15553.20> exited with {unexpected_exit,
      		 
      {'EXIT',<0.16413.20>,
      		 
      {bulk_set_vbucket_state_failed,
      		 
      [{'ns_1@172.23.105.147',
      		 
      {'EXIT',
      		 
      {{badmatch,
      		 
      {error,
      		 
      {setup_replications_failed,
      		 
      [{'ns_1@172.23.107.79',
      		 
      {errors,[{2,630}]}}]}}},
      		 
      {gen_server,call,
      		 
      [{'janitor_agent-bucket-1',
      		 
      'ns_1@172.23.105.147'},
      		 
      {if_rebalance,<0.2056.20>,
      		 
      {update_vbucket_state,630,replica,
      		 
      undefined,undefined}},
      		 
      infinity]}}}}]}}} 	ns_vbucket_mover000 	ns_1@172.23.105.147 	13:05:41 - Tue Feb 2, 2016
      		 
      Bucket "bucket-2" loaded on node 'ns_1@172.23.107.183' in 0 seconds. 	ns_memcached000 	ns_1@172.23.107.183 	13:05:23 - Tue Feb 2, 2016
      		 
      Node 'ns_1@172.23.107.183' saw that node 'ns_1@172.23.121.183' came up. Tags: [] 	ns_node_disco004 	ns_1@172.23.107.183 	13:05:19 - Tue Feb 2, 2016
      		 
      Bucket "bucket-1" rebalance does not seem to be swap rebalance 	ns_vbucket_mover000 	ns_1@172.23.105.147 	13:05:13 - Tue Feb 2, 2016
      		 
      Node 'ns_1@172.23.107.183' saw that node 'ns_1@172.23.107.79' came up. Tags: [] 	ns_node_disco004 	ns_1@172.23.107.183

      In other case, hard failover and rebalance out 2 nodes passed. Then add those 2 nodes back and rebalance. Rebalance failed with same error.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-17628 [system test] Rebalance failed with 'setup_replications_failed'

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              poonam Poonam Dhavale
              thuan Thuan Nguyen
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty