Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-17853

rebalance failed due to set vbuckets failed

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 3.1.4
    • 3.1.4
    • couchbase-bucket
    • Security Level: Public
    • None
    • centos 6.5 64-bit

    Description

      Install cb server 3.1.4 on 7 nodes cluster.
      Create 3 buckets with 2 replicas on each bucket.
      Load 1 M keys to each bucket.
      Hard failover 2 nodes and rebalane. Rebalance failed with errors

      Rebalance exited with reason {unexpected_exit,
      {'EXIT',<0.16413.20>,
      {bulk_set_vbucket_state_failed,
      [{'ns_1@172.23.105.147',
      {'EXIT',
      {{badmatch,
      {error,
      {setup_replications_failed,
      [{'ns_1@172.23.107.79',
      {errors,[{2,630}]}}]}}},
      {gen_server,call,
      [{'janitor_agent-bucket-1',
      'ns_1@172.23.105.147'},
      {if_rebalance,<0.2056.20>,
      {update_vbucket_state,630,replica,
      undefined,undefined}},
      infinity]}}}}]}}}
      	ns_orchestrator002 	ns_1@172.23.105.147 	13:05:41 - Tue Feb 2, 2016
      <0.15553.20> exited with {unexpected_exit,
      {'EXIT',<0.16413.20>,
      {bulk_set_vbucket_state_failed,
      [{'ns_1@172.23.105.147',
      {'EXIT',
      {{badmatch,
      {error,
      {setup_replications_failed,
      [{'ns_1@172.23.107.79',
      {errors,[{2,630}]}}]}}},
      {gen_server,call,
      [{'janitor_agent-bucket-1',
      'ns_1@172.23.105.147'},
      {if_rebalance,<0.2056.20>,
      {update_vbucket_state,630,replica,
      undefined,undefined}},
      infinity]}}}}]}}} 	ns_vbucket_mover000 	ns_1@172.23.105.147 	13:05:41 - Tue Feb 2, 2016
      Bucket "bucket-2" loaded on node 'ns_1@172.23.107.183' in 0 seconds. 	ns_memcached000 	ns_1@172.23.107.183 	13:05:23 - Tue Feb 2, 2016
      Node 'ns_1@172.23.107.183' saw that node 'ns_1@172.23.121.183' came up. Tags: [] 	ns_node_disco004 	ns_1@172.23.107.183 	13:05:19 - Tue Feb 2, 2016
      Bucket "bucket-1" rebalance does not seem to be swap rebalance 	ns_vbucket_mover000 	ns_1@172.23.105.147 	13:05:13 - Tue Feb 2, 2016
      Node 'ns_1@172.23.107.183' saw that node 'ns_1@172.23.107.79' came up. Tags: [] 	ns_node_disco004 	ns_1@172.23.107.183 	
      

      In other case, hard failover and rebalance out 2 nodes passed. Then add those 2 nodes back and rebalance. Rebalance failed with same error.

      Attachments

        Issue Links

          Activity

            People

              poonam Poonam Dhavale
              thuan Thuan Nguyen
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty