Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Duplicate
Priority: Blocker
Fix Version/s: 3.0
Affects Version/s: 3.0
Component/s: couchbase-bucket, ns_server
Security Level: Public
Labels:
None
Environment:
10.6.2.144-10.6.2.150

Triage:
Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump:
https://s3.amazonaws.com/bugdb/jira/MB-11975/1066.rebalance_exit.log.gz
Is this a Regression?:
Unknown

Description

1166, Centos 6x

Test Case:: /testrunner -i ubuntu_x64.ini get-cbcollect-info=False,get-logs=False,stop-on-failure=False,get-coredumps=True,force_kill_memached=False,verify_unacked_bytes=True,total_vbuckets=128,std_vbuckets_dist=5 -t failover.failovertests.FailoverTests.test_failover_then_add_back,replicas=1,num_failed_nodes=1,items=100000,withMutationOps=True,doc_ops=update,upr_check=False,recoveryType=delta,graceful=True,GROUP=P0;GRACEFUL

1. Create 7 node cluster
2. Create default bucket with 100 k items
3. Graceful failover 1 node with mutations running in parallel
4. Add-back with recovery set to delta
5. Rebalance clusters with mutations running in parallel
Starting rebalance, KeepNodes = ['ns_1@10.6.2.146','ns_1@10.6.2.144',
'ns_1@10.6.2.145','ns_1@10.6.2.147',
'ns_1@10.6.2.148','ns_1@10.6.2.150',
'ns_1@10.6.2.149'], EjectNodes = [], Failed over and being ejected nodes = [], Delta recovery nodes = ['ns_1@10.6.2.146'], Delta recovery buckets = all

Step 5 leads to Rebalance exit with bad replicators issue

Bad replicators after rebalance:
Missing = [

{'ns_1@10.6.2.147','ns_1@10.6.2.146',64}

]
Extras = []

ERROR - {u'node': u'ns_1@172.23.107.153', u'code': 2, u'text': u'Rebalance exited with reason {unexpected_exit,\n {\'EXIT\',<0.16515.3>,\n {dcp_wait_for_data_move_failed,"default",83,\n \'ns_1@172.23.107.153\',\n [\'ns_1@172.23.107.156\'],\n {error,no_stats_for_this_vbucket}}}}\n', u'shortText': u'message', u'serverTime': u'2014-08-15T19:43:54.062Z', u'module': u'ns_orchestrator', u'tstamp': 1408157034062, u'type': u'info'}
[2014-08-15 19:39:05,783] - [rest_client:2011] ERROR - {u'node': u'ns_1@172.23.107.153', u'code': 0, u'text': u'<0.16503.3> exited with {unexpected_exit,\n {\'EXIT\',<0.16515.3>,\n {dcp_wait_for_data_move_failed,"default",83,\n \'ns_1@172.23.107.153\',\n [\'ns_1@172.23.107.156\'],\n {error,no_stats_for_this_vbucket}}}}', u'shortText': u'message', u'serverTime': u'2014-08-15T19:43:54.060Z', u'module': u'ns_vbucket_mover', u'tstamp': 1408157034060, u'type': u'critical'}
[2014-08-15 19:39:05,783] - [rest_client:2011] ERROR -

{u'node': u'ns_1@172.23.107.153', u'code': 0, u'text': u'Bucket "default" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'serverTime': u'2014-08-15T19:43:53.999Z', u'module': u'ns_vbucket_mover', u'tstamp': 1408157033999, u'type': u'info'}

[2014-08-15 19:39:05,783] - [rest_client:2011] ERROR -

{u'node': u'ns_1@172.23.107.153', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'serverTime': u'2014-08-15T19:43:53.398Z', u'module': u'ns_rebalancer', u'tstamp': 1408157033398, u'type': u'info'}

[2014-08-15 19:39:05,783] - [rest_client:2011] ERROR -

{u'node': u'ns_1@172.23.107.156', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@172.23.107.156\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2014-08-15T19:43:52.890Z', u'module': u'ns_memcached', u'tstamp': 1408157032890, u'type': u'info'}