Details
-
Bug
-
Resolution: Cannot Reproduce
-
Critical
-
4.5.0
-
Windows 2012 R2 64-bit
-
Untriaged
-
Windows 64-bit
-
Yes
Description
Run test to upgrade from 3.1.0 to 4.5.0-2589
Start with two 3.1.0 nodes and create cluster
Then add 2 nodes 4.5.0-2589 in. Rebalance. Rebalance failed at about 92%
This test was passed in build 4.5.0-2587
Here is the test command to run:
newupgradetests.MultiNodesUpgradeTests.online_upgrade_rebalance_in_with_ops,initial_version=3.1.0-1776,sasl_buckets=1,standard_buckets=1,items=10000,nodes_init=2,nodes_in=1,nodes_out=1,ddocs-num=1,upgrade_version=4.5.0-2589,get-cbcollect-info=True,stop-on-failure=false,skip_init_check_cbserver=true
|
|
{u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
|
[2016-05-24 02:44:09,679] - [rest_client:2529] INFO - Latest logs from UI on 172.23.105.91:
|
[2016-05-24 02:44:09,680] - [rest_client:2530] ERROR - {u'node': u'ns_1@172.23.107.119', u'code': 2, u'text': u"Rebalance exited with reason {unexpected_exit,\n {'EXIT',<0.16098.4>,\n {bulk_set_vbucket_state_failed,\n [{'ns_1@172.23.105.91',\n {'EXIT',\n {{{badmatch,{error,closed}},\n {gen_server,call,\n ['ns_memcached-default',\n {set_vbucket,658,replica},\n 180000]}},\n {gen_server,call,\n [{'janitor_agent-default',\n 'ns_1@172.23.105.91'},\n {if_rebalance,<0.2352.3>,\n {update_vbucket_state,658,replica,\n undefined,undefined}},\n infinity]}}}}]}}}\n", u'shortText': u'message', u'serverTime': u'2016-05-24T02:44:06.118Z', u'module': u'ns_orchestrator', u'tstamp': 1464083046118, u'type': u'info'}
|
[2016-05-24 02:44:09,680] - [rest_client:2530] ERROR - {u'node': u'ns_1@172.23.107.119', u'code': 0, u'text': u"<0.14922.4> exited with {unexpected_exit,\n {'EXIT',<0.16098.4>,\n {bulk_set_vbucket_state_failed,\n [{'ns_1@172.23.105.91',\n {'EXIT',\n {{{badmatch,{error,closed}},\n {gen_server,call,\n ['ns_memcached-default',\n {set_vbucket,658,replica},\n 180000]}},\n {gen_server,call,\n [{'janitor_agent-default',\n 'ns_1@172.23.105.91'},\n {if_rebalance,<0.2352.3>,\n {update_vbucket_state,658,replica,\n undefined,undefined}},\n infinity]}}}}]}}}", u'shortText': u'message', u'serverTime': u'2016-05-24T02:44:06.112Z', u'module': u'ns_vbucket_mover', u'tstamp': 1464083046112, u'type': u'critical'}
|
[2016-05-24 02:44:09,681] - [rest_client:2530] ERROR - {u'node': u'ns_1@172.23.105.91', u'code': 0, u'text': u"Control connection to memcached on 'ns_1@172.23.105.91' disconnected: {badmatch,\n {error,\n closed}}", u'shortText': u'message', u'serverTime': u'2016-05-24T02:44:05.811Z', u'module': u'ns_memcached', u'tstamp': 1464083045811, u'type': u'info'}
|
[2016-05-24 02:44:09,681] - [rest_client:2530] ERROR - {u'node': u'ns_1@172.23.105.91', u'code': 0, u'text': u"Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 1. Restarting. Messages: Tue May 24 02:44:05.533493 Pacific Daylight Time 3: (default) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.107.119->ns_1@172.23.105.91:default - (vb 653) Attempting to add stream with start seqno 390, end seqno 18446744073709551615, vbucket uuid 160618223825921, snap start seqno 390, and snap end seqno 390\nTue May 24 02:44:05.600491 Pacific Daylight Time 3: (default) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.107.119->ns_1@172.23.105.91:default - (vb 654) Attempting to add stream with start seqno 140, end seqno 18446744073709551615, vbucket uuid 106785368150398, snap start seqno 140, and snap end seqno 140\nTue May 24 02:44:05.655491 Pacific Daylight Time 3: (default) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.107.119->ns_1@172.23.105.91:default - (vb 649) Attempting to add stream with start seqno 27, end seqno 18446744073709551615, vbucket uuid 92987305102627, snap start seqno 27, and snap end seqno 27\nTue May 24 02:44:05.721495 Pacific Daylight Time 3: (default) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.107.119->ns_1@172.23.105.91:default - (vb 656) Attempting to add stream with start seqno 336, end seqno 18446744073709551615, vbucket uuid 269900064377010, snap start seqno 336, and snap end seqno 336\nTue May 24 02:44:05.770491 Pacific Daylight Time 3: (default) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.107.119->ns_1@172.23.105.91:default - (vb 655) Attempting to add stream with start seqno 112, end seqno 18446744073709551615, vbucket uuid 119739259461295, snap start seqno 112, and snap end seqno 112", u'shortText': u'message', u'serverTime': u'2016-05-24T02:44:05.811Z', u'module': u'ns_log', u'tstamp': 1464083045811, u'type': u'info'}
|
[2016-05-24 02:44:09,681] - [rest_client:2530] ERROR - {u'node': u'ns_1@172.23.107.119', u'code': 0, u'text': u'Bucket "default" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2016-05-24T02:38:28.318Z', u'module': u'ns_vbucket_mover', u'tstamp': 1464082708318, u'type': u'info'}
|
[2016-05-24 02:44:09,681] - [rest_client:2530] ERROR - {u'node': u'ns_1@172.23.107.119', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@172.23.107.119\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2016-05-24T02:38:26.856Z', u'module': u'ns_memcached', u'tstamp': 1464082706856, u'type': u'info'}
|