Details
-
Bug
-
Resolution: Cannot Reproduce
-
Critical
-
4.0.0
-
Security Level: Public
-
3645
-
Untriaged
-
Windows 64-bit
-
-
Yes
Description
1. Create a cluster of 5 nodes
.144 - n1ql + kv
.145 - kv
.191 - kv
.40 - index
.91 - index
2. Check if the cluster is healthy
3. Create 3 buckets and add 100800 items to each bucket
4. create primary index using views and secondary index using gsi. Wait for all indexes to be built.
5. Eject out of the KV nodes and rebalance.
[2015-07-20 05:50:54,363] - [rest_client:1166] INFO - rebalance params : password=password&ejectedNodes=ns_1%40172.23.105.191&user=Administrator&knownNodes=ns_1%40172.23.105.191%2Cns_1%40172.23.105.40%2Cns_1%40172.23.107.144%2Cns_1%40172.23.107.145%2Cns_1%40172.23.105.91
Rebalance fails:
[2015-07-20 05:50:54,376] - [rest_client:1170] INFO - rebalance operation started
[2015-07-20 05:50:54,469] - [rest_client:1288] INFO - rebalance percentage : 0.00 %
[2015-07-20 05:51:04,510] - [rest_client:1288] INFO - rebalance percentage : 1.56 %
[2015-07-20 05:51:14,613] - [rest_client:1288] INFO - rebalance percentage : 3.31 %
[2015-07-20 05:51:24,632] - [rest_client:1288] INFO - rebalance percentage : 4.18 %
[2015-07-20 05:51:34,650] - [rest_client:1288] INFO - rebalance percentage : 4.18 %
[2015-07-20 05:51:44,670] - [rest_client:1288] INFO - rebalance percentage : 5.00 %
[2015-07-20 05:51:54,749] - [rest_client:1288] INFO - rebalance percentage : 6.96 %
[2015-07-20 05:52:04,801] - [rest_client:1288] INFO - rebalance percentage : 8.53 %
[2015-07-20 05:52:14,821] - [rest_client:1288] INFO - rebalance percentage : 9.96 %
[2015-07-20 05:52:24,840] - [rest_client:1288] INFO - rebalance percentage : 9.96 %
[2015-07-20 05:52:34,860] - [rest_client:1288] INFO - rebalance percentage : 9.96 %
[2015-07-20 05:52:44,878] - [rest_client:1288] INFO - rebalance percentage : 9.96 %
[2015-07-20 05:52:54,898] - [rest_client:1288] INFO - rebalance percentage : 9.96 %
[2015-07-20 05:53:04,940] - [rest_client:1271] ERROR -
- rebalance failed
[2015-07-20 05:53:04,974] - [rest_client:2195] INFO - Latest logs from UI on 172.23.107.144:
[2015-07-20 05:53:04,974] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.107.144', u'code': 2, u'text': u"Rebalance exited with reason {unexpected_exit,\n {'EXIT',<0.27919.31>,\n {bulk_set_vbucket_state_failed,\n [{'ns_1@172.23.107.145',\n {'EXIT',\n error,{badrpc,nodedown,\n {gen_server,call,\n [
,\n {if_rebalance,<0.19056.31>,\n {update_vbucket_state,362,replica,\n undefined,'ns_1@172.23.107.144'}},\n