Details
Description
Execute the following test case:
1. Create a 7 node cluster (node1 - node7)
2. Create 3 buckets and add load to it
3. Failover 2 nodes (node2 and node3) and rebalance
4. Let the rebalance finish
5. Add back node2 (recoveryType=delta) and node3 (recoveryType=full)
6. Rebalance and wait for rebalance to finish
7. Validate current and replicate items as well as failover logs
As part of cleanup - delete the buckets and then remove all nodes from the cluster
Post this test case, one the nodes start with following error. Error is seen when the data is getting loaded. This is node5.
MemcachedError: Memcached error #134 'Temporary failure': Temporary failure for vbucket :211 to mc 172.23.105.105:11210
After few test cases, the setup fails because it cannot add nodes to the cluster.
[2015-05-11 22:31:33,486] - [rest_client:731] ERROR - http://172.23.105.26:8091/controller/addNode error 400 reason: unknown ["Join completion call failed. Failed to start ns_server cluster processes back. Logs might have more details."]
[2015-05-11 22:31:33,534] - [rest_client:2186] INFO - Latest logs from UI on 172.23.105.26:
[2015-05-11 22:31:33,535] - [rest_client:2187] ERROR -
[2015-05-11 22:31:33,535] - [rest_client:2187] ERROR - {u'node': u'ns_1@172.23.105.26', u'code': 0, u'text': u'Add transaction of \'ns_1@172.23.105.105\' failed because of {error,\n complete_join,\n <<"Join completion call failed. Failed to start ns_server cluster processes back. Logs might have more details.">>,\n {error,\n rest_error,\n <<"Failed to start ns_server cluster processes back. Logs might have more details.">>,\n
{client_error,\n [<<"Failed to start ns_server cluster processes back. Logs might have more details.">>]}}}', u'shortText': u'message', u'serverTime': u'2015-05-11T22:31:32.825Z', u'module': u'ns_cluster', u'tstamp': 1431408692825, u'type': u'critical'}
Finally the node becomes unresponsive.