Details
-
Bug
-
Resolution: Duplicate
-
Test Blocker
-
4.0.0
-
Security Level: Public
-
Version: 4.0.0-2029 Enterprise Edition (build-2029)
-
Untriaged
-
Windows 64-bit
-
Yes
Description
1. Install CB Server on 4 nodes
2. Create bucket default and add 1000 documents on node 1
3. Create 2 view and query them on node2
4. Add node 2 to node 1
Rebalance fails.
Logs at - https://s3.amazonaws.com/bugdb/jira/MB-14802/logs.tar
Here is output from test:
2015-05-04 01:54:37 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.105.92:8091 to this cluster @172.23.105.82:8091
2015-05-04 01:54:41 | INFO | MainProcess | test_thread | [rebalance_progress.test_progress_rebalance_in] Got empty progress
2015-05-04 01:54:42 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.105.92%2Cns_1%40172.23.105.82
2015-05-04 01:54:42 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started
2015-05-04 01:54:42 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %
2015-05-04 01:54:53 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]
- rebalance failed
2015-05-04 01:54:53 | INFO | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] Latest logs from UI on 172.23.105.82:
2015-05-04 01:54:53 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs]
2015-05-04 01:54:53 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs]
{u'node': u'ns_1@172.23.105.92', u'code': 1, u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.105.92\'. Version: "4.0.0-2029-rel-enterprise".', u'shortText': u'web start ok', u'serverTime': u'2015-05-04T01:54:40.470Z', u'module': u'menelaus_sup', u'tstamp': 1430729680470, u'type': u'info'}2015-05-04 01:54:53 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs]
{u'node': u'ns_1@172.23.105.92', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@172.23.105.92\' for server shutdown', u'shortText': u'message', u'serverTime': u'2015-05-04T01:54:35.579Z', u'module': u'ns_memcached', u'tstamp': 1430729675579, u'type': u'info'}2015-05-04 01:54:53 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.105.82', u'code': 2, u'text': u"Rebalance exited with reason {unexpected_exit,\n {'EXIT',<0.18203.2>,\n error,{badrpc,nodedown,\n {gen_server,call,\n [
{'janitor_agent-default',\n 'ns_1@172.23.105.92'},\n {if_rebalance,<0.18066.2>,\n {wait_index_updated,1020}},\n infinity]}}}}\n", u'shortText': u'message', u'serverTime': u'2015-05-04T01:54:35.100Z', u'module': u'ns_orchestrator', u'tstamp': 1430729675100, u'type': u'info'}
2015-05-04 01:54:53 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.105.82', u'code': 0, u'text': u"<0.18152.2> exited with {unexpected_exit,\n {'EXIT',<0.18203.2>,\n error,{badrpc,nodedown,\n {gen_server,call,\n [
Errors from UI:
Rebalance exited with reason {unexpected_exit,
{'EXIT',<0.18203.2>,
error,{badrpc,nodedown,
{gen_server,call,
[{'janitor_agent-default', 'ns_1@172.23.105.92'},
{if_rebalance,<0.18066.2>,
{wait_index_updated,1020}},
infinity]}}}}
ns_orchestrator002 ns_1@172.23.105.82 01:54:35 - Mon May 4, 2015
<0.18152.2> exited with {unexpected_exit,
{'EXIT',<0.18203.2>,
error,{badrpc,nodedown,
{gen_server,call,
[{'janitor_agent-default','ns_1@172.23.105.92'}
,
{if_rebalance,<0.18066.2>,
{wait_index_updated,1020}},
infinity]}}}} ns_vbucket_mover000 ns_1@172.23.105.82 01:54:35 - Mon May 4, 2015
Bucket "default" rebalance appears to be swap rebalance ns_vbucket_mover000 ns_1@172.23.105.82 01:54:34 - Mon May 4, 2015