Details
-
Bug
-
Resolution: Fixed
-
Test Blocker
-
5.0.0
-
4.7.0-806
-
Untriaged
-
Windows 64-bit
-
-
Yes
Description
Rebalance fails because - Compactor for database terminated unexpectedly
[2016-06-27 01:02:26,689] - [rest_client:1065] INFO - adding remote node @172.23.105.58:8091 to this cluster @172.23.105.239:8091
|
[2016-06-27 01:02:32,112] - [rest_client:1283] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.105.239%2Cns_1%40172.23.105.58
|
[2016-06-27 01:02:32,118] - [rest_client:1287] INFO - rebalance operation started
|
[2016-06-27 01:02:32,124] - [rest_client:1418] INFO - rebalance percentage : 0.00 %
|
[2016-06-27 01:02:42,151] - [rest_client:1418] INFO - rebalance percentage : 0.00 %
|
[2016-06-27 01:02:52,172] - [rest_client:1418] INFO - rebalance percentage : 0.00 %
|
[2016-06-27 01:03:02,193] - [rest_client:1418] INFO - rebalance percentage : 0.00 %
|
[2016-06-27 01:03:12,213] - [rest_client:1418] INFO - rebalance percentage : 0.00 %
|
[2016-06-27 01:03:22,234] - [rest_client:1418] INFO - rebalance percentage : 0.00 %
|
[2016-06-27 01:03:32,255] - [rest_client:1401] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
|
[2016-06-27 01:03:32,272] - [rest_client:2601] INFO - Latest logs from UI on 172.23.105.239:
|
[2016-06-27 01:03:32,272] - [rest_client:2602] ERROR - {u'node': u'ns_1@172.23.105.239', u'code': 2, u'text': u"Rebalance exited with reason {not_all_nodes_are_ready_yet,\n ['ns_1@172.23.105.58']}\n", u'shortText': u'message', u'serverTime': u'2016-06-27T01:03:32.273Z', u'module': u'ns_orchestrator', u'tstamp': 1467014612273, u'type': u'info'}
|
[2016-06-27 01:03:32,272] - [rest_client:2602] ERROR - {u'node': u'ns_1@172.23.105.58', u'code': 0, u'text': u'Compactor for database `default` (pid [{type,database},\n {important,true},\n {name,<<"default">>},\n {fa,\n {#Fun<compaction_new_daemon.4.92023696>,\n [<<"default">>,\n {config,\n {30,18446744073709551616},\n {30,18446744073709551616},\n undefined,false,false,\n {daemon_config,30,131072,20971520}},\n false,\n {[{type,bucket}]}]}}]) terminated unexpectedly: {{bad_return_value,\n {stop,\n {error,\n couldnt_connect_to_memcached}}},\n {gen_server,\n call,\n [{\'ns_memcached-default\',\n \'ns_1@172.23.105.58\'},\n {raw_stats,\n <<"diskinfo">>,\n #Fun<compaction_new_daemon.18.92023696>,\n {<<"0">>,\n <<"0">>}},\n 180000]}}', u'shortText': u'message', u'serverTime': u'2016-06-27T01:03:02.170Z', u'module': u'compaction_new_daemon', u'tstamp': 1467014582170, u'type': u'warning'}
|
[2016-06-27 01:03:32,272] - [rest_client:2602] ERROR - {u'node': u'ns_1@172.23.105.239', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'serverTime': u'2016-06-27T01:02:32.271Z', u'module': u'ns_rebalancer', u'tstamp': 1467014552271, u'type': u'info'}
|
[2016-06-27 01:03:32,272] - [rest_client:2602] ERROR - {u'node': u'ns_1@172.23.105.239', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.239','ns_1@172.23.105.58'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2016-06-27T01:02:32.237Z', u'module': u'ns_orchestrator', u'tstamp': 1467014552237, u'type': u'info'}
|
Attachments
Issue Links
- is duplicated by
-
MB-20317 Rebalance failure on Windows (spock)
- Closed
For Gerrit Dashboard: MB-20034 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
65531,5 | MB-20034: Try to upgrade gperftools to 2.5 | master | tlm | Status: MERGED | +2 | +1 |
65533,3 | MB-20034: Use gperftools to 2.5 on windows | master | tlm | Status: MERGED | +2 | +1 |
66364,1 | MB-20034: Move extension loading to below creation of threads | master | memcached | Status: ABANDONED | 0 | -1 |
66444,7 | MB-20034: Interlock startup/shutdown | master | memcached | Status: MERGED | +2 | +1 |
66790,2 | MB-20034: Enable BREAKPAD_NO_TERMINATE_THREAD for Windows | master | tlm | Status: MERGED | +2 | +1 |
66800,3 | Revert "MB-20034: Enable BREAKPAD_NO_TERMINATE_THREAD for Windows" | master | tlm | Status: MERGED | +2 | +1 |
66859,2 | MB-20034: Build new breakpad SHA | master | tlm | Status: MERGED | +2 | +1 |
66861,2 | MB-20034: Bump breakpad to newer version | master | tlm | Status: MERGED | +2 | +1 |