Details
-
Bug
-
Resolution: Won't Fix
-
Critical
-
Cheshire-Cat
-
Untriaged
-
1
-
Yes
Description
Install Couchbase server 7.0.0-2310 on 5 ubuntu 18.04
Run test with several backup restore and cluster operations.
One of a test is rebalance in a node, rebalance failed with error service_rebalance_failed,index.
2020-06-10 05:30:02,531 - root - INFO - adding remote node @172.31.18.93:8091 to this cluster @172.31.31.44:8091
|
2020-06-10 05:30:04,176 - root - INFO - rebalance params : {'knownNodes': 'ns_1@172.31.18.93,ns_1@172.31.31.44', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'}
|
2020-06-10 05:30:04,180 - root - INFO - rebalance operation started
|
2020-06-10 05:30:04,183 - root - INFO - rebalance percentage : 0.00 %
|
2020-06-10 05:30:04,183 - root - INFO - Rebalance - status: running, progress: 0.00%
|
2020-06-10 05:30:14,199 - root - ERROR - {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed
|
2020-06-10 05:30:14,203 - root - INFO - Latest logs from UI on 172.31.18.93:
|
2020-06-10 05:30:14,203 - root - ERROR - {'node': 'ns_1@172.31.31.44', 'type': 'critical', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1591767007078, 'shortText': 'message', 'text': 'Rebalance exited with reason {service_rebalance_failed,index,\n {agent_died,<0.21828.8>,\n {lost_connection,shutdown}}}.\nRebalance Operation Id = 95d12eb929df921cea8f545c28636f2d', 'serverTime': '2020-06-10T05:30:07.078Z'}
|
2020-06-10 05:30:14,203 - root - ERROR - {'node': 'ns_1@172.31.31.44', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1591767004163, 'shortText': 'message', 'text': "Starting rebalance, KeepNodes = ['ns_1@172.31.18.93','ns_1@172.31.31.44'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 95d12eb929df921cea8f545c28636f2d", 'serverTime': '2020-06-10T05:30:04.163Z'}
|
2020-06-10 05:30:14,203 - root - ERROR - {'node': 'ns_1@172.31.18.93', 'type': 'info', 'code': 0, 'module': 'memcached_config_mgr', 'tstamp': 1591767003902, 'shortText': 'message', 'text': 'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]', 'serverTime': '2020-06-10T05:30:03.902Z'}
|
2020-06-10 05:30:14,203 - root - ERROR - {'node': 'ns_1@172.31.18.93', 'type': 'info', 'code': 3, 'module': 'ns_cluster', 'tstamp': 1591767003861, 'shortText': 'message', 'text': 'Node ns_1@172.31.18.93 joined cluster', 'serverTime': '2020-06-10T05:30:03.861Z'}
|
20
|
This cluster is in EC2. If you need access to this cluster, I will give credentials to login