Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.0
-
Untriaged
-
1
-
Yes
Description
Issue found in 7.1.0-1238
Last good build is 7.1.0-1234
Test:
./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False,skip_log_scan=False,skip_security_scan=False -t fts.moving_topology_fts.MovingTopFTS.rebalance_2_nodes_during_index_building,items=30000,cluster=D,F,F,F,index_replicas=1,GROUP=P0,num_partitions=20,fts_quota=700,rebalance_out=True,disableFileTransferRebalance=False
From console:
17:39:13 [2021-09-02 17:39:13,562] - [rest_client:3896] ERROR - {'node': 'ns_1@172.23.105.151', 'type': 'critical', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1630629550687, 'shortText': 'message', 'text': 'Rebalance exited with reason {service_rebalance_failed,fts,\n {worker_died,\n {\'EXIT\',<0.27534.1>,\n {rebalance_failed,\n {service_error,\n <<"nodes: sample, res: (*http.Response)(nil), urlUUID: monitor.UrlUUID{Url:\\"https://172.23.105.153:18094\\", UUID:\\"71b244dfe083b79486569c7908eb6444\\"}, kind: /api/stats?partitions=true, err: Get https://@fts-cbauth:***@172.23.105.153:18094/api/stats?partitions=true: remote error: tls: internal error">>}}}}}.\nRebalance Operation Id = 85b17110c2dbb26225cb5146c5a31712', 'serverTime': '2021-09-02T17:39:10.687Z'}
|
17:39:13 [2021-09-02 17:39:13,563] - [rest_client:3896] ERROR - {'node': 'ns_1@172.23.105.151', 'type': 'info', 'code': 0, 'module': 'ns_vbucket_mover', 'tstamp': 1630629545105, 'shortText': 'message', 'text': 'Bucket "default" rebalance appears to be swap rebalance', 'serverTime': '2021-09-02T17:39:05.105Z'}
|
17:39:13 [2021-09-02 17:39:13,563] - [rest_client:3896] ERROR - {'node': 'ns_1@172.23.105.151', 'type': 'info', 'code': 0, 'module': 'ns_rebalancer', 'tstamp': 1630629544608, 'shortText': 'message', 'text': 'Started rebalancing bucket default', 'serverTime': '2021-09-02T17:39:04.608Z'}
|
17:39:13 [2021-09-02 17:39:13,563] - [rest_client:3896] ERROR - {'node': 'ns_1@172.23.105.151', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1630629543322, 'shortText': 'message', 'text': "Starting rebalance, KeepNodes = ['ns_1@172.23.105.151','ns_1@172.23.105.153'], EjectNodes = ['ns_1@172.23.107.160',\n 'ns_1@172.23.107.184'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 85b17110c2dbb26225cb5146c5a31712", 'serverTime': '2021-09-02T17:39:03.322Z'}
|
17:39:13 [2021-09-02 17:39:13,563] - [rest_client:3896] ERROR - {'node': 'ns_1@172.23.105.151', 'type': 'info', 'code': 0, 'module': 'ns_memcached', 'tstamp': 1630629497198, 'shortText': 'message', 'text': 'Bucket "default" loaded on node \'ns_1@172.23.105.151\' in 0 seconds.', 'serverTime': '2021-09-02T17:38:17.198Z'}
|
17:39:13 [2021-09-02 17:39:13,563] - [rest_client:3896] ERROR - {'node': 'ns_1@172.23.105.151', 'type': 'info', 'code': 12, 'module': 'menelaus_web', 'tstamp': 1630629497084, 'shortText': 'message', 'text': 'Created bucket "default" of type: couchbase\n[{num_replicas,1},\n {replica_index,true},\n {ram_quota,3079667712},\n {durability_min_level,none},\n {flush_enabled,true},\n {num_threads,3},\n {eviction_policy,value_only},\n {conflict_resolution_type,seqno},\n {storage_mode,couchstore},\n {max_ttl,0},\n {compression_mode,passive}]', 'serverTime': '2021-09-02T17:38:17.084Z'}
|
17:39:13 [2021-09-02 17:39:13,563] - [rest_client:3896] ERROR - {'node': 'ns_1@172.23.105.151', 'type': 'info', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1630629486087, 'shortText': 'message', 'text': 'Rebalance completed successfully.\nRebalance Operation Id = 6d4972a0aee15d3c7825f97e0477e74c', 'serverTime': '2021-09-02T17:38:06.087Z'}
|
17:39:13 [2021-09-02 17:39:13,563] - [rest_client:3896] ERROR - {'node': 'ns_1@172.23.107.184', 'type': 'info', 'code': 0, 'module': 'memcached_config_mgr', 'tstamp': 1630629485114, 'shortText': 'message', 'text': 'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]', 'serverTime': '2021-09-02T17:38:05.114Z'}
|
Attaching logs
Might be due to MB-47177