Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: 7.6.0
Affects Version/s: 7.6.0
Component/s: fts
Labels:
- approved-for-trinity
- candidate-for-trinity

Triage:
Untriaged
Story Points:
0
Is this a Regression?:
Yes

Description

7.6.0-2090

4 node Cluster with ['kv', 'fts', 'fts', 'fts'] ==> [172.23.109.94, 172.23.121.228, 172.23.121.247, 172.23.121.248]
Create a magma bucket: default

Enforce TLS:

2024-02-06 21:41:09 | INFO | MainProcess | test_thread | [remote_util.execute_couchbase_cli] command to run: /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost -u Administrator -p password --enable --no-ssl-verify

set disableFileTransferRebalance to True

2024-02-06 21:41:23 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=https://172.23.121.228:18094/api/managerOptions verb=PUT params={"disableFileTransferRebalance": "True"} client_cert=None verify=False

2024-02-06 21:41:23 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_disableFileTransferRebalance] SUCCESS: FTS disableFileTransferRebalance set to True

Create FTS index:

2024-02-06 21:41:23 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=https://172.23.121.247:18094/api/index/default_index verb=PUT params={"type": "fulltext-index", "name": "default_index", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}, "indexType": "scorch"}}, "sourceType": "couchbase", "sourceName": "default", "sourceUUID": "", "planParams": {"numReplicas": 1, "maxPartitionsPerPIndex": 171, "indexPartitions": 20}, "sourceParams": {}} client_cert=None verify=False

2024-02-06 21:41:24 | INFO | MainProcess | test_thread | [on_prem_rest_client.create_fts_index] Index default_index created

Load 30000 docs to bucket
While index is building, rebalance out 2 nodes (172.23.121.228, 172.23.121.247) and Rebalance fails

2024-02-06 21:41:52 | INFO | MainProcess | test_thread | [moving_topology_fts.rebalance_2_nodes_during_index_building] Index count for default_index: 28534

2024-02-06 21:41:52 | INFO | MainProcess | test_thread | [fts_base.__async_rebalance_out] Starting rebalance-out nodes:[ip:172.23.121.228 port:8091 ssh_username:root, ip:172.23.121.247 port:8091 ssh_username:root] at C1 cluster 172.23.109.94

2024-02-06 21:41:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.rebalance] rebalance params : {'knownNodes': 'ns_1@172.23.109.94,ns_1@172.23.121.228,ns_1@172.23.121.247,ns_1@172.23.121.248', 'ejectedNodes': 'ns_1@172.23.121.228,ns_1@172.23.121.247', 'user': 'Administrator', 'password': 'password'}

2024-02-06 21:41:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.rebalance] rebalance operation started

2024-02-06 21:42:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client._rebalance_status_and_progress] rebalance percentage : 62.00 %

2024-02-06 21:42:04 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 62.00%

2024-02-06 21:42:24 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._rebalance_status_and_progress] {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed

2024-02-06 21:42:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.print_UI_logs] Latest logs from UI on 172.23.109.94:

2024-02-06 21:42:24 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client.print_UI_logs] {'node': 'ns_1@172.23.121.248', 'type': 'critical', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1707284533882, 'shortText': 'message', 'text': 'Rebalance exited with reason {service_rebalance_failed,fts,\n                              {agent_died,<35220.6303.0>,\n                               {linked_process_died,<35220.6490.0>,\n                                {\'ns_1@172.23.121.228\',\n                                 {{badmatch,\n                                   {false,\n                                    {topology,[],\n                                     [<<"657c10b44672ae139d7a3b796a2935db">>,\n                                      <<"7b386f112f892b5afd47c4d255865400">>,\n                                      <<"ac47a7f0f7c443755b392e7f42f967ba">>],\n                                     true,[]},\n                                    {topology,[],\n                                     [<<"657c10b44672ae139d7a3b796a2935db">>,\n                                      <<"7b386f112f892b5afd47c4d255865400">>,\n                                      <<"ac47a7f0f7c443755b392e7f42f967ba">>],\n                                     false,[]}}},\n                                  [{service_agent,long_poll_worker_loop,5,\n                                    [{file,"src/service_agent.erl"},\n                                     {line,750}]},\n                                   {proc_lib,init_p,3,\n                                    [{file,"proc_lib.erl"},{line,225}]}]}}}}}.\nRebalance Operation Id = f8914302088fee95f881b15968877b7f', 'serverTime': '2024-02-06T21:42:13.882Z'}