Details
-
Bug
-
Resolution: Unresolved
-
Major
-
7.6.0
-
Capella
-
Untriaged
-
-
0
-
Unknown
Description
1. Created a capella cluster with the following configuration
| Service | Number of nodes | CPU cores | Memory | Disk size |
|---|---|---|---|---|
| KV | 3 | 8 | 32GB | 250GB |
| FTS | 5 | 16 | 32GB | 250B |
| N1QL | 2 | 4 | 16GB | 250GB |
2. Created a bucket and loaded 5M documents into the bucket. Each document has a 'vector_data' with vector of dimension 384.
3. Created an index on the 'vector_data' field.
4. Wait for indexing to complete.
5.Started an incremental load where we are inserting new documents as well as updating existing documents.
6. Loaded new documents to the bucket till the total docs in the bucket reached 15M. (During this insertion there were also mutations happening to the existing docs).
7. Wait for indexing to complete.
8. Started running queries on the index.
9. Started rebalance in operation(added a new FTS node).
10. Rebalance failed
Rebalance exited with reason {service_rebalance_failed,fts,\n {agent_died,<36743.2953.0>,noconnection}}. |
[ns_server:error,2024-02-19T06:39:17.236Z,ns_1@svc-s-node-006.489ywyi5z763ajqn.sandbox.nonprod-project-avengers.com:service_agent-fts<0.4531.0>:service_agent:handle_info:325]Service Manager <35147.9395.8> died unexpectedly: {agent_died,<35868.2953.0>, noconnection}[ns_server:error,2024-02-19T06:39:37.248Z,ns_1@svc-s-node-006.489ywyi5z763ajqn.sandbox.nonprod-project-avengers.com:service_agent-fts<0.4531.0>:service_agent:handle_info:320]Linked process <0.3034.6> died with reason {{badmatch, {false, {topology,[], [<<"24d2754028c8e123d8d174da008a8ae5">>, <<"7e18e9e0793da6ebdc31e5be04148bcb">>, <<"8c86178125eb72af47e7e33797275580">>, <<"8cb7b6e58837b20a2c2ebac67b3265f4">>, <<"9a9056753e585cd139c1b1a1ed57effe">>, <<"f3f29bce190196425eb15971cfd33750">>], true,[]}, {topology,[], [<<"24d2754028c8e123d8d174da008a8ae5">>, <<"7e18e9e0793da6ebdc31e5be04148bcb">>, <<"8c86178125eb72af47e7e33797275580">>, <<"8cb7b6e58837b20a2c2ebac67b3265f4">>, <<"9a9056753e585cd139c1b1a1ed57effe">>, <<"f3f29bce190196425eb15971cfd33750">>], false,[]}}}, [{service_agent, long_poll_worker_loop,5, [{file,"src/service_agent.erl"}, {line,750}]}, {proc_lib,init_p,3, [{file,"proc_lib.erl"}, {line,225}]}]}. Terminating[ns_server:error,2024-02-19T06:39:37.248Z,ns_1@svc-s-node-006.489ywyi5z763ajqn.sandbox.nonprod-project-avengers.com:service_agent-fts<0.4531.0>:service_agent:terminate:350]Terminating abnormally[ns_server:error,2024-02-19T06:39:37.248Z,ns_1@svc-s-node-006.489ywyi5z763ajqn.sandbox.nonprod-project-avengers.com:service_agent-fts<0.4531.0>:service_agent:terminate:355]Terminating json rpc connection for fts: <0.4742.0> |
11. I also see 2 of the FTS nodes are failed over. (svc-s-node-004 and svc-s-node-005).
I did not see any FTS crashes.
Attachments
Issue Links
- relates to
-
-
- Closed
-