Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
7.6.0
-
7.6.0-2017
-
Untriaged
-
-
0
-
Unknown
Description
- Create a 2 KV, 4 FTS nodes cluster. FTS service RAM quota is 19037MB
- Create 1 Magma bucket with 2 collections.
- Start loading fashion data with 384 dimensional embedding data on product description. 1M in each collection.
- Create fts vector indexes 2 for each collection. In total 4 FTS indexes
- Start an upsert workload on vector fields
- Start 10 queries per sec workload randomly running on any FTS index
- Rebalance in a FTS node - Passed
- Rebalance out a FTS node - Passed
- Rebalance in 2 nodes, out 1 FTS node - Passed
- Swap rebalance FTS node - Passed
- Failover 1 node and RebalanceOut that node - Rebalance failed
172.23.107.237 |
Rebalance exited with reason {service_rebalance_failed,fts,
|
{agent_died,<35518.29035.1>,
|
{lost_connection,
|
{'ns_1@172.23.107.221',shutdown}}}}.
|
Rebalance Operation Id = a479bfcb5cca4dda9e4644311e1a57b4
|
|
Service 'fts' exited with status 137. Restarting. Messages:
|
2024-01-16T23:16:30.919-08:00 [INFO] grpc_client: grpc ClientConn Created 3 for host: b405d5108d6d3bd77a1e9e4ec411c224-172.23.107.237:19130
|
2024-01-16T23:16:30.919-08:00 [INFO] grpc_client: grpc ClientConn Created 4 for host: b405d5108d6d3bd77a1e9e4ec411c224-172.23.107.237:19130
|
2024-01-16T23:16:30.919-08:00 [INFO] grpc_client: grpc ClientConn Created 0 for host: 39a447ba1854e950a75b534411c3adf5-172.23.107.232:19130
|
2024-01-16T23:16:30.919-08:00 [INFO] grpc_client: grpc ClientConn Created 1 for host: 39a447ba1854e950a75b534411c3adf5-172.23.107.232:19130
|
2024-01-16T23:16:30.919-08:00 [INFO] grpc_client: grpc ClientConn Created 2 for host: 39a447ba1854e950a75b534411c3adf5-172.23.107.232:19130
|
2024-01-16T23:16:30.920-08:00 [INFO] grpc_client: grpc ClientConn Created 3 for host: 39a447ba1854e950a75b534411c3adf5-172.23.107.232:19130
|
2024-01-16T23:16:30.920-08:00 [INFO] grpc_client: grpc ClientConn Created 4 for host: 39a447ba1854e950a75b534411c3adf5-172.23.107.232:19130
|
2024-01-16T23:20:24.354-08:00 [WARN] (GOCBCORE) Config block decode failure (context canceled) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:742
|