Details
-
Bug
-
Resolution: User Error
-
Critical
-
None
-
7.1.0
-
Untriaged
-
1
-
Unknown
Description
Test:
1 bucket 10 collections ,275M docs , uneven data in collections, mix replica (0-2), 3 partitioned , total 600 index instances , 3 indexer nodes, Rebalance in 1 index node
KV DGM - ~20% and reached 0 because of background mutation.
Indexer DGM - ~10%
Seeing the following errors in the scan
03:09:30 2021-12-14T13:39:30.643-08:00 [Error] [GsiScanClient:"172.23.110.55:9101"] Range(./opt/couchbase/bin/cbindexperf2282) response failed `Index scan timed out`
|
03:09:30 2021-12-14T13:39:30.643-08:00 [Error] [GsiScanClient:"172.23.110.55:9101"] Range(./opt/couchbase/bin/cbindexperf2279) response failed `Index scan timed out`
|
03:09:30 2021-12-14T13:39:30.643-08:00 [Error] PickRandom: Fail to find indexer for all index partitions. Num partition 3. Partition with instances 2
|
03:09:30 REQ:1 scan error occurred: Index scan timed out from [172.23.110.72:9101 172.23.110.55:9101]
|
03:09:30 2021-12-14T13:39:30.643-08:00 [Error] [GsiScanClient:"172.23.110.55:9101"] Range(./opt/couchbase/bin/cbindexperf2426) response failed `Index scan timed out`
|
03:09:30 2021-12-14T13:39:30.643-08:00 [Error] PickRandom: Fail to find indexer for all index partitions. Num partition 3. Partition with instances 2
|
03:09:30 REQ:1 scan error occurred: Index scan timed out from [172.23.110.72:9101 172.23.110.72:9101 172.23.110.55:9101]
|
03:09:30 2021-12-14T13:39:30.643-08:00 [Error] PickRandom: Fail to find indexer for all index partitions. Num partition 3. Partition with instances 0
|
03:09:30 REQ:1 scan error occurred: Index scan timed out from [172.23.110.56:9101 172.23.110.55:9101]
|
03:09:30 2021-12-14T13:39:30.649-08:00 [Error] [GsiScanClient:"172.23.110.55:9101"] Range(./opt/couchbase/bin/cbindexperf2284) response failed `Index scan timed out`
|
03:09:30 2021-12-14T13:39:30.649-08:00 [Error] PickRandom: Fail to find indexer for all index partitions. Num partition 3. Partition with instances 2
|
03:09:30 REQ:1 scan error occurred: Index scan timed out from [172.23.110.55:9101 172.23.110.72:9101]
|
03:09:30 2021-12-14T13:39:30.654-08:00 [Error] [GsiScanClient:"172.23.110.55:9101"] Range(./opt/couchbase/bin/cbindexperf2430) response failed `Index scan timed out`
|
03:09:30 2021-12-14T13:39:30.654-08:00 [Error] PickRandom: Fail to find indexer for all index partitions. Num partition 3. Partition with instances 2
|
03:09:30 2021-12-14T13:39:30.654-08:00 [Error] [GsiScanClient:"172.23.110.55:9101"] Range(./opt/couchbase/bin/cbindexperf2431) response failed `Index scan timed out`
|
03:09:30 2021-12-14T13:39:30.654-08:00 [Error] PickRandom: Fail to find indexer for all index partitions. Num partition 3. Partition with instances 2
|
03:09:30 2021-12-14T13:39:30.664-08:00 [Error] [GsiScanClient:"172.23.110.55:9101"] Range(./opt/couchbase/bin/cbindexperf2292) response failed `Index scan timed out`
|
03:09:30 2021-12-14T13:39:30.664-08:00 [Error] PickRandom: Fail to find indexer for all index partitions. Num partition 3. Partition with instances 2
|
03:09:30 REQ:1 scan error occurred: Index scan timed out from [172.23.110.72:9101 172.23.110.72:9101 172.23.110.55:9101]
|
Test job : http://perf.jenkins.couchbase.com/job/aether/1240/console
Cbmonitor:
http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=aether_710-1887_rebalance_e43d
Same test with rebalance out passed http://perf.jenkins.couchbase.com/job/aether/1238/console