Details
-
Bug
-
Resolution: Duplicate
-
Major
-
7.2.1
-
Enterprise Edition 7.2.1 build 5917 on GCP
-
Untriaged
-
0
-
Unknown
Description
During the system test run, it was seen that during an ongoing rebalance in, the index node 011 served 3 scan requests.
From couchbase.log
{"num_connections":41,"index_not_found_errcount":459646455,"memory_quota":92593455104,"memory_used":89808819184,"memory_used_storage":73201337473,"memory_total_storage":88034570240,"memory_used_queue":268426663,"needs_restart":false,"num_cpu_core":48,"avg_resident_percent":9,"avg_mutation_rate":61050,"avg_drain_rate":235771,"avg_disk_bps":1013678585,"total_data_size":831202722964,"total_disk_size":226327368096,"num_storage_instances":45,"num_indexes":37,"num_goroutine":2452,"num_cgo_call":31980591504,"timestamp":"1692676920178034226","uptime":"9h31m2.690494132s","storage_mode":"plasma","cpu_utilization":3846.666666666667,"memory_rss":88685842432,"memory_free":797536256,"memory_total":101345796096,"indexer_state":"Active","timings/stats_response":"41077 88255464242 -8840484915598922670","total_requests":3,"total_rows_returned":407469,"total_rows_scanned":60476727,"MAINT_STREAM/svc-d-node-001.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com:8091/projector_latency":25561554,"MAINT_STREAM/svc-d-node-003.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com:8091/projector_latency":11648850,"MAINT_STREAM/svc-d-node-002.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com:8091/projector_latency":30449991,"MAINT_STREAM/svc-d-node-010.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com:8091/projector_latency":11652366,"default1:scope_0:coll_0:idx12_f3mR (replica 1):index_state":4,"default1:scope_0:coll_0:idx12_f3mR (replica 1):total_scan_duration":0,"default1:scope_0:coll_0:idx12_f3mR (replica 1):total_scan_request_duration":0,"default1:scope_0:coll_0:idx12_f3mR (replica 1):num_docs_pending":305,"default1:scope_0:coll_0:idx12_f3mR (replica 1):scan_wait_duration":0,"default1:scope_0:coll_0:idx12_f3mR (replica 1):num_docs_processed":464791012,"default1:scope_0:coll_0:idx12_f3mR (replica 1):num_rows_returned":0,"default1:scope_0:coll_0:idx12_f3mR (replica 1):build_progress":100,"default1:scope_0:coll_0:idx12_f3mR (replica 1):num_docs_queued":25070,"default1:scope_0:coll_0:idx12_f3mR (replica 1):scan_bytes_read":0,"default1:scope_0:coll_0:idx12_f3mR (replica 1):avg_ts_interval":145925896,"default1:scope_0:coll_0:idx12_f3mR (replica |
cbcollect during the ongoing rebalance ->
s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-i-node-004.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip |
s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-i-node-005.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip |
s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-i-node-006.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip |
s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-i-node-007.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip |
s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-i-node-011.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip |
s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-q-node-008.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip |
s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-q-node-009.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip |
s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-q-node-012.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip |
s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-d-node-001.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip |
s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-d-node-002.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip |
s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-d-node-003.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip |
s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-d-node-010.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip |
Supportal snapshot -> https://supportal.couchbase.com/customer/systestaug21/cluster/647603f42451cfc9228e25c30530215e
Cbcollect after the completion of rebalance ->
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-d-node-001.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-d-node-002.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-d-node-003.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-d-node-010.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-i-node-004.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-i-node-005.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-i-node-006.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-i-node-007.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-i-node-011.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-q-node-008.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-q-node-009.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-q-node-012.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
Supportal snapshot->
https://supportal.couchbase.com/snapshot/647603f42451cfc9228e25c30530215e::3
The test polls for total_requests from the /stats endpoint. This seems to have incremented to 3 on the incoming node at some point.
Attachments
Issue Links
- is cloned by
-
MB-58353 [System Test on cloud] Index node seems to be continuously serving scan requests during ongoing rebalance
- Open