Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
7.2.1
-
Enterprise Edition 7.2.1 build 5917 on GCP
-
Untriaged
-
0
-
Unknown
Description
During the system test run, it was seen that during an ongoing rebalance in, the index node 019 has continuously served scan requests. Unlike that in https://issues.couchbase.com/browse/MB-58336 which was just 3 scan requests, this rebalance has served almost ~45k requests. This has caused the rebalance time to increase (It's still not done after 12 hours).
From couchbase.log
{"num_connections":368,"index_not_found_errcount":99203,"memory_quota":92593455104,"memory_used":76329953872,"memory_used_storage":51364449296,"memory_total_storage":74378715136,"memory_used_queue":268436048,"needs_restart":false,"num_cpu_core":72,"avg_resident_percent":7,"avg_mutation_rate":2955,"avg_drain_rate":11865,"avg_disk_bps":32792439,"total_data_size":575969420540,"total_disk_size":153241640498,"num_storage_instances":45,"num_indexes":26,"num_goroutine":5608,"num_cgo_call":47971998972,"timestamp":"1692763587024397331","uptime":"11h54m54.207584052s","storage_mode":"plasma","cpu_utilization":7035.441603040709,"memory_rss":75660312576,"memory_free":696401920,"memory_total":152071409664,"indexer_state":"Active","timings/stats_response":"55700 136882430997 -5390748120798101091","total_requests":46107,"total_rows_returned":18390455,"total_rows_scanned":1495552075200,"MAINT_STREAM/ |
cbcollect during the ongoing rebalance ->
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-23T040425-ns_1%40svc-d-node-013.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-23T040425-ns_1%40svc-d-node-014.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-23T040425-ns_1%40svc-d-node-015.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-23T040425-ns_1%40svc-d-node-016.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-23T040425-ns_1%40svc-i-node-004.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-23T040425-ns_1%40svc-i-node-007.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-23T040425-ns_1%40svc-i-node-011.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-23T040425-ns_1%40svc-i-node-017.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-23T040425-ns_1%40svc-i-node-018.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-23T040425-ns_1%40svc-i-node-019.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-23T040425-ns_1%40svc-q-node-008.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-23T040425-ns_1%40svc-q-node-009.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-23T040425-ns_1%40svc-q-node-012.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
Supportal snapshot -> https://supportal.couchbase.com/snapshot/647603f42451cfc9228e25c30530215e%3A%3A5
I didn't really want to change anything in MB-58336 since it was just 3 requests and didn't really affect the performance, but this one is slightly concerning. Please feel free to close this as duplicate in case the root cause happens to be the same.
Attachments
Issue Links
- Clones
-
-
- Closed
-