Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-58336

[System Test on cloud] Index node seems to have served scan requests during ongoing rebalance

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • 7.6.0, 7.2.4
    • 7.2.1
    • secondary-index
    • Enterprise Edition 7.2.1 build 5917 on GCP
    • Untriaged
    • 0
    • Unknown

    Description

      During the system test run, it was seen that during an ongoing rebalance in, the index node 011 served 3 scan requests.

      From couchbase.log

      {"num_connections":41,"index_not_found_errcount":459646455,"memory_quota":92593455104,"memory_used":89808819184,"memory_used_storage":73201337473,"memory_total_storage":88034570240,"memory_used_queue":268426663,"needs_restart":false,"num_cpu_core":48,"avg_resident_percent":9,"avg_mutation_rate":61050,"avg_drain_rate":235771,"avg_disk_bps":1013678585,"total_data_size":831202722964,"total_disk_size":226327368096,"num_storage_instances":45,"num_indexes":37,"num_goroutine":2452,"num_cgo_call":31980591504,"timestamp":"1692676920178034226","uptime":"9h31m2.690494132s","storage_mode":"plasma","cpu_utilization":3846.666666666667,"memory_rss":88685842432,"memory_free":797536256,"memory_total":101345796096,"indexer_state":"Active","timings/stats_response":"41077 88255464242 -8840484915598922670","total_requests":3,"total_rows_returned":407469,"total_rows_scanned":60476727,"MAINT_STREAM/svc-d-node-001.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com:8091/projector_latency":25561554,"MAINT_STREAM/svc-d-node-003.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com:8091/projector_latency":11648850,"MAINT_STREAM/svc-d-node-002.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com:8091/projector_latency":30449991,"MAINT_STREAM/svc-d-node-010.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com:8091/projector_latency":11652366,"default1:scope_0:coll_0:idx12_f3mR (replica 1):index_state":4,"default1:scope_0:coll_0:idx12_f3mR (replica 1):total_scan_duration":0,"default1:scope_0:coll_0:idx12_f3mR (replica 1):total_scan_request_duration":0,"default1:scope_0:coll_0:idx12_f3mR (replica 1):num_docs_pending":305,"default1:scope_0:coll_0:idx12_f3mR (replica 1):scan_wait_duration":0,"default1:scope_0:coll_0:idx12_f3mR (replica 1):num_docs_processed":464791012,"default1:scope_0:coll_0:idx12_f3mR (replica 1):num_rows_returned":0,"default1:scope_0:coll_0:idx12_f3mR (replica 1):build_progress":100,"default1:scope_0:coll_0:idx12_f3mR (replica 1):num_docs_queued":25070,"default1:scope_0:coll_0:idx12_f3mR (replica 1):scan_bytes_read":0,"default1:scope_0:coll_0:idx12_f3mR (replica 1):avg_ts_interval":145925896,"default1:scope_0:coll_0:idx12_f3mR (replica
      

      cbcollect during the ongoing rebalance ->

      s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-i-node-004.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-i-node-005.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-i-node-006.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-i-node-007.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-i-node-011.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-q-node-008.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-q-node-009.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-q-node-012.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-d-node-001.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-d-node-002.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-d-node-003.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      s3://cb-customers-secure/systestaug21/2023-08-22/collectinfo-2023-08-22t040004-ns_1@svc-d-node-010.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      

      Supportal snapshot -> https://supportal.couchbase.com/customer/systestaug21/cluster/647603f42451cfc9228e25c30530215e

      Cbcollect after the completion of rebalance ->

      https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-d-node-001.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-d-node-002.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-d-node-003.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-d-node-010.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-i-node-004.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-i-node-005.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-i-node-006.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-i-node-007.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-i-node-011.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-q-node-008.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-q-node-009.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestAug21/collectinfo-2023-08-22T065840-ns_1%40svc-q-node-012.jhbuvuqazioyn5kh.sandbox.nonprod-project-avengers.com.zip

      Supportal snapshot->

      https://supportal.couchbase.com/snapshot/647603f42451cfc9228e25c30530215e::3

      The test polls for total_requests from the /stats endpoint. This seems to have incremented to 3 on the incoming node at some point.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              pavan.pb Pavan PB
              pavan.pb Pavan PB
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty