Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50381

Stress test: indexer build is hung

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.1.0-2049 

      As part of MB-50138 , we are running 10 Bucket, 10K index, 10k ops. Index build seems to be hung even when all the load is stopped. Everything is in memory. Scans are also timing out 

      Test job : http://perf.jenkins.couchbase.com/job/aether/1403/console 

      Scan fails 

      12:42:03 2022-01-12T23:17:19 [INFO] To be applied: ./opt/couchbase/bin/cbindexperf -cluster 172.23.110.72:8091 -auth="Administrator:password" -configfile tests/gsi/scan_config/config_scanlatency_10k_indexes_1s_100c_10b.json -resultfile result.json -statsfile /root/statsfile -cpuprofile cpuprofile.prof -memprofile memprofile.prof 
      12:42:03 Log Level = error
      		 
      12:42:03 2022-01-12T23:17:20.703-08:00 [Error] updateIndexerList(): Cannot locate cluster node.
      		 
      12:42:04 2022-01-12T23:17:22.706-08:00 [Error] watcher.updateIndexStats2NoLock: unexpected nil *DedupedIndexStats. bucket bucket-1, dedupedIndexStats &{97425 0 1642052460775939398 1642058235781069653 map[]}, clientStatsPtr.Stats[bucket] <nil>
      		 
      12:42:06 2022-01-12T23:17:25.726-08:00 [Error] updateIndexerList(): Cannot locate cluster node.
      		 
      12:42:09 2022-01-12T23:17:26.593-08:00 [Error] updateIndexerList(): Cannot locate cluster node.
      		 
      12:42:10 2022-01-12T23:17:26.946-08:00 [Error] watcher.updateIndexStats2NoLock: unexpected nil *DedupedIndexStats. bucket bucket-7, dedupedIndexStats &{72589 0 1642053669811278999 1642058241164600231 map[]}, clientStatsPtr.Stats[bucket] <nil>
      		 
      12:42:11 2022-01-12T23:19:34.397-08:00 [Error] [GsiScanClient:"172.23.110.72:9101"] Range(./opt/couchbase/bin/cbindexperf1) response failed `Index scan timed out`
      		 
      12:44:18 2022-01-12T23:19:34.397-08:00 [Error] PickRandom: Fail to find indexer for all index partitions. Num partition 1.  Partition with instances 0 
      12:44:18 2022-01-12T23:21:34.760-08:00 [Error] [GsiScanClient:"172.23.110.72:9101"] Range(./opt/couchbase/bin/cbindexperf1) response failed `Index scan timed out`
      		 
      12:46:18 2022-01-12T23:21:34.760-08:00 [Error] PickRandom: Fail to find indexer for all index partitions. Num partition 1.  Partition with instances 0

      Deque waiting in indexer log

      indexer.log:2022-01-13T00:02:43.939-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 52 KeyspaceId bucket-3:scope-1:collection-81 Vbucket 21 for 1625000 ms. Curr Count 8 Seq 6388.
      		 
      indexer.log:2022-01-13T00:02:43.942-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 43 KeyspaceId bucket-3:scope-1:collection-81 Vbucket 4 for 1625000 ms. Curr Count 3 Seq 5254.
      		 
      indexer.log:2022-01-13T00:02:43.943-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 76015559177012 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 546 for 1240000 ms. Curr Count 1 Seq 7139.
      		 
      indexer.log:2022-01-13T00:02:43.944-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 114302738965535 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 738 for 1240000 ms. Curr Count 0 Seq 0.
      		 
      indexer.log:2022-01-13T00:02:43.957-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 161010715798602 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 324 for 1240000 ms. Curr Count 0 Seq 0.
      		 
      indexer.log:2022-01-13T00:02:43.964-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 264370254302919 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 323 for 1240000 ms. Curr Count 1 Seq 5265.
      		 
      indexer.log:2022-01-13T00:02:43.985-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 93033960398863 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 35 for 1240000 ms. Curr Count 2 Seq 6720.
      		 
      indexer.log:2022-01-13T00:02:43.994-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 56 KeyspaceId bucket-3:scope-1:collection-81 Vbucket 15 for 1625000 ms. Curr Count 10 Seq 6968.
      		 
      indexer.log:2022-01-13T00:02:44.000-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 187482132278586 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 480 for 1240000 ms. Curr Count 1 Seq 7944.
      		 
      indexer.log:2022-01-13T00:02:44.005-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 111587831130591 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 421 for 1240000 ms. Curr Count 1 Seq 5092.
      		 
      indexer.log:2022-01-13T00:02:44.010-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 60 KeyspaceId bucket-3:scope-1:collection-81 Vbucket 6 for 1625000 ms. Curr Count 11 Seq 7541.
      		 
      indexer.log:2022-01-13T00:02:44.035-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 28069123767533 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 451 for 1240000 ms. Curr Count 1 Seq 7268.
      		 
      indexer.log:2022-01-13T00:02:44.044-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 176092820318188 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 165 for 1240000 ms. Curr Count 1 Seq 4943.
      		 
      indexer.log:2022-01-13T00:02:44.044-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 62364981380720 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 743 for 1240000 ms. Curr Count 1 Seq 7860.
      		 
      indexer.log:2022-01-13T00:02:44.058-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 145768316621524 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 644 for 1240000 ms. Curr Count 2 Seq 6841.
      		 
      indexer.log:2022-01-13T00:02:44.075-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 183116481699841 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 420 for 1240000 ms. Curr Count 1 Seq 5795.
      		 
      indexer.log:2022-01-13T00:02:44.084-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 271967262864975 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 640 for 1240000 ms. Curr Count 0 Seq 0.
      		 
      indexer.log:2022-01-13T00:02:44.085-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 218599757712147 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 321 for 1240000 ms. Curr Count 4 Seq 6924.
      		 
      indexer.log:2022-01-13T00:02:44.089-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 200869071814805 KeyspaceId bucket-3:scope-1:collection-44 Vbucket 58 for 1410000 ms. Curr Count 50 Seq 5716.
      		 
      indexer.log:2022-01-13T00:02:44.097-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 219933842797571 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 67 for 1240000 ms. Curr Count 4 Seq 5480.
      		 
      indexer.log:2022-01-13T00:02:44.099-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 109881965724181 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 839 for 1240000 ms. Curr Count 2 Seq 6845.
      		 
      indexer.log:2022-01-13T00:02:44.103-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 66366208733307 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 705 for 1240000 ms. Curr Count 1 Seq 5268.
      		 
      indexer.log:2022-01-13T00:02:44.124-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 89210615538836 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 164 for 1240000 ms. Curr Count 2 Seq 5366.
      		 
      indexer.log:2022-01-13T00:02:44.134-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 142759230002364 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 258 for 1240000 ms. Curr Count 4 Seq 5041.
      		 
      indexer.log:2022-01-13T00:02:44.137-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 223542260814336 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 384 for 1240000 ms. Curr Count 3 Seq 6146.
      		 
      indexer.log:2022-01-13T00:02:44.141-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 178463877315014 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 449 for 1240000 ms. Curr Count 1 Seq 5786.
      		 
      indexer.log:2022-01-13T00:02:44.144-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 94089818405022 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 481 for 1240000 ms. Curr Count 1 Seq 6777.
      		 
      indexer.log:2022-01-13T00:02:44.165-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 44 KeyspaceId bucket-3:scope-1:collection-81 Vbucket 11 for 1625000 ms. Curr Count 6 Seq 5386.
      		 
      indexer.log:2022-01-13T00:02:44.170-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 75747163732513 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 419 for 1240000 ms. Curr Count 2 Seq 5307.
      		 
      indexer.log:2022-01-13T00:02:44.181-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 53345724560720 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 802 for 1240000 ms. Curr Count 2 Seq 6837.
      		 
      indexer.log:2022-01-13T00:02:44.182-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 171180467146265 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 97 for 1240000 ms. Curr Count 1 Seq 5255.
      		 
      indexer.log:2022-01-13T00:02:44.184-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 144120952798453 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 736 for 1240000 ms. Curr Count 1 Seq 6787.
      		 
      indexer.log:2022-01-13T00:02:44.187-08:00 [Warn] Indexer::MutationQueue Dequeue Waiting For Count 149245757917979 KeyspaceId bucket-5:scope-1:collection-53 Vbucket 101 for 1240000 ms. Curr Count 3 Seq 6585.

       

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-50138 Autofailover stress tests

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              vikas.chaudhary Vikas Chaudhary
              vikas.chaudhary Vikas Chaudhary
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty