Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-56195

[BP to 7.1.5] Avoid high indexer memory usage: change minVbQueueLength

    XMLWordPrintable

Details

    Description

      As part of CBPS-1037 , we are running following test on AWS

      Test: 

      • 4 Node cluster (3 Kv, 1 index)
      • Instance: EC2 - c6gd.4xlarge (16 vCPU, 32 GB memory) with NVME cache in front of EBS
      • KV : 20 buckets, Each bucket with 5 collections 
      • GSI
        • 2 indexes per collection, total of 200 indexes
        • memory quota: 23G (23552MB)
        • Key size = 256Bytes (200M total keys)
        • 10% RR

      Observed indexer restarted during incremental phase

      Service 'index' exited with status 2. Restarting. Messages:
      		 
      github.com/couchbase/indexing/secondary/indexer.(*timekeeper).sendNewStabilityTS.func2(0x4004035710, 0x423a775140, 0x4035cea260, 0x9, 0x1, 0x458b9ef000, 0x400, 0x400, 0x0, 0x0, ...)
      		 
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3049 +0x1ac
      		 
      created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).sendNewStabilityTS
      		 
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:3032 +0x404
      		 
       
      		 
       
      		 
      goroutine 5032953 [select]:
      		 
      github.com/couchbase/indexing/secondary/indexer.(*timekeeper).startTimer.func1(0x40422a0190, 0x4004035710, 0x4008420001, 0x40297e25c0, 0x9, 0x4007010540)
      		 
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:4349 +0x88
      		 
      created by github.com/couchbase/indexing/secondary/indexer show...ns_log 000ns_1@ec2-44-199-231-119.compute-1.amazonaws.com 7:53:37 AM 30 Jun, 2022

       

      Issue Resolution
      When indexer was under high memory pressure, queuing 256k mutations added to more memory pressure. For each bucket, indexer queued a minimum of 256k mutations before throttling for memory. The number of queued mutations has been reduced so that indexer can handle high memory pressure situations much better.

      Attachments

        Issue Links

          is a backport of

          Bug - A problem which impairs or prevents the functions of the product. MB-56162 Avoid high indexer memory usage: change minVbQueueLength

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              vikas.chaudhary Vikas Chaudhary
              varun.velamuri Varun Velamuri
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty