Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-55505

Indexer is OOM killed while 50 Index building is hung infinitely for volumetestbucket-6-bc528t

    XMLWordPrintable

Details

    Description

      1. Create 19 databases to avoid kv/gsi auto-scaling due to num_tenants.
      2. Create 2 collections(VolumeCollection0, VolumeCollection1) in each database
      3. Create 100 indexes on each database i.e. 50 on each collection. This results in total 1900 indexes.
      4. Build 50 indexes per database and 10 databases at a time. Total = 10 * 50 * 2(replica) = 1000
      5. Example:

        Bucket0:VolumeCollection0 - 50 indexes under build
        Bucket1:VolumeCollection0 - 50 indexes under build
        Bucket2:VolumeCollection0 - 50 indexes under build
        .
        .
        .
        Bucket10:VolumeCollection0 - 10 indexes under build
        

      6. Repeat the above 2 steps to build all the 1900 *2 indexes
      7. Observe Indexer crashed during the above process:

        Service 'index' exited with status 137. Restarting. Messages:
        2023-02-07T23:02:18.949+00:00 [Info] volumetestbucket-2-zatn7d/volumetestbucket_2_zatn7d_idx_VolumeCollection1_61/Mainstore#164344719020020347:0 Plasma: SMR reclaim pending is higher than expected: pending = 101 KB (expected = 101 KB), wCtxCnt = 25, objCnt 5, changed reclaimList flush threshold from 1 to 0, changed reclaimSize flush threshold from 3 KB to 3 KB.
        2023-02-07T23:02:18.939+00:00 [Info] volumetestbucket-2-zatn7d/volumetestbucket_2_zatn7d_idx_VolumeCollection1_13/Mainstore#3640576359923206691:0 Plasma: SMR reclaim pending is higher than expected: pending = 174 KB (expected = 101 KB), wCtxCnt = 25, objCnt 7, changed reclaimList flush threshold from 1 to 0, changed reclaimSize flush threshold from 3 KB to 3 KB.
        2023-02-07T23:02:19.833+00:00 [Info] volumetestbucket-2-zatn7d/volumetestbucket_2_zatn7d_idx_VolumeCollection1_79/Mainstore#2068248693329795510:0 Plasma: SMR reclaim pending is higher than expected: pending = 102 KB (expected = 101 KB), wCtxCnt = 26, objCnt 7, changed reclaimList flush threshold from 2 to 0, changed reclaimSize flush threshold from 3 KB to 3 KB.
        

      QE Test

      # GSI auto-scaling patch
      git fetch https://review.couchbase.org/TAF refs/changes/05/185605/19 && git checkout FETCH_HEAD
       
      sudo guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P "args=-i /tmp/ElixirVolumeTest1.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.serverlessHospital.Murphy.ElixirVolume,skip_cleanup=False,num_buckets=${num_buckets},bucket_names=GleamBook,doc_size=1024,bucket_type=membase,eviction_policy=fullEviction,iterations=${iterations},batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,maxttl=10,pc=${pc},mutation_perc=20,key_type=RandomKey,capella_run=true,skip_teardown_cleanup=false,wait_timeout=14400,index_timeout=3600,dataplane_id=${dataplane_id},cb_image=${cb_image},dn_image=${dn_image},dapi_image=${dapi_image},num_dataplanes=${num_dataplanes},runtype=serverless,kv_disk_size=${kv_disk_size},clients_per_db=5,skip_teardown_cleanup=true -m rest"
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty