Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-56339

[BP to 7.2.x] - Indexer rebalance seems to be hung since 7+ hours.

    XMLWordPrintable

Details

    Description

      QE test

      git fetch https://review.couchbase.org/TAF refs/changes/80/188180/45 && git checkout FETCH_HEAD
      pc=200
      #PrivatePreview
      sudo guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P "args=-i /tmp/ElixirVolumeTest_COGS.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.serverlessHospital.Murphy.PrivatePreview,skip_cleanup=False,bucket_names=GleamBook,doc_size=1024,bucket_type=membase,eviction_policy=fullEviction,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,maxttl=10,pc=${pc},mutation_perc=20,key_type=RandomKey,capella_run=true,skip_teardown_cleanup=true,wait_timeout=14400,index_timeout=3600,dataplane_id=${dataplane_id},num_dataplanes=${num_dataplanes},runtype=serverless,kv_disk_size=${kv_disk_size},get-cbcollect-info=false,throttling_off=true,kv_disk_size=500,index_disk_size=400,fts_disk_size=400,ten_together=True -m rest"
      

      There are 10 databases in the cluster. The GSI nodes were scaled from 2 to 4 due to the 2 old nodes over HWM. During the scaling operation, GSI is internally doing a indexer rebalance which seems to be hanging from many hours

       

      Issue Resolution
      During scaling, an GSI indexer rebalance froze and did not successfully complete. This was because an index snapshot was not correctly deleted and recreated. A flag now handles snapshots to ensure they are correctly deleted or recreated when indexes are updated during rebalancing.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            pavan.pb Pavan PB
            varun.velamuri Varun Velamuri
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty