Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-53118

Warmup can fail to make forward progress if Warmup LoadStorageKVPairCallback takes takes longer than 10ms

    XMLWordPrintable

Details

    Description

      Steps:

      1. Create a 3 node cluster
      2. Create a magma bucket with 256MB, 256vB and 2 replicas.
      3. Load 25M items to put bucket under 1% DGM
      4. Kill memcached on all the nodes.
      5. Bucket is not able to load on any of the nodes and stuck in warmup forever.

      QE Test

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job5.ini -p bucket_storage=magma,rerun=false,GROUP=P2;kill,randomize_value=true,doc_size=256,bucket_eviction_policy=fullEviction,replica=0,nodes_init=4,enable_dp=false,collect_pcaps=True,get-cbcollect-info=True,autoCompactionDefined=true,vbuckets=256,nodes_init=3,replicas=2,bucket_ram_quota=256,num_items=25000000,multiplier=1,doc_size=1024,upgrade_version=7.2.0-1684 -t storage.magma.magma_crash_recovery.MagmaCrashTests.test_crash_before_multi_update_deletes,num_items=20000000,sdk_timeout=60,test_itr=2,graceful=False,update_itr=2,replicas=0,GROUP=P2;kill'
      

      Live cluster is available for debugging here: http://172.23.107.222:8091/ui/index.html#/buckets?commonBucket=default&scenarioZoom=minute&scenario=o31gzz6mc

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-8.0.0-1123 contains kv_engine commit 2622573 with commit message:
            Merge "MB-53118: Merge branch 'cheshire-cat' into neo" into neo

            build-team Couchbase Build Team added a comment - Build couchbase-server-8.0.0-1123 contains kv_engine commit 2622573 with commit message: Merge " MB-53118 : Merge branch 'cheshire-cat' into neo" into neo

            Build couchbase-server-8.0.0-1123 contains kv_engine commit 83fac44 with commit message:
            MB-53118: Merge branch 'cheshire-cat' into neo

            build-team Couchbase Build Team added a comment - Build couchbase-server-8.0.0-1123 contains kv_engine commit 83fac44 with commit message: MB-53118 : Merge branch 'cheshire-cat' into neo

            Build couchbase-server-8.0.0-1123 contains kv_engine commit 23f6d5c with commit message:
            MB-53118: Merge branch 'mad-hatter' into cheshire-cat

            build-team Couchbase Build Team added a comment - Build couchbase-server-8.0.0-1123 contains kv_engine commit 23f6d5c with commit message: MB-53118 : Merge branch 'mad-hatter' into cheshire-cat

            Build couchbase-server-8.0.0-1123 contains kv_engine commit ffa4bbb with commit message:
            MB-53118: Increase warmup_backfill_scan_chunk_duration to 100ms

            build-team Couchbase Build Team added a comment - Build couchbase-server-8.0.0-1123 contains kv_engine commit ffa4bbb with commit message: MB-53118 : Increase warmup_backfill_scan_chunk_duration to 100ms

            Validated using the automated magma test against Enterprise Edition 7.0.5 build 7639.

            Closing this ticket.

            ashwin.govindarajulu Ashwin Govindarajulu added a comment - Validated using the automated magma test against Enterprise Edition 7.0.5 build 7639. Closing this ticket.

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty