Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43801

Rebalance In is stuck at 94% while bucket is in ~1% DGM.

    XMLWordPrintable

Details

    Description

      This is a must fix for CC(Magma-Beta) release.

      1. Create a 15 node cluster. Create a magma bucket with replica=2 and having 2GB/node memory quota.
      2. Start data loading until bucket reaches to 4% dgm.
      3. Reduce bucket quota to 1100MB/Node i.e. 17GB total.
      4. Bucket foes into ~1-2% DGM.
      5. Start Rebalance IN of a node.
      6. It is stuck at 94%.

      Test:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/test_job_magma.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t volumetests.Magma.volume.test_long_rebalance,nodes_init=15,replicas=2,skip_cleanup=True,num_items=100000000,num_buckets=1,bucket_names=GleamBook,doc_size=256,bucket_type=membase,compression_mode=off,iterations=10,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=info,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,maxttl=60,num_collections=50,doc_ops=create:update:delete:expiry,durability=None,ramQuota=1100,pc=1,dgm=2 -m rest'
      

      mem_used stays in between hwm-lwm:

      Attachments

        1. Bucket_Stats.png
          Bucket_Stats.png
          265 kB
        2. image-2021-01-21-18-15-37-266.png
          image-2021-01-21-18-15-37-266.png
          207 kB
        3. memory-quota-reduction.png
          memory-quota-reduction.png
          1.01 MB
        4. Servers.png
          Servers.png
          398 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty