Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48569

[Magma, 30TB, 1% DGM]: Indexer drain rate is extremely slow

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Duplicate
    • 7.1.0
    • 7.1.0
    • couchbase-bucket
    • 7.1.0-1312

    Description

      1. Create a 3 kv node and 1 index/n1ql node cluster
      2. Create magma bucket, 100 collections under default scope
      3. Load 5B items and upsert them
      4. Load another 5B items and upsert them as well
      5. Create 100 indexes on 100 collections and build them. Start 100 QPS
      6. Rebalance In 1 KV node. Rebalance failed
      7. At this moment the indexer drain rate is very slow and building indexes is taking ages.
      If we look at idx45, the remaining mutations are 10M while the drain rate is ~500/s

      For today, the cluster is live at: http://172.23.110.64:8091/ui/index.htm

      Note: The total drain rate across indexes is around 10k - 75k/s on a 72 core machine, ssd disk, 200GB indexer RAM
      Total Disk Writes are varying between 5-30Mbps

      QE Test

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.test_rebalance,nodes_init=3,graceful=True,skip_cleanup=True,num_items=100000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=1,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest'
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            No work has yet been logged on this issue.

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty