Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48569

[Magma, 30TB, 1% DGM]: Indexer drain rate is extremely slow

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Duplicate
    • 7.1.0
    • 7.1.0
    • couchbase-bucket
    • 7.1.0-1312

    Description

      1. Create a 3 kv node and 1 index/n1ql node cluster
      2. Create magma bucket, 100 collections under default scope
      3. Load 5B items and upsert them
      4. Load another 5B items and upsert them as well
      5. Create 100 indexes on 100 collections and build them. Start 100 QPS
      6. Rebalance In 1 KV node. Rebalance failed
      7. At this moment the indexer drain rate is very slow and building indexes is taking ages.
      If we look at idx45, the remaining mutations are 10M while the drain rate is ~500/s

      For today, the cluster is live at: http://172.23.110.64:8091/ui/index.htm

      Note: The total drain rate across indexes is around 10k - 75k/s on a 72 core machine, ssd disk, 200GB indexer RAM
      Total Disk Writes are varying between 5-30Mbps

      QE Test

      		 
      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.test_rebalance,nodes_init=3,graceful=True,skip_cleanup=True,num_items=100000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=1,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest'

      Attachments

        Issue Links

          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. MB-48834 Improve XDCR performance with Magma

          • Major - Major loss of function.
          • Open
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-49685 Seqno-ordered collection filtered backfills monopolise task/thread

          • Major - Major loss of function.
          • Open
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Ascending order - Click to sort in descending order
            ritesh.agarwal Ritesh Agarwal created issue -
            ritesh.agarwal Ritesh Agarwal made changes -
            Field Original Value New Value
            Attachment idx45.png [ 160871 ]
            ritesh.agarwal Ritesh Agarwal made changes -
            Description 1. Create a 3 kv node and 1 index/n1ql node cluster
            2. Create magma bucket, 100 collections under default scope
            3. Load 5B items and upsert them
            4. Load another 5B items and upsert them as well
            5. Create 100 indexes on 100 collections and build them. Start 100 QPS
            6. Rebalance In 1 KV node. Rebalance failed
            7. At this moment the indexer drain rate is very slow and building indexes is taking ages.

            For today, the cluster is live at: http://172.23.110.64:8091/ui/index.htm             		1. Create a 3 kv node and 1 index/n1ql node cluster
            2. Create magma bucket, 100 collections under default scope
            3. Load 5B items and upsert them
            4. Load another 5B items and upsert them as well
            5. Create 100 indexes on 100 collections and build them. Start 100 QPS
            6. Rebalance In 1 KV node. Rebalance failed
            7. At this moment the indexer drain rate is very slow and building indexes is taking ages.
            If we look at idx45, the remaining mutations are 10M while the drain rate is ~500/s
             !idx45.png|thumbnail!

            For today, the cluster is live at: http://172.23.110.64:8091/ui/index.htm
            ritesh.agarwal Ritesh Agarwal made changes -
            Description 1. Create a 3 kv node and 1 index/n1ql node cluster
            2. Create magma bucket, 100 collections under default scope
            3. Load 5B items and upsert them
            4. Load another 5B items and upsert them as well
            5. Create 100 indexes on 100 collections and build them. Start 100 QPS
            6. Rebalance In 1 KV node. Rebalance failed
            7. At this moment the indexer drain rate is very slow and building indexes is taking ages.
            If we look at idx45, the remaining mutations are 10M while the drain rate is ~500/s
             !idx45.png|thumbnail!

            For today, the cluster is live at: http://172.23.110.64:8091/ui/index.htm             		1. Create a 3 kv node and 1 index/n1ql node cluster
            2. Create magma bucket, 100 collections under default scope
            3. Load 5B items and upsert them
            4. Load another 5B items and upsert them as well
            5. Create 100 indexes on 100 collections and build them. Start 100 QPS
            6. Rebalance In 1 KV node. Rebalance failed
            7. At this moment the indexer drain rate is very slow and building indexes is taking ages.
            If we look at idx45, the remaining mutations are 10M while the drain rate is ~500/s
             !idx45.png|thumbnail!

            For today, the cluster is live at: http://172.23.110.64:8091/ui/index.htm

            *Note: The total drain rate across is around 75k/s on a 72 core machine, ssd disk, 200GB indexer RAM*
            ritesh.agarwal Ritesh Agarwal made changes -
            Description 1. Create a 3 kv node and 1 index/n1ql node cluster
            2. Create magma bucket, 100 collections under default scope
            3. Load 5B items and upsert them
            4. Load another 5B items and upsert them as well
            5. Create 100 indexes on 100 collections and build them. Start 100 QPS
            6. Rebalance In 1 KV node. Rebalance failed
            7. At this moment the indexer drain rate is very slow and building indexes is taking ages.
            If we look at idx45, the remaining mutations are 10M while the drain rate is ~500/s
             !idx45.png|thumbnail!

            For today, the cluster is live at: http://172.23.110.64:8091/ui/index.htm

            *Note: The total drain rate across is around 75k/s on a 72 core machine, ssd disk, 200GB indexer RAM*             		1. Create a 3 kv node and 1 index/n1ql node cluster
            2. Create magma bucket, 100 collections under default scope
            3. Load 5B items and upsert them
            4. Load another 5B items and upsert them as well
            5. Create 100 indexes on 100 collections and build them. Start 100 QPS
            6. Rebalance In 1 KV node. Rebalance failed
            7. At this moment the indexer drain rate is very slow and building indexes is taking ages.
            If we look at idx45, the remaining mutations are 10M while the drain rate is ~500/s
             !idx45.png|thumbnail!

            For today, the cluster is live at: http://172.23.110.64:8091/ui/index.htm

            *Note: The total drain rate across indexes is around 75k/s on a 72 core machine, ssd disk, 200GB indexer RAM*
            ritesh.agarwal Ritesh Agarwal made changes -
            Description 1. Create a 3 kv node and 1 index/n1ql node cluster
            2. Create magma bucket, 100 collections under default scope
            3. Load 5B items and upsert them
            4. Load another 5B items and upsert them as well
            5. Create 100 indexes on 100 collections and build them. Start 100 QPS
            6. Rebalance In 1 KV node. Rebalance failed
            7. At this moment the indexer drain rate is very slow and building indexes is taking ages.
            If we look at idx45, the remaining mutations are 10M while the drain rate is ~500/s
             !idx45.png|thumbnail!

            For today, the cluster is live at: http://172.23.110.64:8091/ui/index.htm

            *Note: The total drain rate across indexes is around 75k/s on a 72 core machine, ssd disk, 200GB indexer RAM*             		1. Create a 3 kv node and 1 index/n1ql node cluster
            2. Create magma bucket, 100 collections under default scope
            3. Load 5B items and upsert them
            4. Load another 5B items and upsert them as well
            5. Create 100 indexes on 100 collections and build them. Start 100 QPS
            6. Rebalance In 1 KV node. Rebalance failed
            7. At this moment the indexer drain rate is very slow and building indexes is taking ages.
            If we look at idx45, the remaining mutations are 10M while the drain rate is ~500/s
             !idx45.png|thumbnail!

            For today, the cluster is live at: http://172.23.110.64:8091/ui/index.htm

            *Note: The total drain rate across indexes is around 75k/s on a 72 core machine, ssd disk, 200GB indexer RAM*
            Total Disk Writes are varying between 5-30Mbps
            ritesh.agarwal Ritesh Agarwal made changes -
            Description 1. Create a 3 kv node and 1 index/n1ql node cluster
            2. Create magma bucket, 100 collections under default scope
            3. Load 5B items and upsert them
            4. Load another 5B items and upsert them as well
            5. Create 100 indexes on 100 collections and build them. Start 100 QPS
            6. Rebalance In 1 KV node. Rebalance failed
            7. At this moment the indexer drain rate is very slow and building indexes is taking ages.
            If we look at idx45, the remaining mutations are 10M while the drain rate is ~500/s
             !idx45.png|thumbnail!

            For today, the cluster is live at: http://172.23.110.64:8091/ui/index.htm

            *Note: The total drain rate across indexes is around 75k/s on a 72 core machine, ssd disk, 200GB indexer RAM*
            Total Disk Writes are varying between 5-30Mbps             		1. Create a 3 kv node and 1 index/n1ql node cluster
            2. Create magma bucket, 100 collections under default scope
            3. Load 5B items and upsert them
            4. Load another 5B items and upsert them as well
            5. Create 100 indexes on 100 collections and build them. Start 100 QPS
            6. Rebalance In 1 KV node. Rebalance failed
            7. At this moment the indexer drain rate is very slow and building indexes is taking ages.
            If we look at idx45, the remaining mutations are 10M while the drain rate is ~500/s
             !idx45.png|thumbnail!

            For today, the cluster is live at: http://172.23.110.64:8091/ui/index.htm

            *Note: The total drain rate across indexes is around 10k - 75k/s on a 72 core machine, ssd disk, 200GB indexer RAM*
            Total Disk Writes are varying between 5-30Mbps
            ritesh.agarwal Ritesh Agarwal made changes -
            Description 1. Create a 3 kv node and 1 index/n1ql node cluster
            2. Create magma bucket, 100 collections under default scope
            3. Load 5B items and upsert them
            4. Load another 5B items and upsert them as well
            5. Create 100 indexes on 100 collections and build them. Start 100 QPS
            6. Rebalance In 1 KV node. Rebalance failed
            7. At this moment the indexer drain rate is very slow and building indexes is taking ages.
            If we look at idx45, the remaining mutations are 10M while the drain rate is ~500/s
             !idx45.png|thumbnail!

            For today, the cluster is live at: http://172.23.110.64:8091/ui/index.htm

            *Note: The total drain rate across indexes is around 10k - 75k/s on a 72 core machine, ssd disk, 200GB indexer RAM*
            Total Disk Writes are varying between 5-30Mbps             		1. Create a 3 kv node and 1 index/n1ql node cluster
            2. Create magma bucket, 100 collections under default scope
            3. Load 5B items and upsert them
            4. Load another 5B items and upsert them as well
            5. Create 100 indexes on 100 collections and build them. Start 100 QPS
            6. Rebalance In 1 KV node. Rebalance failed
            7. At this moment the indexer drain rate is very slow and building indexes is taking ages.
            If we look at idx45, the remaining mutations are 10M while the drain rate is ~500/s
             !idx45.png|thumbnail!

            For today, the cluster is live at: http://172.23.110.64:8091/ui/index.htm

            *Note: The total drain rate across indexes is around 10k - 75k/s on a 72 core machine, ssd disk, 200GB indexer RAM*
            Total Disk Writes are varying between 5-30Mbps

            {code:title=QE Test}
            guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.test_rebalance,nodes_init=3,graceful=True,skip_cleanup=True,num_items=100000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=1,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest'
            {code}
            ritesh.agarwal Ritesh Agarwal made changes -
            Labels volume-test magma volume-test
            jeelan.poola Jeelan Poola made changes -
            Assignee Jeelan Poola [ jeelan.poola ] Varun Velamuri [ varun.velamuri ]
            ritam.sharma Ritam Sharma made changes -
            Labels magma volume-test affects-neo-testing magma volume-test
            mihir.kamdar Mihir Kamdar (Inactive) made changes -
            Component/s secondary-index [ 11211 ]
            Component/s indexing [ 15110 ]
            varun.velamuri Varun Velamuri made changes -
            Component/s couchbase-bucket [ 10173 ]
            Component/s secondary-index [ 11211 ]
            varun.velamuri Varun Velamuri made changes -
            Assignee Varun Velamuri [ varun.velamuri ] Daniel Owen [ owend ]
            owend Daniel Owen made changes -
            Epic Link MB-30659 [ 88207 ]
            owend Daniel Owen made changes -
            Rank Ranked higher
            ritesh.agarwal Ritesh Agarwal made changes -
            Labels affects-neo-testing magma volume-test magma volume-test
            drigby Dave Rigby made changes -
            Assignee Daniel Owen [ owend ] Dave Rigby [ drigby ]
            drigby Dave Rigby made changes -
            Assignee Dave Rigby [ drigby ] Daniel Owen [ owend ]
            james.harrison James Harrison made changes -
            Assignee Daniel Owen [ owend ] James Harrison [ james.harrison ]
            james.harrison James Harrison made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            owend Daniel Owen made changes -
            Sprint KV 2021-Nov [ 1866 ]
            owend Daniel Owen made changes -
            Rank Ranked lower
            james.harrison James Harrison made changes -
            Link This issue relates to MB-49685 [ MB-49685 ]
            owend Daniel Owen made changes -
            Rank Ranked higher
            owend Daniel Owen made changes -
            Rank Ranked lower
            owend Daniel Owen made changes -
            Sprint KV 2021-Nov [ 1866 ] KV 2021-Nov, Magma 2021-Sep-21 [ 1866, 1906 ]
            james.harrison James Harrison made changes -
            Assignee James Harrison [ james.harrison ] Ritesh Agarwal [ ritesh.agarwal ]
            Resolution Duplicate [ 3 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            james.harrison James Harrison made changes -
            Link This issue duplicates MB-48834 [ MB-48834 ]
            ritam.sharma Ritam Sharma made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty