Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-58123

[Backport MB-57970 to 7.2.1] [7.2.1- Volume Test] - Indexer rebalance hung for ~ 5hrs

    XMLWordPrintable

Details

    Description

      Steps:

      1. Create a 3 KV and 2 index/query node cluster(redistributeIndexes is set to true).
      2. Create a magma bucket(replicas=1) and collections(total collection count including default collections is 51)
      3. Create 100000000 items sequentially(doc size -512 bytes, historyRetentionBytes=4000000000000, historyRetentionSeconds=259200)
      4. Update 100000000 created in above step
      5. Create 100000000 new items sequentially
      6. Update 100000000 created in above step
      7. Create 10 indexes Wait for index building.
      8. Rebalance in KV with Loading of docs. (Rebalance completed successfully)
      9. Rebalance Out KV with Loading of docs.(Rebalance completed successfully)
      10. Rebalance In indexing node with loading of docs(queries also running)
      11. Indexer rebalance was hung for more than 5 hours(It was stuck at 52.5 %, Tried twice and both the times it was stuck at same percentage(52.5).

      http://172.23.110.64:8091/settings/indexes
      		 
       
      		 
      {
      		 
        "redistributeIndexes": true,
      		 
        "numReplica": 0,
      		 
        "enablePageBloomFilter": false,
      		 
        "indexerThreads": 0,
      		 
        "memorySnapshotInterval": 200,
      		 
        "stableSnapshotInterval": 5000,
      		 
        "maxRollbackPoints": 2,
      		 
        "logLevel": "info",
      		 
        "storageMode": "plasma"
      		 
      }

      QE-TEST:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/ankush_temp_job3.ini bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.ClusterOpsVolume,nodes_init=3,graceful=True,skip_cleanup=True,num_items=100000000,num_buckets=1,bucket_names=GleamBook,doc_size=512,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=22,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=10,pc=10,indexer_mem_quota=204795,index_nodes=2,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=30000,key_type=RandomKey'

      Attachments

        Issue Links

          is a backport of

          Bug - A problem which impairs or prevents the functions of the product. MB-57970 [7.2.1- Volume Test] - Indexer rebalance hung for ~ 5hrs

          • Major - Major loss of function.
          • Closed
          is caused by

          Improvement - An improvement or enhancement to an existing feature or task. MB-57057 [Backport MB-56564 to 7.2.1] Build all indexes on new node before switching query traffic during rebalance

          • Major - Major loss of function.
          • Closed

          Activity

            People

              ankush.sharma Ankush Sharma
              amit.kulkarni Amit Kulkarni
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty