Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-56565

[CDC] Perpetual Backfilling loop, suspect eager cp removal. was: Rebalance hung seen in Graceful failover + Delta Recovery

    XMLWordPrintable

Details

    Description

      1. Create a 2 KV and 1 index/query node cluster.
      2. Create a magma bucket(replicas=1) and collections(total collection count including default collections is 51)
      3. Create 500000000 items sequentially(After creation of few thousands of documents set bucket_history_retention_seconds=259200,bucket_history_retention_bytes=12000000000000)
      4. Update 500000000 created in above step
      5. Create 500000000 items sequentially
      6. Update 500000000 created in above step
      7. Create five indexes Wait for index building.
      8. Rebalance in KV with Loading of docs.
      9. Rebalance Out KV with Loading of docs.(Rebalance completed successfully)
      10. Rebalance In_Out KV with Loading of docs. (Rebalance completed successfully)
      11. Swap Rebalance with Loading of docs (Rebalance completed successfully)
      12. Graceful Failover 1 node and RebalanceOut that node with loading in parallel (Rebalance completed successfully)
      13. Graceful Failover a node and FullRecovery that node with loading in parallel(Rebalance completed successfully)
      14. Graeful Failover a node and DeltaRecovery that node with loading in parallel
      15. Observed Rebalance was hung , After starting rebalance not even a single doc has moved (Wiated for close to one hour before reporting this)

      Note:

      Cluster is still in same state:
      http://172.23.110.64:8091/ui/index.html#/logs?commonBucket=GleamBookUsers0&scenarioZoom=minute&scenario=7udweu5h0

      QE-TEST:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/ankush_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.ClusterOpsVolume,nodes_init=2,graceful=True,skip_cleanup=True,num_items=50000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,bucket_type=membase,eviction_policy=fullEviction,iterations=5,batch_size=1000,sdk_timeout=60,log_level=info,infra_log_level=error,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=5,pc=10,index_nodes=1,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=30000,key_type=RandomKey -m rest'
      

      Attachments

        Activity

          People

            ankush.sharma Ankush Sharma
            ankush.sharma Ankush Sharma
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty