Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43460

[Magma]: Mis-match in active/replica num_items after rebalance + collection drop

    XMLWordPrintable

Details

    Description

      Steps:

      1. Create a 3 node cluster
      2. Create required buckets and collections(2 + 1 default).
      3. Create 1000000 items and Delete everything. checkout fragmentation
      4. Load 2000000 items, sequential keys. Total Items = 6M
      5. Update the first set of 10 percent (100000) items 1 times. Total Items = 6M
      6. Reverse Update last set of 10 percent (999999-900001) items 1 times. Total Items = 6M
      7. Create 1000000 random keys. Total Items = 9M
      8. Update all 1000000 random items 1 times
      9. Update all 1000000 random items 1 times
      10. Delete/Re-Create all 1000000 random items. Total Items = 9M
      11. Update 10 percent(100000) items 1 times and crash during recovery
      12. Update 10 percent(100000) items 1 times and crash during recovery
      13. Normal Rollback with creates
      14. Normal Rollback with updates
      15. Drop a collection. Total Items = 6M
      16. Normal Rollback with deletes. Stop persistence on active and delete items belongs to active vB(deleted items aren't persisted on active) so that they will get deleted from replica. Kill memcached on active so that delete items info is lost from memcached and active will trigger rollback on replica.

      In the last step, the number of Active and replica items expected are 6M.

      ssh root@172.23.97.38
      vb_replica_curr_items: 2002898
      vb_active_curr_items: 2006036

      ssh root@172.23.97.38
      vb_replica_curr_items: 2084164
      vb_active_curr_items: 1997158

      ssh root@172.23.97.39
      vb_replica_curr_items: 1998162
      vb_active_curr_items: 1996806

      Total Active = 6M
      Total Replica = 60,85,224 ====> Which is way more than expected items.

      QE Test:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P "args=-i /tmp/dp_magma_job.ini -p bucket_storage=${storage},bucket_eviction_policy=fullEviction,rerun=False -t volumetests.Magma.volume.SteadyStateVolume,nodes_init=3,replicas=1,skip_cleanup=True,num_items=1000000,num_buckets=1,bucket_names=GleamBook,doc_size=256,bucket_type=membase,compression_mode=off,iterations=1,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=info,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,step_iterations=1,rollback=True,durability=None,fragmentation=50,rollbacks=10,num_collections=2,pc=1,ramQuota=10240 -m rest"
      

      Attachments

        Issue Links

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty