Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-55533

[Magma-CDC] History start seq number gets toggled/history gets wiped off after the recovery

    XMLWordPrintable

Details

    Description

      Steps to repro:

      1. Create a three node cluster
      2. Create a magma buckets with (history_retention_collection_default=true,bucket_history_retention_seconds=0,bucket_history_retention_bytes=96636764160)
      3. Create forty nine collections(total collections including default are fifty)
      4. Create 100000 items in each of the non default collection.(doc size : 1024 bytes)
      5. Start new doc ops(expiry)
      6. While doc ops are going on keep crashing memcached (sleep between two SIGKILL is 60 to 90 seconds)
      7. Once above doc ops are finsihed recreate expired docs

      Logs:

       /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default vbucket-details 1 | grep -e history -e high_seqno
       vb_1:high_seqno:                          1151892
       vb_1:history_start_seqno:                 919518
       
      [root@localhost seqIndex]# cat state.798
      {"historyStartKey":[0,0,0,0,0,13,230,161,0,0,0,0,0,24,147,150,0],"levels":[{"tables":[]},{"tables":[]},{"tables":[{"attr":"479232","tabid":5329},{"attr":"188416","tabid":5330}]},{"tables":[]},{"tables":[{"attr":"4087808","tabid":5152},{"attr":"2174976","tabid":4566},{"attr":"2195456","tabid":4593},{"attr":"2150400","tabid":4616},{"attr":"2215936","tabid":4637},{"attr":"2359296","tabid":4660},{"attr":"2666496","tabid":4685},{"attr":"3080192","tabid":4710},{"attr":"3063808","tabid":4739},{"attr":"3313664","tabid":4757},{"attr":"3678208","tabid":4775},{"attr":"4014080","tabid":4807},{"attr":"6971392","tabid":4822},{"attr":"6434816","tabid":4828},{"attr":"5730304","tabid":4832},{"attr":"6250496","tabid":4850},{"attr":"6623232","tabid":4885},{"attr":"7405568","tabid":4933},{"attr":"7675904","tabid":4975},{"attr":"8380416","tabid":5015},{"attr":"9105408","tabid":5066},{"attr":"10539008","tabid":5089},{"attr":"11096064","tabid":5117},{"attr":"12210176","tabid":5142},{"attr":"14127104","tabid":5240},{"attr":"9420800","tabid":5310}]}],"maxSn":2083720,"meta":"{\"highSeqno\":1144171,\"historyEnabledSeqno\":1}","parent":0,"tableBytesAdded":22113996800,"tableBytesRemoved":20501745664,"type":1,"userStats":"","version":0,"walOffset":6049792,"walSegID":248}���/MUSKCEGC[root@localhost seqIndex]#
      

      QE-Test:

      git fetch https://review.couchbase.org/TAF refs/changes/62/186462/3 && git checkout -b change-186462 FETCH_HEAD
      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/temp_vol_3.ini -p bucket_storage=magma,bucket_ram_quota=256,init_loading=False,bucket_eviction_policy=fullEviction,rerun=False -t storage.magma.magma_crash_recovery.MagmaCrashTests.test_magma_failures_with_CDC,nodes_init=3,skip_cleanup=True,num_items=100000,doc_size=1024,batch_size=1000,sdk_timeout=60,log_level=info,infra_log_level=info,key_size=22,num_collections=50,ops_rate=20000,key_type=SimpleKey,vbuckets=16,replicas=2,doc_ops=expiry,test_itr=50,bucket_history_retention_seconds=0,bucket_history_retention_bytes=96636764160,standard_buckets=1,magma_buckets=1,num_scopes=1,induce_failures=True -m rest'
      

      Attachments

        For Gerrit Dashboard: MB-55533
        # Subject Branch Project Status CR V

        Activity

          People

            ankush.sharma Ankush Sharma
            ankush.sharma Ankush Sharma
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty