Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-56010

CDC: Inconsistency between active / replica's history_start_seqno

    XMLWordPrintable

Details

    Description

      Build: 7.2.0-5251

      Scenario:

      • 5 node kv cluster
      • Create magma bucket with CDC enabled 
        {magma_key_tree_data_blocksize,4096},
        {magma_seq_tree_data_blocksize,4096},
        {history_retention_seconds,180},
        {history_retention_bytes,4000000000},
        {history_retention_collection_default,true}]

       

      • Start initial data + historical data using regular updates on same documents
      • Rebalance out one node from the cluster 
        -----------------------------------------
        | Nodes          | CPU           | Status       |
        -----------------------------------------
        | 172.23.105.212 | 3.960484191   | Cluster node |
        | 172.23.105.244 | 1.58539445053 | — OUT ---> |
        | 172.23.105.155 | 4.18180756272 | Cluster node |
        | 172.23.105.213 | 3.85192113071 | Cluster node |
        | 172.23.105.211 | 3.25121870299 | Cluster node |
        -----------------------------------------

      Observation:

      For few vbuckets, history_start seqno is moving back to zero

      Example: vbucket 127 on node .155 (active) and .213 (replica)

      On node: 172.23.195.155 (Active)
      		 
      vb_127:history_start_seqno:                  39281
      		 
       
      		 
      On node: 172.23.105.213 (Replica)
      		 
      vb_127:history_start_seqno:                 1

      TAF test:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i node.ini rerun=False,skip_cluster_reset=True,get-cbcollect-info=False,num_items=10000,upgrade_version=7.2.0-5251 -t bucket_collections.history_retention.DocHistoryRetention.test_rebalance_with_dedupe,nodes_init=5,bucket_spec=single_bucket.history_retention_tests,replicas=1,num_items=100000,bucket_history_retention_seconds=180,bucket_history_retention_bytes=4000000000,nodes_out=1'

       

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-56009 CDC: history_start_seqno got reset to zero

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          For Gerrit Dashboard: MB-56010
          # Subject Branch Project Status CR V
          187874,3 MB-56010 magma: Retain atleast one sstable in history neo magma Status: MERGED +2 +1

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty