CDC: history_start_seqno got reset to zero

Build: 7.2.0-5251

Scenario:

• 5 node kv cluster
• Create magma bucket with CDC enabled

  {magma_key_tree_data_blocksize,4096},

  {magma_seq_tree_data_blocksize,4096},

  {history_retention_seconds,180},

  {history_retention_bytes,4000000000},

  {history_retention_collection_default,true}]

• Start initial data + historical data using regular updates on same documents
• Rebalance out one node from the cluster

  +----------------+---------------+--------------+

  | Nodes          | CPU           | Status       |

  +----------------+---------------+--------------+

  | 172.23.105.212 | 3.960484191   | Cluster node |

  | 172.23.105.244 | 1.58539445053 | --- OUT ---> |

  | 172.23.105.155 | 4.18180756272 | Cluster node |

  | 172.23.105.213 | 3.85192113071 | Cluster node |

  | 172.23.105.211 | 3.25121870299 | Cluster node |

  +----------------+---------------+--------------+

Observation:

For few vbuckets, history_start seqno is moving back to zero

Example: vbucket 115 on node .155

# /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 vbucket-details 115 | grep hist

 vb_115:history_start_seqno:                 0

# /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b bucket-0 vbucket | grep 115

 vb_115:  active

TAF test:

guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i node.ini rerun=False,skip_cluster_reset=True,get-cbcollect-info=False,num_items=10000,upgrade_version=7.2.0-5251 -t bucket_collections.history_retention.DocHistoryRetention.test_rebalance_with_dedupe,nodes_init=5,bucket_spec=single_bucket.history_retention_tests,replicas=1,num_items=100000,bucket_history_retention_seconds=180,bucket_history_retention_bytes=4000000000,nodes_out=1'