[Durability]: Seeing expiry docs not getting expired

Build: 6.6.0-7883

Scenario: (All doc_ops with doc_durability=MAJORITY)

• 2 node cluster, Couchbase bucket (replica=1)
• Load 250K docs into the bucket
• Rebalance-in 2 more nodes into the cluster. (4 node cluster with 250K docs)
• Start following doc update ops in a thread:
  • Update with expiry=0 for doc indexes 0-125K
  • Update with expiry=10 for doc indexes 127K-250K
• Rebalance-out one node from the cluster and rebalance-in the same node back
• Wait for doc_updates to complete and then sleep 12 secs for docs to expire
• Validate doc count=127K

Observation:

At the end of the above scenario, the doc_count is ~ 128K (Expected 127K)

Test log: http://qa.sc.couchbase.com/job/oel6-4node-rebalance_in_jython/1107/consoleText

SDK Client side pcap:https://cb-jira.s3.us-east-2.amazonaws.com/logs/expiry_docs/client.pcap

Cluster nodes pcaps: http://qa.sc.couchbase.com/job/oel6-4node-rebalance_in_jython/1107/artifact/logs/testrunner-20-Jul-21_02-01-13/test_1/