Resolution: Not a Bug
1. Create a 2 node cluster:
2. Create bucket:
3. Loaded 100k(test_docs-0:test_docs-99999) docs with durability=majority
4. Change bucket replica to 2, add 10.112.180.103, remove 10.112.180.102, hit rebalance. Load another 100k(test_docs-100000:test_docs-199999) in parallel
5. Kill memcahced on 10.112.180.101 when rebalance reaches ~40%. Rebalance failed(Intentionally)
Data loading is still in progress with expected exceptions.
6. Restart rebalance. Wait for rebalance finish and it finished properly.
7. Wait for data loading to finish and retry of all the catch exceptions succeeds.
8. Validate the data
Data validation failed as few keys are missing from which there was success for sync-write
Missing keys: ['test_docs-130287', 'test_docs-130289', 'test_docs-130282', 'test_docs-130294', 'test_docs-130291']
All the data should be present as all the exceptions were watched and re-inserted.
In the attached pcap, apply the filter as: couchbase.opaque == 0xe3080000 and see packet number 619311 which is an insert request for key: test_docs-130287. Packet number 619329 is the success response for it.
But the key is missing from the bucket.
Note: Pcap is quite big, please apply the filters. I tried to save the filtered packets through wireshark but some issue is coming while doing that so couldn't do it.