Details
-
Bug
-
Resolution: Fixed
-
Critical
-
5.5.0
-
None
-
Triaged
-
Yes
Description
During Sync Gateway upgrade testing (with convergence/xattrs enabled), some deleted documents aren't found post-upgrade.
The (failing) test has a lot of moving parts, so I'm not 100% sure this is a server issue, but haven't been able to identify any issues on the SG/mobile side, and would like some assistance reviewing the server logs to see if there's anything suspicious.
The test does the following:
- Writes 10K docs to a 2 node server cluster (5.1.1-5721) via 2 Sync Gateway nodes (2.0.0-832) running w/ convergence enabled.
- Starts a thread to update the 10K docs. Performs 9K updates, and 1K mobile deletes. Mobile delete results in a server tombstone w/ sync xattr intact. This thread is running while steps 3 and 4 are completed
- Upgrade Sync Gateway nodes to 2.1.0-95.
- Upgrade server to 5.5.0-2764.
- Verify that the 9K updates completed successfully
- Verify that the 1K mobile deletes completed successfully.
Step 6 is failing - 704 of the tombstones are present, but 296 are missing.
Based on the Sync Gateway logs, the tombstones are all created as expected. However, when we subsequently attempt to retrieve from Couchbase Server, they aren't found.
To reiterate - the missing documents will be Couchbase Server tombstones w/ associated '_sync' system xattrs. After upgrade, the tombstones don't appear to be present.
The cbcollect for the server logs are here:
[s3://cb-customers-secure/raghu-sg-upgrade/collectinfo-2018-05-26t050743-ns_1@192.168.33.20.zip|s3://AKIAIYNLVHFWDSSGDORQ@cb-customers-secure/raghu-sg-upgrade/collectinfo-2018-05-26t050743-ns_1@192.168.33.20.zip]
[s3://cb-customers-secure/raghu-sg-upgrade/collectinfo-2018-05-26t050743-ns_1@192.168.33.21.zip|s3://AKIAIYNLVHFWDSSGDORQ@cb-customers-secure/raghu-sg-upgrade/collectinfo-2018-05-26t050743-ns_1@192.168.33.21.zip]
A few of the keys that are missing post-upgrade:
ls_db_upgrade_doc_0_3
ls_db_upgrade_doc_0_688
ls_db_upgrade_doc_0_941
ls_db_upgrade_doc_0_971