Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
5.0.0
-
5.0.0-3193
-
Untriaged
-
Unknown
Description
Sync Gateway is intermittently getting a Key Not Found error when trying to update the system xattr on a tombstone. We've got a test that attempts the following for a large number of documents:
1. Create docs via Sync Gateway (creates doc and xattr)
2. Update the docs via SDK
3. SG updates system xattr (import)
4. Delete the docs via SDK
5. SG updates system xattr (import)
Step 5 fails rarely for a single doc, possibly when the system is under load. (e.g. when running this flow for 2000 docs, one doc fails).
Step 5 is doing a multi-part subdoc operation to upsert the xattr and update the cas via macro expansion, like (gocb code excerpt):
_, err = bucket.MutateInEx(k, gocb.SubdocDocFlagAccessDeleted, gocb.Cas(cas), uint32(exp)).
UpsertEx("_sync", xv, gocb.SubdocFlagXattr). // Update the xattr
UpsertEx("_sync.cas", "${Mutation.CAS}", gocb.SubdocFlagXattr|gocb.SubdocFlagUseMacros). // Stamp the cas on the xattr
Execute()
This is successful for 99.9% percent of the docs in the test, but fails sporadically with a KeyNotFound. We've been through the SG code and are pretty confident there isn't a race condition going on in the SG code, and that the document does actually exist in the server at the time the request is made.
Additional details in this comment, including a link to a pcap for the entire test, and a pcap excerpt for the failing operation.
Given that this seems to only be reproducible under load, it feels a bit like the server is returning KeyNotFound instead of TMPFAIL, but we haven't been able to identify whether that's actually what's going on. SG has retry handling in place for TMPFAIL, but the KeyNotFound is causing SG to error out of the import.
Attachments
Issue Links
- mentioned in
-
Page Loading...
For Gerrit Dashboard: MB-25398 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
81423,2 | MB-25398: Allow store_deleted_with_CAS on non-resident items | master | kv_engine | Status: ABANDONED | -1 | -1 |
81479,3 | MB-25398: Allow store_deleted_with_CAS on non-resident items | spock | kv_engine | Status: MERGED | +2 | +1 |
81514,1 | Merge remote-tracking branch 'couchbase/spock' | master | kv_engine | Status: ABANDONED | 0 | 0 |
81544,1 | Merge remote-tracking branch 'couchbase/spock' | master | kv_engine | Status: MERGED | +2 | +1 |