Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29861

Tombstoned documents missing after server upgrade

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 5.5.0
    • 5.5.0
    • couchbase-bucket
    • None
    • Triaged
    • Yes

    Description

      During Sync Gateway upgrade testing (with convergence/xattrs enabled), some deleted documents aren't found post-upgrade.

      The (failing) test has a lot of moving parts, so I'm not 100% sure this is a server issue, but haven't been able to identify any issues on the SG/mobile side, and would like some assistance reviewing the server logs to see if there's anything suspicious.

      The test does the following:

      1. Writes 10K docs to a 2 node server cluster (5.1.1-5721) via 2 Sync Gateway nodes (2.0.0-832) running w/ convergence enabled.
      2. Starts a thread to update the 10K docs.  Performs 9K updates, and 1K mobile deletes.  Mobile delete results in a server tombstone w/ sync xattr intact. This thread is running while steps 3 and 4 are completed
      3. Upgrade Sync Gateway nodes to 2.1.0-95.
      4. Upgrade server to 5.5.0-2764.
      5. Verify that the 9K updates completed successfully
      6. Verify that the 1K mobile deletes completed successfully.

      Step 6 is failing - 704 of the tombstones are present, but 296 are missing.

      Based on the Sync Gateway logs, the tombstones are all created as expected.  However, when we subsequently attempt to retrieve from Couchbase Server, they aren't found. 

      To reiterate - the missing documents will be Couchbase Server tombstones w/ associated '_sync' system xattrs.  After upgrade, the tombstones don't appear to be present.

      The cbcollect for the server logs are here:

      [s3://cb-customers-secure/raghu-sg-upgrade/collectinfo-2018-05-26t050743-ns_1@192.168.33.20.zip|s3://AKIAIYNLVHFWDSSGDORQ@cb-customers-secure/raghu-sg-upgrade/collectinfo-2018-05-26t050743-ns_1@192.168.33.20.zip]

      [s3://cb-customers-secure/raghu-sg-upgrade/collectinfo-2018-05-26t050743-ns_1@192.168.33.21.zip|s3://AKIAIYNLVHFWDSSGDORQ@cb-customers-secure/raghu-sg-upgrade/collectinfo-2018-05-26t050743-ns_1@192.168.33.21.zip]

      A few of the keys that are missing post-upgrade:
      ls_db_upgrade_doc_0_3
      ls_db_upgrade_doc_0_688
      ls_db_upgrade_doc_0_941
      ls_db_upgrade_doc_0_971

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              raghu.sarangapani Raghu Sarangapani (Inactive)
              adamf Adam Fraser
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty