Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-21924

cbbackupmgr incorrectly restores tombstones as full documents

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 4.5.0, 4.5.1
    • 4.6.0
    • tools
    • Untriaged
    • Yes

    Description

      cbbackupmgr restores tombstones as full documents. The documents are created with a empty value. This basically means that restoring from backup will cause data corruption and as such I have marked this defect as critical.

      Steps to reproduce

      Simple environment using our vagrants, setup two one node cluster each with the default bucket:

      Create and then delete a single document:

      [patrick:~] $ echo hello | cbc-create test -Ucouchbase://10.111.151.101/default
      test                Stored. CAS=0x148dcfa15c3c0000
      [patrick:~] $ cbc-rm test -Ucouchbase://10.111.151.101/default                                       
      test                Deleted. CAS=0x148dcfa656200000
      [patrick:~] $ cbc-cat test -Ucouchbase://10.111.151.102/default                                      
      test                 CAS=0x148dcfa656200000, Flags=0x0. Size=0
      

      Create a backup:

      [patrick:~] $ /opt/couchbase/bin/cbbackupmgr config --archive ~/backup-new/ --repo cluster
      Backup repository `cluster` created successfully in archive `/home/vagrant/backup-new/`
      [patrick:~] $ /opt/couchbase/bin/cbbackupmgr backup --archive ~/backup-new/ --rep
      o cluster --host couchbase://127.0.0.1 --username Administrator --password password
       
      Backing up to 2016-12-06T23_52_39.777541024Z
      Copied all data in 2.01s (Avg. 14.04KB/Sec)                                        1 items / 28.09KB
      default                 [==================================================================] 100.00%
       
      Backup successfully completed
      

      Restore the back up to the 2nd cluster:

      $ /opt/couchbase/bin/cbbackupmgr restore --archive ~/backup-new/ --re
      po cluster --host couchbase://10.111.151.102 --username Administrator --password password
       
      (1/1) Restoring backup 2016-12-06T23_52_39.777541024Z
      Copied all data in 3s (Avg. 20B/Sec)                                                   1 items / 60B
      default                 [==================================================================] 100.00%
       
      Restore completed successfully
      

      The UI on the 2nd cluster will show an item count of 1:

      Get the "zombie" document from the second cluster and see that it's empty:

      [patrick:~] 1 $ cbc-cat test -Ucouchbase://10.111.151.102/default                                    
      test                 CAS=0x148dcfa656200000, Flags=0x0. Size=0
      

      Root cause

      The same process works correctly using the old cbbackup tool. Looking at the tcpdump (which are attached) it looks like cbbackmgr is using a SetWithMeta operation for the tombstone where cbbackup is using the DeleteWithMeta operation:

      cbbackupmgr

      cbbackup

      Matt Carabine reviewed the code and looks like the issue is with gocb the DeleteMeta function is incorrectly using the wrong opcode of cmdSetMeta. I have opened GOCBC-129 for this issue.

      Workarounds

      1. Use cbbackup and cbrestore
      2. When restoring with cbbackupmgr the --force-updates argument can be used when restoring to a new/flushed buckets. This argument disables conflict so restoring to a bucket with items might be a problem.

      Symptoms

      When restoring with cbbackupmgr deleted documents will appear as empty documents.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            No you were correct before. I just updated the manifest. Please retest it.

            mikew Mike Wiederhold [X] (Inactive) added a comment - No you were correct before. I just updated the manifest. Please retest it.

            Behaviour in 4.5.1-2855:

            1. Create a doc in source and delete it - back up source - restore in empty target - deleted document was restored - get failed on the document but metadata could be retrieved using views - meta data had "att_reason":"invalid_json" field set
            2. Create a doc in source and delete it - back up source - restore in target that has same key value doc but without force updates - document stays intact in target as expected
            3. Create a doc in source and delete it - back up source - restore in target that has same key value doc but with force updates - document deleted in target as expected

            Behaviour in 4.5.1-2856:

            1. Create a doc in source and delete it - back up source - restore in empty target - no document restored in target - get failed on the document with no key error and metadata could not be retrieved using views
            2. Create a doc in source and delete it - back up source - restore in target that has same key value doc but without force updates - document deleted in target even without using force-updates
            3. Create a doc in source and delete it - back up source - restore in target that has same key value doc but with force updates - document deleted in target as expected

            attaching screenshots

            Mike Wiederhold [X] please confirm if the behavior seen in 4.5.1-2856 is correct

            arunkumar Arunkumar Senthilnathan (Inactive) added a comment - - edited Behaviour in 4.5.1-2855: 1. Create a doc in source and delete it - back up source - restore in empty target - deleted document was restored - get failed on the document but metadata could be retrieved using views - meta data had "att_reason":"invalid_json" field set 2. Create a doc in source and delete it - back up source - restore in target that has same key value doc but without force updates - document stays intact in target as expected 3. Create a doc in source and delete it - back up source - restore in target that has same key value doc but with force updates - document deleted in target as expected Behaviour in 4.5.1-2856: 1. Create a doc in source and delete it - back up source - restore in empty target - no document restored in target - get failed on the document with no key error and metadata could not be retrieved using views 2. Create a doc in source and delete it - back up source - restore in target that has same key value doc but without force updates - document deleted in target even without using force-updates 3. Create a doc in source and delete it - back up source - restore in target that has same key value doc but with force updates - document deleted in target as expected attaching screenshots Mike Wiederhold [X] please confirm if the behavior seen in 4.5.1-2856 is correct

            1. You're comments are mostly correct. The only thing I want to point out is the comment "metadata could not be retrieved using views". When we do verification to see if a delete was properly handled we should be checking with the get meta command. I don't know how views handles deletes and it's possible that the behavior of views could change between releases. As a result I don't think this is a full proof way to check for deletes. I ran your first scenario on 4.5.1-2856 and checked the couchbase data files and the delete was properly persisted.

            2. This is the correct behavior. The delete probably won conflict resolution.

            3. Not needed for verification of this bug, but the behavior is correct.

            mikew Mike Wiederhold [X] (Inactive) added a comment - 1. You're comments are mostly correct. The only thing I want to point out is the comment "metadata could not be retrieved using views". When we do verification to see if a delete was properly handled we should be checking with the get meta command. I don't know how views handles deletes and it's possible that the behavior of views could change between releases. As a result I don't think this is a full proof way to check for deletes. I ran your first scenario on 4.5.1-2856 and checked the couchbase data files and the delete was properly persisted. 2. This is the correct behavior. The delete probably won conflict resolution. 3. Not needed for verification of this bug, but the behavior is correct.

            Verified in 4.5.1-2856 as per Mike Wiederhold [X]'s comment above:

            1. Create a doc in source and delete it - back up source - restore in empty target - no document restored in target - used get meta to retrive metadata - source: (1, 0, 1482437254, 4, 1482437255378763776) - target: (1, 0, 1482437305, 4, 1482437255378763776)
            2. Create a doc in source and delete it - back up source - restore in target that has same key value doc but without force updates - document deleted in target even without using force-updates
            3. Create a doc in source and delete it - back up source - restore in target that has same key value doc but with force updates - document deleted in target as expected

            closing defect

            arunkumar Arunkumar Senthilnathan (Inactive) added a comment - Verified in 4.5.1-2856 as per Mike Wiederhold [X] 's comment above: 1. Create a doc in source and delete it - back up source - restore in empty target - no document restored in target - used get meta to retrive metadata - source: (1, 0, 1482437254, 4, 1482437255378763776) - target: (1, 0, 1482437305, 4, 1482437255378763776) 2. Create a doc in source and delete it - back up source - restore in target that has same key value doc but without force updates - document deleted in target even without using force-updates 3. Create a doc in source and delete it - back up source - restore in target that has same key value doc but with force updates - document deleted in target as expected closing defect

            changing fix version back to 4.6. Quoting this as fixed in 4.5.1 (which refers to the initial GA rather than the maintenance patch) is potentially misleading

            dhaikney David Haikney added a comment - changing fix version back to 4.6. Quoting this as fixed in 4.5.1 (which refers to the initial GA rather than the maintenance patch) is potentially misleading

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              pvarley Patrick Varley
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty