Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-22043

CLONE - cbbackupmgr incorrectly restores tombstones as full documents

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 4.6.0
    • 4.5.0, 4.5.1
    • tools
    • Untriaged
    • Yes

    Description

      cbbackupmgr restores tombstones as full documents. The documents are created with a empty value. This basically means that restoring from backup will cause data corruption and as such I have marked this defect as critical.

      Steps to reproduce

      Simple environment using our vagrants, setup two one node cluster each with the default bucket:

      Create and then delete a single document:

      [patrick:~] $ echo hello | cbc-create test -Ucouchbase://10.111.151.101/default
      test                Stored. CAS=0x148dcfa15c3c0000
      [patrick:~] $ cbc-rm test -Ucouchbase://10.111.151.101/default                                       
      test                Deleted. CAS=0x148dcfa656200000
      [patrick:~] $ cbc-cat test -Ucouchbase://10.111.151.102/default                                      
      test                 CAS=0x148dcfa656200000, Flags=0x0. Size=0
      

      Create a backup:

      [patrick:~] $ /opt/couchbase/bin/cbbackupmgr config --archive ~/backup-new/ --repo cluster
      Backup repository `cluster` created successfully in archive `/home/vagrant/backup-new/`
      [patrick:~] $ /opt/couchbase/bin/cbbackupmgr backup --archive ~/backup-new/ --rep
      o cluster --host couchbase://127.0.0.1 --username Administrator --password password
       
      Backing up to 2016-12-06T23_52_39.777541024Z
      Copied all data in 2.01s (Avg. 14.04KB/Sec)                                        1 items / 28.09KB
      default                 [==================================================================] 100.00%
       
      Backup successfully completed
      

      Restore the back up to the 2nd cluster:

      $ /opt/couchbase/bin/cbbackupmgr restore --archive ~/backup-new/ --re
      po cluster --host couchbase://10.111.151.102 --username Administrator --password password
       
      (1/1) Restoring backup 2016-12-06T23_52_39.777541024Z
      Copied all data in 3s (Avg. 20B/Sec)                                                   1 items / 60B
      default                 [==================================================================] 100.00%
       
      Restore completed successfully
      

      The UI on the 2nd cluster will show an item count of 1:

      Get the "zombie" document from the second cluster and see that it's empty:

      [patrick:~] 1 $ cbc-cat test -Ucouchbase://10.111.151.102/default                                    
      test                 CAS=0x148dcfa656200000, Flags=0x0. Size=0
      

      Root cause

      The same process works correctly using the old cbbackup tool. Looking at the tcpdump (which are attached) it looks like cbbackmgr is using a SetWithMeta operation for the tombstone where cbbackup is using the DeleteWithMeta operation:

      cbbackupmgr

      cbbackup

      Matt Carabine reviewed the code and looks like the issue is with gocb the DeleteMeta function is incorrectly using the wrong opcode of cmdSetMeta. I have opened GOCBC-129 for this issue.

      Workarounds

      1. Use cbbackup and cbrestore
      2. When restoring with cbbackupmgr the --force-updates argument can be used when restoring to a new/flushed buckets. This argument disables conflict so restoring to a bucket with items might be a problem.

      Symptoms

      When restoring with cbbackupmgr deleted documents will appear as empty documents.

      Attachments

        Issue Links

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              pvarley Patrick Varley (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty