Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-21924

cbbackupmgr incorrectly restores tombstones as full documents



    • Untriaged
    • Yes


      cbbackupmgr restores tombstones as full documents. The documents are created with a empty value. This basically means that restoring from backup will cause data corruption and as such I have marked this defect as critical.

      Steps to reproduce

      Simple environment using our vagrants, setup two one node cluster each with the default bucket:

      Create and then delete a single document:

      [patrick:~] $ echo hello | cbc-create test -Ucouchbase://
      test                Stored. CAS=0x148dcfa15c3c0000
      [patrick:~] $ cbc-rm test -Ucouchbase://                                       
      test                Deleted. CAS=0x148dcfa656200000
      [patrick:~] $ cbc-cat test -Ucouchbase://                                      
      test                 CAS=0x148dcfa656200000, Flags=0x0. Size=0

      Create a backup:

      [patrick:~] $ /opt/couchbase/bin/cbbackupmgr config --archive ~/backup-new/ --repo cluster
      Backup repository `cluster` created successfully in archive `/home/vagrant/backup-new/`
      [patrick:~] $ /opt/couchbase/bin/cbbackupmgr backup --archive ~/backup-new/ --rep
      o cluster --host couchbase:// --username Administrator --password password
      Backing up to 2016-12-06T23_52_39.777541024Z
      Copied all data in 2.01s (Avg. 14.04KB/Sec)                                        1 items / 28.09KB
      default                 [==================================================================] 100.00%
      Backup successfully completed

      Restore the back up to the 2nd cluster:

      $ /opt/couchbase/bin/cbbackupmgr restore --archive ~/backup-new/ --re
      po cluster --host couchbase:// --username Administrator --password password
      (1/1) Restoring backup 2016-12-06T23_52_39.777541024Z
      Copied all data in 3s (Avg. 20B/Sec)                                                   1 items / 60B
      default                 [==================================================================] 100.00%
      Restore completed successfully

      The UI on the 2nd cluster will show an item count of 1:

      Get the "zombie" document from the second cluster and see that it's empty:

      [patrick:~] 1 $ cbc-cat test -Ucouchbase://                                    
      test                 CAS=0x148dcfa656200000, Flags=0x0. Size=0

      Root cause

      The same process works correctly using the old cbbackup tool. Looking at the tcpdump (which are attached) it looks like cbbackmgr is using a SetWithMeta operation for the tombstone where cbbackup is using the DeleteWithMeta operation:



      Matt Carabine reviewed the code and looks like the issue is with gocb the DeleteMeta function is incorrectly using the wrong opcode of cmdSetMeta. I have opened GOCBC-129 for this issue.


      1. Use cbbackup and cbrestore
      2. When restoring with cbbackupmgr the --force-updates argument can be used when restoring to a new/flushed buckets. This argument disables conflict so restoring to a bucket with items might be a problem.


      When restoring with cbbackupmgr deleted documents will appear as empty documents.


        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.



              arunkumar Arunkumar Senthilnathan (Inactive)
              pvarley Patrick Varley (Inactive)
              0 Vote for this issue
              16 Start watching this issue



                Gerrit Reviews

                  There are no open Gerrit changes