Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
4.5.0, 4.5.1
-
Untriaged
-
Yes
Description
cbbackupmgr restores tombstones as full documents. The documents are created with a empty value. This basically means that restoring from backup will cause data corruption and as such I have marked this defect as critical.
Steps to reproduce
Simple environment using our vagrants, setup two one node cluster each with the default bucket:
Create and then delete a single document:
[patrick:~] $ echo hello | cbc-create test -Ucouchbase://10.111.151.101/default
|
test Stored. CAS=0x148dcfa15c3c0000
|
[patrick:~] $ cbc-rm test -Ucouchbase://10.111.151.101/default
|
test Deleted. CAS=0x148dcfa656200000
|
[patrick:~] $ cbc-cat test -Ucouchbase://10.111.151.102/default
|
test CAS=0x148dcfa656200000, Flags=0x0. Size=0
|
Create a backup:
[patrick:~] $ /opt/couchbase/bin/cbbackupmgr config --archive ~/backup-new/ --repo cluster
|
Backup repository `cluster` created successfully in archive `/home/vagrant/backup-new/`
|
[patrick:~] $ /opt/couchbase/bin/cbbackupmgr backup --archive ~/backup-new/ --rep
|
o cluster --host couchbase://127.0.0.1 --username Administrator --password password
|
|
|
Backing up to 2016-12-06T23_52_39.777541024Z
|
Copied all data in 2.01s (Avg. 14.04KB/Sec) 1 items / 28.09KB
|
default [==================================================================] 100.00%
|
|
|
Backup successfully completed
|
Restore the back up to the 2nd cluster:
$ /opt/couchbase/bin/cbbackupmgr restore --archive ~/backup-new/ --re
|
po cluster --host couchbase://10.111.151.102 --username Administrator --password password
|
|
|
(1/1) Restoring backup 2016-12-06T23_52_39.777541024Z
|
Copied all data in 3s (Avg. 20B/Sec) 1 items / 60B
|
default [==================================================================] 100.00%
|
|
|
Restore completed successfully
|
The UI on the 2nd cluster will show an item count of 1:
Get the "zombie" document from the second cluster and see that it's empty:
[patrick:~] 1 $ cbc-cat test -Ucouchbase://10.111.151.102/default
|
test CAS=0x148dcfa656200000, Flags=0x0. Size=0
|
Root cause
The same process works correctly using the old cbbackup tool. Looking at the tcpdump (which are attached) it looks like cbbackmgr is using a SetWithMeta operation for the tombstone where cbbackup is using the DeleteWithMeta operation:
cbbackupmgr
cbbackup
Matt Carabine reviewed the code and looks like the issue is with gocb the DeleteMeta function is incorrectly using the wrong opcode of cmdSetMeta. I have opened GOCBC-129 for this issue.
Workarounds
- Use cbbackup and cbrestore
- When restoring with cbbackupmgr the --force-updates argument can be used when restoring to a new/flushed buckets. This argument disables conflict so restoring to a bucket with items might be a problem.
Symptoms
When restoring with cbbackupmgr deleted documents will appear as empty documents.
