Details
Description
It seems that cbbackup when running accumulative backup mode only backs up the difference since the last accumulative backup.
Based on the design for accumulative backup (https://github.com/couchbase/couchbase-cli/blob/master/docs/design/cbbackup-incr-func-spec.md#use-case-for-cumulative-incremental-backups), all changes since the last full backup should be backed up in the new accumulative backup.
However, based on empirical evidence and basic code review, this does not seem to be the case.
This fundamentally breaks the model of performing accumulative backups as users expect to be able to delete older accumulative backups once they have a new one, since that should contain all of the old changes as well as the new changes.
While restoring these incremental backups does seem to work okay (all docs get restored to the bucket), if you do delete intermediate accumulative backups then you will be missing documents.
Reproduction
This was verified on Couchbase Server 3.0.1 and Couchbase Server 4.6.5. Couchbase Server 5.x was attempted but there is another issue related to this preventing accumulative backups from working properly in this context (see MB-31517).
- Create the beer-sample bucket
- Create an 'accumulative' backup of the beer-sample bucket, this should just be a full backup as the archive does not exist yet
$ /opt/couchbase/bin/cbbackup http://localhost:8091 /backup -m accu -u Administrator -p password
[####################] 100.0% (7303/estimated 7303 msgs)
- Verify the number of items in the backup is 7303 (total docs in the sample bucket)
$ sqlite3 /backup/2018-10-04T145813Z/2018-10-04T145813Z-full/bucket-beer-sample/node-127.0.0.1%3A8091/data-0000.cbb 'SELECT count(key) FROM cbb_msg WHERE length(val) != 0'
7303
- Create a new document, test1 in the Couchbase Server UI (or however you'd like). There are now 7304 items in the bucket.
- Take an accumulative backup, re-using the same archive
$ /opt/couchbase/bin/cbbackup http://localhost:8091 /backup -m accu -u Administrator -p password
[ ] 0.0% (1/estimated 7304 msgs)
- Verify that this contains a single document as expected
$ sqlite3 /backup/2018-10-04T145813Z/2018-10-04T150110Z-accu/bucket-beer-sample/node-127.0.0.1%3A8091/data-0000.cbb 'SELECT count(key) FROM cbb_msg WHERE length(val) != 0'
1
- Create a new document in the same bucket, test2
- Perform another accumulative backup
$ /opt/couchbase/bin/cbbackup http://localhost:8091 /backup -m accu -u Administrator -p password
[ ] 0.0% (1/estimated 7305 msgs)
- Verify that this contains 2 documents as expected (since there's 2 new docs since the last full backup)
$ sqlite3 /backup/2018-10-04T145813Z/2018-10-04T150649Z-accu/bucket-beer-sample/node-127.0.0.1%3A8091/data-0000.cbb 'SELECT count(key) FROM cbb_msg WHERE length(val) != 0'
1
As you can see, the last accumulative backup only contains the 1 new doc since the last accumulative backup, rather than the 2 new docs since the full backup.