Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Critical
-
Resolution: Done
-
Affects Version/s: 6.5.0, 6.5.1, 6.6.0
-
Fix Version/s: 6.6.1
-
Component/s: couchbase-bucket, tools, XDCR
-
Labels:
-
Triage:Untriaged
-
Story Points:1
-
Is this a Regression?:Unknown
Description
TL;DR:
In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail:
Affected versions
The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to.
| Backup (rows) \ Restore (columns) Versions | <=6.0.4 | 6.5.0 | 6.5.1 | 6.6.0 |
|---|---|---|---|---|
| <=6.0.4 |  |
 |
|
* |
| 6.5.0 | N/A | |
|
* |
| 6.5.1 | N/A | N/A | |
* |
| 6.6.0 | N/A | N/A | N/A | * |
* When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below.
—
Problem
cbbackupmgr restore fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.
Notes
I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.
Interestingly on 6.6.0 the following config is set:
|
6.6.0 |
# /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
|
ep_allow_del_with_meta_prune_user_data: false
|
There does not seem to be away to see what this is set to on old versions.
Steps to reproduce
1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
/opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
|
subdoc> set test value -x xattr=100 -e 10
|
2. Wait 10 seconds
3. Take a backup
/opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
|
/opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
|
4. From the examine command we can see that the tombstone contains the xattr and a value:
/opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
|
Key: test
|
SeqNo: 2
|
Backup: 2020-10-29T15_05_45.745791997Z
|
Deleted: true
|
Size: 4B (key), 29B (meta), 3B (value)
|
Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
|
Xattrs: {"xattr":100}
|
Value: value
|
5. Restore the backup to Couchbase Server 6.6.0
/opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
|
(1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
|
Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
|
[===============================================================================================================================================================================================================================================================================] 100.00%
|
Error restoring cluster: invalid argument
|
Restore bucket 'test' failed
|
Mutations restored: 0, Mutations failed to restore: 0
|
Deletions restored: 0, Deletions failed to restore: 1
|
Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
|
|
memcached.log |
2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
|
2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
|
The restore on 6.5.1 and 6.5.0 also fails:
|
6.5.1 Restore |
[root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
|
rd
|
(1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
|
Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
|
[==================================================================================================================================] 100.00%
|
Error restoring cluster: invalid argument
|
Restore bucket 'test' failed
|
Mutations restored: 0, Mutations failed to restore: 0
|
Deletions restored: 0, Deletions failed to restore: 1
|
Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
|
|
6.5.0 Restore |
# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
|
(1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
|
Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
|
[==================================================================================================================================] 100.00%
|
Error restoring cluster: invalid argument
|
Restore bucket 'test' failed
|
Mutations restored: 0, Mutations failed to restore: 0
|
Deletions restored: 0, Deletions failed to restore: 1
|
Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
|
|
—
Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.
The restore on 6.0.4:
|
6.0.4 Restore |
/opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
|
rd
|
(1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
|
Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
|
[==================================================================================================================================] 100.00%
|
Restore bucket 'test' succeeded
|
Mutations restored: 0, Mutations failed to restore: 0
|
Deletions restored: 1, Deletions failed to restore: 0
|
Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
|
Restore completed successfully
|
—
The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:
|
6.0.4 goxdcr.log |
2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
|
Workaround
There is a work around but it only works for 6.6.0:
|
cbepctl workaround |
# /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
|
setting param: allow_del_with_meta_prune_user_data true
|
set allow_del_with_meta_prune_user_data to true
|
This will allow the restore to work:
|
6.6.0 restore with workaround |
# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
|
(1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
|
Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
|
[==================================================================================================================================] 100.00%
|
Restore bucket 'test' succeeded
|
Mutations restored: 0, Mutations failed to restore: 0
|
Deletions restored: 1, Deletions failed to restore: 0
|
Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
|
Restore completed successfully
|
Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.
|
6.5.1 restore with workaround hangs |
# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
|
rd
|
(1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
|
Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
|
[================================================================================================================================== ] 99.81%
|
|
memcached exception |
2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
|
Attachments
Issue Links
Activity
| Field | Original Value | New Value |
|---|---|---|
| Summary | delete_with_meta incompatibility issues | delete_with_meta incompatibility issues between versions |
| Description |
+Problem+
cbbackupmgr restore fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back. +Steps to reproduce+ # Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 # Wait 10 seconds # Take a backup # Restore the backup to Couchbase Server 6.6.0 |
+Problem+
cbbackupmgr restore fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back. +Steps to reproduce+ # Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 # Wait 10 seconds # Take a backup # Restore the backup to Couchbase Server 6.6.0 |
| Description |
+Problem+
cbbackupmgr restore fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back. +Steps to reproduce+ # Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 # Wait 10 seconds # Take a backup # Restore the backup to Couchbase Server 6.6.0 |
+Problem+
cbbackupmgr restore fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 2. Wait 10 seconds 3. Take a backup 4. Restore the backup to Couchbase Server 6.6.0 Restores to previous versions of Couchbase Server (6.50 and lower) work fine. |
| Description |
+Problem+
cbbackupmgr restore fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 2. Wait 10 seconds 3. Take a backup 4. Restore the backup to Couchbase Server 6.6.0 Restores to previous versions of Couchbase Server (6.50 and lower) work fine. |
+Problem+
cbbackupmgr restore fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 2. Wait 10 seconds 3. Take a backup 4. Restore the backup to Couchbase Server 6.6.0 Restores to previous versions of Couchbase Server (6.50 and lower) work fine. |
| Description |
+Problem+
cbbackupmgr restore fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 2. Wait 10 seconds 3. Take a backup 4. Restore the backup to Couchbase Server 6.6.0 Restores to previous versions of Couchbase Server (6.50 and lower) work fine. |
+Problem+
cbbackupmgr restore fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 2. Wait 10 seconds 3. Take a backup 4. Restore the backup to Couchbase Server 6.6.0 Restores to previous versions of Couchbase Server (6.50 and lower) work fine. |
| Description |
+Problem+
cbbackupmgr restore fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 2. Wait 10 seconds 3. Take a backup 4. Restore the backup to Couchbase Server 6.6.0 Restores to previous versions of Couchbase Server (6.50 and lower) work fine. |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 2. Wait 10 seconds 3. Take a backup 4. Restore the backup to Couchbase Server 6.6.0 Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 2. Wait 10 seconds 3. Take a backup 4. Restore the backup to Couchbase Server 6.6.0 Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.0.3: {noformat} {noformat} |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.0.3: {noformat} {noformat} |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.0.3: {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.0.3: {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat} {noformat} The restore on 6.0.3: {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat} {noformat} The restore on 6.0.3: {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat} {noformat} The restore on 6.0.3: {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat} {noformat} The restore on 6.0.3: {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} --- Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat} {noformat} The restore on 6.0.3: {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- It should be noted that a tombstone having a body is invalid unfortunately that issue ( |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} --- Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat} {noformat} The restore on 6.0.3: {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- It should be noted that a tombstone having a body is invalid unfortunately that issue ( |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} --- Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.1: {noformat:title=Restore to 6.5.1} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.3: {noformat:title=Restore to 6.0.3} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- It should be noted that a tombstone having a body is invalid unfortunately that issue ( |
| Summary | delete_with_meta incompatibility issues between versions | delete_with_meta incompatibility issues between 6.6.0 and older versions |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} --- Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.1: {noformat:title=Restore to 6.5.1} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.3: {noformat:title=Restore to 6.0.3} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- It should be noted that a tombstone having a body is invalid unfortunately that issue ( |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} --- Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.1: {noformat:title=Restore to 6.5.1} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.3: {noformat:title=Restore to 6.0.3} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- It should be noted that a tombstone having a body is invalid unfortunately that issue ( |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} --- Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.1: {noformat:title=Restore to 6.5.1} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.3: {noformat:title=Restore to 6.0.3} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- It should be noted that a tombstone having a body is invalid unfortunately that issue ( |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies ( Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} --- +Restoring on older versions+ Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.1: {noformat:title=Restore to 6.5.1} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.3: {noformat:title=Restore to 6.0.3} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies ( Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} --- +Restoring on older versions+ Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.1: {noformat:title=Restore to 6.5.1} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.3: {noformat:title=Restore to 6.0.3} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies ( Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} +Restoring on older versions+ Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.1: {noformat:title=Restore to 6.5.1} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.3: {noformat:title=Restore to 6.0.3} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies ( Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} +Restoring on older versions+ Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.1: {noformat:title=Restore to 6.5.1} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.3: {noformat:title=Restore to 6.0.3} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies ( Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The invalid operations is *not* logged in the memcached.log: {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} +Restoring on older versions+ Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.1: {noformat:title=Restore to 6.5.1} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.3: {noformat:title=Restore to 6.0.3} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies ( Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The invalid operations is *not* logged in the memcached.log: {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} +Restoring on older versions+ Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.1: {noformat:title=Restore to 6.5.1} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.3: {noformat:title=Restore to 6.0.3} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat} {noformat} The restore on 6.0.4: {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat} {noformat} The restore on 6.0.4: {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 2, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat} {noformat} The restore on 6.0.4: {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat} {noformat} The restore on 6.0.4: {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat:title=6.5.0 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} |
| Component/s | XDCR [ 10136 ] | |
| Component/s | tools [ 10223 ] |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat:title=6.5.0 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} --- Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat:title=6.5.0 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} I think this means two of our upgrade processes will not work for 6.6.0 |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} --- Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat:title=6.5.0 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} I think this means two of our upgrade processes will not work for 6.6.0 |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} --- Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat:title=6.5.0 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} I think this means two of our upgrade processes will not work for 6.6.0 |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} --- Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat:title=6.5.0 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} I think this means two of our upgrade processes will not work for 6.6.0 |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} --- Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat:title=6.5.0 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} --- Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat:title=6.5.0 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} --- Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat:title=6.5.0 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} |
| Summary | delete_with_meta incompatibility issues between 6.6.0 and older versions | delete_with_meta incompatibility issues between 6.6.0 and older versions when using user xattrs |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} --- Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works. The restore on 6.5.0: {noformat:title=6.5.0 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.5.0: {noformat:title=6.5.0 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} |
| Affects Version/s | 6.5.1 [ 16622 ] |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.5.0: {noformat:title=6.5.0 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} |
| Summary | delete_with_meta incompatibility issues between 6.6.0 and older versions when using user xattrs | delete_with_meta incompatibility issues between 6.6.0/6.5.1 and older versions when using user xattrs |
| Affects Version/s | 6.5.0 [ 15037 ] |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception: |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception: |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} |
| Link | This issue blocks CBSE-9117 [ CBSE-9117 ] |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} --- +Summary of affected versions+ ||Backup / Restore|| || 6.0.4|| 6.5.0||6.5.1||6.6.0|| ||6.0.4.||(/)|(x)|(x)|(x)| ||6.5.0||N/A|(x)|(x)|(x)| ||6.5.1||N/A|N/A|(x)|(x)| ||6.6.0||N/A|N/A|N/A|(y)| |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} --- +Summary of affected versions+ ||Backup / Restore|| || 6.0.4|| 6.5.0||6.5.1||6.6.0|| ||6.0.4.||(/)|(x)|(x)|(x)| ||6.5.0||N/A|(x)|(x)|(x)| ||6.5.1||N/A|N/A|(x)|(x)| ||6.6.0||N/A|N/A|N/A|(y)| |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} --- +Summary of affected versions+ ||Backup \ Restore|| 6.0.4|| 6.5.0||6.5.1||6.6.0|| ||6.0.4.||(/)|(x)|(x)|(x)| ||6.5.0||N/A|(x)|(x)|(x)| ||6.5.1||N/A|N/A|(x)|(x)| ||6.6.0||N/A|N/A|N/A|(y)| |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} --- +Summary of affected versions+ ||Backup \ Restore|| 6.0.4|| 6.5.0||6.5.1||6.6.0|| ||6.0.4.||(/)|(x)|(x)|(x)| ||6.5.0||N/A|(x)|(x)|(x)| ||6.5.1||N/A|N/A|(x)|(x)| ||6.6.0||N/A|N/A|N/A|(y)| |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} --- +Summary of affected versions+ ||Backup V \ Restore Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0|| ||6.0.4.||(/)|(x)|(x)|(x)| ||6.5.0||N/A|(x)|(x)|(x)| ||6.5.1||N/A|N/A|(x)|(x)| ||6.6.0||N/A|N/A|N/A|(y)| |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} --- +Summary of affected versions+ ||Backup V \ Restore Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0|| ||6.0.4.||(/)|(x)|(x)|(x)| ||6.5.0||N/A|(x)|(x)|(x)| ||6.5.1||N/A|N/A|(x)|(x)| ||6.6.0||N/A|N/A|N/A|(y)| |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} --- +Summary of affected versions+ ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0|| ||6.0.4.||(/)|(x)|(x)|(x)| ||6.5.0||N/A|(x)|(x)|(x)| ||6.5.1||N/A|N/A|(x)|(x)| ||6.6.0||N/A|N/A|N/A|(y)| |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} --- +Summary of affected versions+ ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0|| ||6.0.4.||(/)|(x)|(x)|(x)| ||6.5.0||N/A|(x)|(x)|(x)| ||6.5.1||N/A|N/A|(x)|(x)| ||6.6.0||N/A|N/A|N/A|(y)| |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} --- +Summary of affected versions+ The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to. ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0|| ||6.0.4.||(/)|(x)|(x)|(x)| ||6.5.0||N/A|(x)|(x)|(x)| ||6.5.1||N/A|N/A|(x)|(x)| ||6.6.0||N/A|N/A|N/A|(y)| |
| Description |
+Problem+
{{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} --- +Summary of affected versions+ The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to. ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0|| ||6.0.4.||(/)|(x)|(x)|(x)| ||6.5.0||N/A|(x)|(x)|(x)| ||6.5.1||N/A|N/A|(x)|(x)| ||6.6.0||N/A|N/A|N/A|(y)| |
H1. TL;DR:
In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail: +Affected versions+ The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to. ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0|| ||6.0.4.||(/)|(x)|(x)|(x)| ||6.5.0||N/A|(x)|(x)|(x)| ||6.5.1||N/A|N/A|(x)|(x)| ||6.6.0||N/A|N/A|N/A|(y)| --- +Problem+ {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} |
| Description |
H1. TL;DR:
In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail: +Affected versions+ The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to. ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0|| ||6.0.4.||(/)|(x)|(x)|(x)| ||6.5.0||N/A|(x)|(x)|(x)| ||6.5.1||N/A|N/A|(x)|(x)| ||6.6.0||N/A|N/A|N/A|(y)| --- +Problem+ {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} |
H1. TL;DR:
In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail: +Affected versions+ The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to. ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0|| ||6.0.4.||(/)|(x)|(x)|(x)| ||6.5.0||N/A|(x)|(x)|(x)| ||6.5.1||N/A|N/A|(x)|(x)| ||6.6.0||N/A|N/A|N/A|(/)| --- +Problem+ {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} |
| Description |
H1. TL;DR:
In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail: +Affected versions+ The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to. ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0|| ||6.0.4.||(/)|(x)|(x)|(x)| ||6.5.0||N/A|(x)|(x)|(x)| ||6.5.1||N/A|N/A|(x)|(x)| ||6.6.0||N/A|N/A|N/A|(/)| --- +Problem+ {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} |
H1. TL;DR:
In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail: +Affected versions+ The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to. ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0|| ||6.0.4.||(/)|(x)|(x)|(x)*| ||6.5.0||N/A|(x)|(x)|(x*)| ||6.5.1||N/A|N/A|(x)|(x)*| ||6.6.0||N/A|N/A|N/A|(/)*| * When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below. --- +Problem+ {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} |
| Description |
H1. TL;DR:
In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail: +Affected versions+ The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to. ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0|| ||6.0.4.||(/)|(x)|(x)|(x)*| ||6.5.0||N/A|(x)|(x)|(x*)| ||6.5.1||N/A|N/A|(x)|(x)*| ||6.6.0||N/A|N/A|N/A|(/)*| * When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below. --- +Problem+ {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} |
H1. TL;DR:
In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail: +Affected versions+ The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to. ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0|| ||6.0.4||(/)|(x)|(x)|(x)*| ||6.5.0||N/A|(x)|(x)|(x*)| ||6.5.1||N/A|N/A|(x)|(x)*| ||6.6.0||N/A|N/A|N/A|(/)*| \* When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below. --- +Problem+ {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} |
| Description |
H1. TL;DR:
In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail: +Affected versions+ The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to. ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0|| ||6.0.4||(/)|(x)|(x)|(x)*| ||6.5.0||N/A|(x)|(x)|(x*)| ||6.5.1||N/A|N/A|(x)|(x)*| ||6.6.0||N/A|N/A|N/A|(/)*| \* When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below. --- +Problem+ {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} |
H1. TL;DR:
In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail: +Affected versions+ The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to. ||Backup (rows) \ Restore (columns) Versions || <=6.0.4|| 6.5.0||6.5.1||6.6.0|| ||<=6.0.4||(/)|(x)|(x)|(x)*| ||6.5.0||N/A|(x)|(x)|(x*)| ||6.5.1||N/A|N/A|(x)|(x)*| ||6.6.0||N/A|N/A|N/A|(/)*| \* When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below. --- +Problem+ {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} |
| Assignee | Patrick Varley [ pvarley ] | Paolo Cocchi [ paolo.cocchi ] |
| Fix Version/s | 6.6.1 [ 17002 ] |
| Description |
H1. TL;DR:
In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail: +Affected versions+ The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to. ||Backup (rows) \ Restore (columns) Versions || <=6.0.4|| 6.5.0||6.5.1||6.6.0|| ||<=6.0.4||(/)|(x)|(x)|(x)*| ||6.5.0||N/A|(x)|(x)|(x*)| ||6.5.1||N/A|N/A|(x)|(x)*| ||6.6.0||N/A|N/A|N/A|(/)*| \* When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below. --- +Problem+ {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} |
H1. TL;DR:
In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail: +Affected versions+ The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to. ||Backup (rows) \ Restore (columns) Versions || <=6.0.4|| 6.5.0||6.5.1||6.6.0|| ||<=6.0.4||(/)|(x)|(x)|(x)*| ||6.5.0||N/A|(x)|(x)|(x)*| ||6.5.1||N/A|N/A|(x)|(x)*| ||6.6.0||N/A|N/A|N/A|(/)*| \* When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below. --- +Problem+ {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} |
Patrick Varley I've reviewed all possible cases where we may generate an invalid payload at expiration (MB-40467, pre-6.6):
Summary of when we do/don't hit the issue in pre-6.6:
SysXattrs in the paylaod -> all good (VBucket::handlePreExpiry behaves correctly)
No SysXattrs:
- only Body: all good (VBucket::handlePreExpiry generates invalid payloads but VBucket::processExpiredItem rectifies)
- only UserXattrs: we hit the issue
- Body + UserXattrs: we hit the issue
.. the final behaviour is that we end up with invalid payloads in expirations iff we don't have SysXattrs but we do have UserXattrs in the value.
| Labels | approved-for-6.6.1 |
| Link | This issue blocks MB-40528 [ MB-40528 ] |
| Link | This issue relates to CBQE-6179 [ CBQE-6179 ] |
| Description |
H1. TL;DR:
In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail: +Affected versions+ The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to. ||Backup (rows) \ Restore (columns) Versions || <=6.0.4|| 6.5.0||6.5.1||6.6.0|| ||<=6.0.4||(/)|(x)|(x)|(x)*| ||6.5.0||N/A|(x)|(x)|(x)*| ||6.5.1||N/A|N/A|(x)|(x)*| ||6.6.0||N/A|N/A|N/A|(/)*| \* When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below. --- +Problem+ {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} |
h1. TL;DR:
h1. In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail: +Affected versions+ The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to. ||Backup (rows) \ Restore (columns) Versions || <=6.0.4|| 6.5.0||6.5.1||6.6.0|| ||<=6.0.4||(/)|(x)|(x)|(x)*| ||6.5.0||N/A|(x)|(x)|(x)*| ||6.5.1||N/A|N/A|(x)|(x)*| ||6.6.0||N/A|N/A|N/A|(/)*| \* When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below. --- +Problem+ {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted. +Notes+ I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0. Interestingly on 6.6.0 the following config is set: {noformat:title=6.6.0} # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del ep_allow_del_with_meta_prune_user_data: false {noformat} There does not seem to be away to see what this is set to on old versions. +Steps to reproduce+ 1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3 {noformat} /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password subdoc> set test value -x xattr=100 -e 10 {noformat} 2. Wait 10 seconds 3. Take a backup {noformat} /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password {noformat} 4. From the examine command we can see that the tombstone contains the xattr and a value: {noformat} /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test Key: test SeqNo: 2 Backup: 2020-10-29T15_05_45.745791997Z Deleted: true Size: 4B (key), 29B (meta), 3B (value) Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6} Xattrs: {"xattr":100} Value: value {noformat} 5. Restore the backup to Couchbase Server 6.6.0 {noformat} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB [===============================================================================================================================================================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=memcached.log} 2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ] 2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud> {noformat} The restore on 6.5.1 and 6.5.0 also fails: {noformat:title=6.5.1 Restore} [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} {noformat:title=6.5.0 Restore} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB [==================================================================================================================================] 100.00% Error restoring cluster: invalid argument Restore bucket 'test' failed Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 0, Deletions failed to restore: 1 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 {noformat} --- Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works. The restore on 6.0.4: {noformat:title=6.0.4 Restore} /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} --- The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected: {noformat:title=6.0.4 goxdcr.log} 2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0] {noformat} +Workaround+ There is a work around but it only works for 6.6.0: {noformat:title=cbepctl workaround} # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true setting param: allow_del_with_meta_prune_user_data true set allow_del_with_meta_prune_user_data to true {noformat} This will allow the restore to work: {noformat:title=6.6.0 restore with workaround} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB [==================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully {noformat} Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception. {noformat:title=6.5.1 restore with workaround hangs} # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo rd (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z' Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB [================================================================================================================================== ] 99.81% {noformat} {noformat:title=memcached exception} 2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0 {noformat} |
Patrick Varley allow_del_with_meta_prune_user_data enabled, assigning back to you. Thanks
| Assignee | Paolo Cocchi [ paolo.cocchi ] | Patrick Varley [ pvarley ] |
| Resolution | Done [ 6 ] | |
| Status | Open [ 1 ] | Resolved [ 5 ] |
Seeing following error in memcached log while trying to restore backup on version 6.6.1-9182.
[root@node1-cb601-centos7 ~]# grep -R "XATTR, XERROR" /opt/couchbase/
|
bin/ etc/ lib/ LICENSE.txt manifest.xml README.txt samples/ share/ var/ VARIANT.txt VERSION.txt
|
[root@node1-cb601-centos7 ~]# grep -R "XATTR, XERROR" /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt
|
2020-11-12T22:34:34.205412-08:00 INFO 50: HELO [{"a":"java/3.0.8 (Mac OS X 10.15.6 x86_64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08)","i":"4DC2B38A00000001/0000000044555396"}] Mutation seqno, XATTR, XERROR, Select bucket, Snappy, Unordered execution, Tracing, AltRequestSupport, SyncReplication, VAttr, SubdocCreateAsDeleted [ {"ip":"10.112.191.1","port":53878} - {"ip":"10.112.191.101","port":11210} (not authenticated) ]
|
2020-11-12T22:34:34.206164-08:00 INFO 55: HELO [{"a":"java/3.0.8 (Mac OS X 10.15.6 x86_64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08)","i":"4DC2B38A00000001/00000000844E2CDF"}] Mutation seqno, XATTR, XERROR, Select bucket, Snappy, Unordered execution, Tracing, AltRequestSupport, SyncReplication, VAttr, SubdocCreateAsDeleted [ {"ip":"10.112.191.1","port":53879} - {"ip":"10.112.191.101","port":11210} (not authenticated) ]
|
2020-11-12T23:47:43.597165-08:00 INFO 49: HELO [{"a":"gocbcore/v9.0.7 cbbackupmgr-6.6.1-9182-305333a","i":"332db416ed6b1e65/1204d8b4fe5fcaa9"}] XATTR, XERROR, Select bucket, Snappy, JSON, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ {"ip":"10.112.191.101","port":56770} - {"ip":"10.112.191.101","port":11210} (not authenticated) ]
|
Though the restore completed successfully.
[root@node1-cb601-centos7 ~]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.191.101 -u Administrator -p password
|
(1/1) Restoring backup 2020-11-12T23_45_12.320791066-08_00 '2020-11-12T23_45_12.320791066-08_00'
|
Copied all data in 113.363318ms (Avg. 28.11KB/Sec) 1 items / 28.11KB
|
[====================================================================================================================================================================================================] 100.00%
|
Restore bucket 'test' succeeded
|
Mutations restored: 0,
|
Mutations failed to restore: 0
|
Deletions restored: 1,
|
Deletions failed to restore: 0
|
Skipped due to purge number or conflict resolution:
|
Mutations: 0 Deletions: 0
|
Restore completed successfully
|
Patrick Varley / Paolo Cocchi is the mc_error log we see is expected ?
Ashwin Govindarajulu XERROR (ie, Extended Error) is a memcached feature. HELO logs above mean that those connections support XERROR.
Validated using '6.6.1-9182'.
Closing this ticket.
| VERIFICATION STEPS |
1. Created doc with ttl=10 on Enterprise Edition 6.0.1 build 2037 cluster
2. Took backup using 'cbbackupmgr backup' command 3. Restored on ' 6.6.0 build 7897' node and backup failed with reason 'Deletions failed to restore: 1' 4. Restored on '6.6.1-9182' and backup succeeded |
|
| Assignee | Patrick Varley [ pvarley ] | Ashwin Govindarajulu [ ashwin.govindarajulu ] |
| Status | Resolved [ 5 ] | Closed [ 6 ] |
Setting to 6.6.1 as I think we want to set ep_allow_del_with_meta_prune_user_data = true