Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-42352

delete_with_meta incompatibility issues between 6.6.0/6.5.1 and older versions when using user xattrs

    XMLWordPrintable

    Details

    • Triage:
      Untriaged
    • Story Points:
      1
    • Is this a Regression?:
      Unknown

      Description

      TL;DR:

      In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail:

      Affected versions
      The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to.

      Backup (rows) \ Restore (columns) Versions <=6.0.4 6.5.0 6.5.1 6.6.0
      <=6.0.4 *
      6.5.0 N/A *
      6.5.1 N/A N/A *
      6.6.0 N/A N/A N/A *

      * When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below.


      Problem
      cbbackupmgr restore fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

      Notes

      I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

      Interestingly on 6.6.0 the following config is set:

      6.6.0

      # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
      ep_allow_del_with_meta_prune_user_data:                false
      

      There does not seem to be away to see what this is set to on old versions.

      Steps to reproduce
      1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3

       /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
       subdoc> set test value -x xattr=100 -e 10
      

      2. Wait 10 seconds
      3. Take a backup

      /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
      /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie  -c 10.112.194.101 -u Administrator -p password
      

      4. From the examine command we can see that the tombstone contains the xattr and a value:

      /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie   --bucket test --tombstones -k test
      Key: test
        SeqNo: 2
        Backup: 2020-10-29T15_05_45.745791997Z
        Deleted: true
        Size: 4B (key), 29B (meta), 3B (value)
        Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
        Xattrs: {"xattr":100}
        Value: value
      

      5. Restore the backup to Couchbase Server 6.6.0

       /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie  -c localhost -u Administrator -p password
      (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
      Transferring key value data for 'test' at 0B/s (about 0s remaining)                                                                                                                                                                                                     2 items / 28.20KB
      [===============================================================================================================================================================================================================================================================================] 100.00%
      Error restoring cluster: invalid argument
      Restore bucket 'test' failed
      Mutations restored: 0, Mutations failed to restore: 0
      Deletions restored: 0, Deletions failed to restore: 1
      Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
      

      memcached.log

      2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
      2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
      

      The restore on 6.5.1 and 6.5.0 also fails:

      6.5.1 Restore

      [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie  -c 10.112.201.101 -u Administrator -p passwo
      rd
      (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
      Transferring key value data for 'test' at 0B/s (about 0s remaining)                                                        1 items / 28.20KB
      [==================================================================================================================================] 100.00%
      Error restoring cluster: invalid argument
      Restore bucket 'test' failed
      Mutations restored: 0, Mutations failed to restore: 0
      Deletions restored: 0, Deletions failed to restore: 1
      Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
      

      6.5.0 Restore

      # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie  -c 10.112.200.101 -u Administrator -p password
      (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
      Transferring key value data for 'test' at 0B/s (about 0s remaining)                                                        1 items / 28.20KB
      [==================================================================================================================================] 100.00%
      Error restoring cluster: invalid argument
      Restore bucket 'test' failed
      Mutations restored: 0, Mutations failed to restore: 0
      Deletions restored: 0, Deletions failed to restore: 1
      Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
      
      


      Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

      The restore on 6.0.4:

      6.0.4 Restore

      /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie  -c 10.112.194.101 -u Administrator -p passwo
      rd
      (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
      Copied all data in 156.041335ms (Avg. 28.20KB/Sec)                                                                         1 items / 28.20KB
      [==================================================================================================================================] 100.00%
      Restore bucket 'test' succeeded
      Mutations restored: 0, Mutations failed to restore: 0
      Deletions restored: 1, Deletions failed to restore: 0
      Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
      Restore completed successfully
      

      The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

      6.0.4 goxdcr.log

      2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
      

      Workaround

      There is a work around but it only works for 6.6.0:

      cbepctl workaround

      # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
      setting param: allow_del_with_meta_prune_user_data true
      set allow_del_with_meta_prune_user_data to true
      

      This will allow the restore to work:

      6.6.0 restore with workaround

      # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie  -c localhost -u Administrator -p password
      (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
      Copied all data in 90.191456ms (Avg. 28.20KB/Sec)                                                                          1 items / 28.20KB
      [==================================================================================================================================] 100.00%
      Restore bucket 'test' succeeded
      Mutations restored: 0, Mutations failed to restore: 0
      Deletions restored: 1, Deletions failed to restore: 0
      Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
      Restore completed successfully
      

      Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

      6.5.1 restore with workaround hangs

      # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie  -c 10.112.201.101 -u Administrator -p passwo
      rd
      (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
      Transferring key value data for 'test' at 192B/s (about 0s remaining)                                                      1 items / 28.20KB
      [================================================================================================================================== ] 99.81%
      

      memcached exception

      2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate.  buffer.size:21 uncompressedLength:0
      

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            pvarley Patrick Varley created issue -
            pvarley Patrick Varley made changes -
            Field Original Value New Value
            Summary delete_with_meta incompatibility issues delete_with_meta incompatibility issues between versions
            pvarley Patrick Varley made changes -
            Description +Problem+
            cbbackupmgr restore fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back.

            +Steps to reproduce+
            # Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            # Wait 10 seconds
            # Take a backup
            # Restore the backup to Couchbase Server 6.6.0

            +Problem+
            cbbackupmgr restore fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back.

            +Steps to reproduce+
            # Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            # Wait 10 seconds
            # Take a backup
            # Restore the backup to Couchbase Server 6.6.0



            pvarley Patrick Varley made changes -
            Description +Problem+
            cbbackupmgr restore fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back.

            +Steps to reproduce+
            # Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            # Wait 10 seconds
            # Take a backup
            # Restore the backup to Couchbase Server 6.6.0



            +Problem+
            cbbackupmgr restore fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            2. Wait 10 seconds
            3. Take a backup
            4. Restore the backup to Couchbase Server 6.6.0

            Restores to previous versions of Couchbase Server (6.50 and lower) work fine.



            pvarley Patrick Varley made changes -
            Description +Problem+
            cbbackupmgr restore fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            2. Wait 10 seconds
            3. Take a backup
            4. Restore the backup to Couchbase Server 6.6.0

            Restores to previous versions of Couchbase Server (6.50 and lower) work fine.



            +Problem+
            cbbackupmgr restore fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
             ep_allow_del_with_meta_prune_user_data: false
            {noformat}


            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            2. Wait 10 seconds
            3. Take a backup
            4. Restore the backup to Couchbase Server 6.6.0

            Restores to previous versions of Couchbase Server (6.50 and lower) work fine.




            pvarley Patrick Varley made changes -
            Description +Problem+
            cbbackupmgr restore fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
             ep_allow_del_with_meta_prune_user_data: false
            {noformat}


            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            2. Wait 10 seconds
            3. Take a backup
            4. Restore the backup to Couchbase Server 6.6.0

            Restores to previous versions of Couchbase Server (6.50 and lower) work fine.




            +Problem+
            cbbackupmgr restore fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
             ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            2. Wait 10 seconds
            3. Take a backup
            4. Restore the backup to Couchbase Server 6.6.0

            Restores to previous versions of Couchbase Server (6.50 and lower) work fine.




            pvarley Patrick Varley made changes -
            Description +Problem+
            cbbackupmgr restore fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. cbbbackupmgr backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
             ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            2. Wait 10 seconds
            3. Take a backup
            4. Restore the backup to Couchbase Server 6.6.0

            Restores to previous versions of Couchbase Server (6.50 and lower) work fine.




            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
             ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            2. Wait 10 seconds
            3. Take a backup
            4. Restore the backup to Couchbase Server 6.6.0

            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.




            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
             ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            2. Wait 10 seconds
            3. Take a backup
            4. Restore the backup to Couchbase Server 6.6.0

            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.




            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.
            pvarley Patrick Varley made changes -
            Link This issue depends on MB-41944 [ MB-41944 ]
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.0.3:

            {noformat}

            {noformat}
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.0.3:

            {noformat}

            {noformat}
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.0.3:

            {noformat}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.0.3:

            {noformat}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat}

            {noformat}

            The restore on 6.0.3:

            {noformat}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat}

            {noformat}

            The restore on 6.0.3:

            {noformat}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat}

            {noformat}

            The restore on 6.0.3:

            {noformat}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}
            pvarley Patrick Varley made changes -
            Link This issue depends on MB-41944 [ MB-41944 ]
            pvarley Patrick Varley made changes -
            Link This issue relates to MB-40467 [ MB-40467 ]
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat}

            {noformat}

            The restore on 6.0.3:

            {noformat}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}
            ---
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat}

            {noformat}

            The restore on 6.0.3:

            {noformat}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            It should be noted that a tombstone having a body is invalid unfortunately that issue (MB-40467) is in Couchbase Server 6.5.X and 6.0.X.
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}
            ---
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat}

            {noformat}

            The restore on 6.0.3:

            {noformat}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            It should be noted that a tombstone having a body is invalid unfortunately that issue (MB-40467) is in Couchbase Server 6.5.X and 6.0.X.
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}
            ---
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.1:

            {noformat:title=Restore to 6.5.1}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            The restore on 6.0.3:

            {noformat:title=Restore to 6.0.3}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            It should be noted that a tombstone having a body is invalid unfortunately that issue (MB-40467) is in Couchbase Server 6.5.X and 6.0.X.
            pvarley Patrick Varley made changes -
            Summary delete_with_meta incompatibility issues between versions delete_with_meta incompatibility issues between 6.6.0 and older versions
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}
            ---
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.1:

            {noformat:title=Restore to 6.5.1}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            The restore on 6.0.3:

            {noformat:title=Restore to 6.0.3}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            It should be noted that a tombstone having a body is invalid unfortunately that issue (MB-40467) is in Couchbase Server 6.5.X and 6.0.X.
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}
            ---
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.1:

            {noformat:title=Restore to 6.5.1}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            The restore on 6.0.3:

            {noformat:title=Restore to 6.0.3}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            It should be noted that a tombstone having a body is invalid unfortunately that issue (MB-40467) is in Couchbase Server 6.5.X and 6.0.X. This affects the compatibility of restoring older backup against 6.6.0 clusters.
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}
            ---
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.1:

            {noformat:title=Restore to 6.5.1}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            The restore on 6.0.3:

            {noformat:title=Restore to 6.0.3}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            It should be noted that a tombstone having a body is invalid unfortunately that issue (MB-40467) is in Couchbase Server 6.5.X and 6.0.X. This affects the compatibility of restoring older backup against 6.6.0 clusters.
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies (MB-40467). {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}
            ---

            +Restoring on older versions+
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.1:

            {noformat:title=Restore to 6.5.1}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            The restore on 6.0.3:

            {noformat:title=Restore to 6.0.3}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies (MB-40467). {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}
            ---

            +Restoring on older versions+
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.1:

            {noformat:title=Restore to 6.5.1}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            The restore on 6.0.3:

            {noformat:title=Restore to 6.0.3}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies (MB-40467). {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This affects the compatibility of restoring older backup against Couchbase Server 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            +Restoring on older versions+
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.1:

            {noformat:title=Restore to 6.5.1}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            The restore on 6.0.3:

            {noformat:title=Restore to 6.0.3}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies (MB-40467). {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This affects the compatibility of restoring older backup against Couchbase Server 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            +Restoring on older versions+
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.1:

            {noformat:title=Restore to 6.5.1}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            The restore on 6.0.3:

            {noformat:title=Restore to 6.0.3}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies (MB-40467). {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This affects the compatibility of restoring older backup against Couchbase Server 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            The invalid operations is *not* logged in the memcached.log:

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            +Restoring on older versions+
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.1:

            {noformat:title=Restore to 6.5.1}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            The restore on 6.0.3:

            {noformat:title=Restore to 6.0.3}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies (MB-40467). {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This affects the compatibility of restoring older backup against Couchbase Server 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            The invalid operations is *not* logged in the memcached.log:

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            +Restoring on older versions+
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.1:

            {noformat:title=Restore to 6.5.1}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            The restore on 6.0.3:

            {noformat:title=Restore to 6.0.3}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat}

            {noformat}

            The restore on 6.0.4:

            {noformat}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat}

            {noformat}

            The restore on 6.0.4:

            {noformat}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 2, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat}

            {noformat}

            The restore on 6.0.4:

            {noformat}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat}

            {noformat}

            The restore on 6.0.4:

            {noformat}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat:title=6.5.0 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}
            pvarley Patrick Varley made changes -
            Component/s XDCR [ 10136 ]
            Component/s tools [ 10223 ]
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat:title=6.5.0 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}
            ---
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat:title=6.5.0 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}

            I think this means two of our upgrade processes will not work for 6.6.0
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}
            ---
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat:title=6.5.0 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}

            I think this means two of our upgrade processes will not work for 6.6.0
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}
            ---
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat:title=6.5.0 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}

            I think this means two of our upgrade processes will not work for 6.6.0
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}
            ---
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat:title=6.5.0 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}

            I think this means two of our upgrade processes will not work for 6.6.0
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}
            ---
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat:title=6.5.0 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}

            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}
            ---
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat:title=6.5.0 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}

            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}
            ---
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat:title=6.5.0 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}

            pvarley Patrick Varley made changes -
            Summary delete_with_meta incompatibility issues between 6.6.0 and older versions delete_with_meta incompatibility issues between 6.6.0 and older versions when using user xattrs
            pvarley Patrick Varley made changes -
            Link This issue relates to MB-42356 [ MB-42356 ]
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}
            ---
            Restores to previous versions of Couchbase Server (6.5.0 and lower) the restore works.

            The restore on 6.5.0:

            {noformat:title=6.5.0 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}

            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}
            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.5.0:

            {noformat:title=6.5.0 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}

            pvarley Patrick Varley made changes -
            Affects Version/s 6.5.1 [ 16622 ]
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}
            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.5.0:

            {noformat:title=6.5.0 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}

            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}

            pvarley Patrick Varley made changes -
            Summary delete_with_meta incompatibility issues between 6.6.0 and older versions when using user xattrs delete_with_meta incompatibility issues between 6.6.0/6.5.1 and older versions when using user xattrs
            pvarley Patrick Varley made changes -
            Affects Version/s 6.5.0 [ 15037 ]
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}

            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}

            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}

            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception:

            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception:

            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}
            pvarley Patrick Varley made changes -
            Link This issue relates to MB-42359 [ MB-42359 ]
            pvarley Patrick Varley made changes -
            Link This issue blocks CBSE-9117 [ CBSE-9117 ]
            owend Daniel Owen made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}


            ---
            +Summary of affected versions+

            ||Backup / Restore|| || 6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||6.0.4.||(/)|(x)|(x)|(x)|
            ||6.5.0||N/A|(x)|(x)|(x)|
            ||6.5.1||N/A|N/A|(x)|(x)|
            ||6.6.0||N/A|N/A|N/A|(y)|
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}


            ---
            +Summary of affected versions+

            ||Backup / Restore|| || 6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||6.0.4.||(/)|(x)|(x)|(x)|
            ||6.5.0||N/A|(x)|(x)|(x)|
            ||6.5.1||N/A|N/A|(x)|(x)|
            ||6.6.0||N/A|N/A|N/A|(y)|
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}


            ---
            +Summary of affected versions+

            ||Backup \ Restore|| 6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||6.0.4.||(/)|(x)|(x)|(x)|
            ||6.5.0||N/A|(x)|(x)|(x)|
            ||6.5.1||N/A|N/A|(x)|(x)|
            ||6.6.0||N/A|N/A|N/A|(y)|
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}


            ---
            +Summary of affected versions+

            ||Backup \ Restore|| 6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||6.0.4.||(/)|(x)|(x)|(x)|
            ||6.5.0||N/A|(x)|(x)|(x)|
            ||6.5.1||N/A|N/A|(x)|(x)|
            ||6.6.0||N/A|N/A|N/A|(y)|
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}


            ---
            +Summary of affected versions+

            ||Backup V \ Restore Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||6.0.4.||(/)|(x)|(x)|(x)|
            ||6.5.0||N/A|(x)|(x)|(x)|
            ||6.5.1||N/A|N/A|(x)|(x)|
            ||6.6.0||N/A|N/A|N/A|(y)|
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}


            ---
            +Summary of affected versions+

            ||Backup V \ Restore Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||6.0.4.||(/)|(x)|(x)|(x)|
            ||6.5.0||N/A|(x)|(x)|(x)|
            ||6.5.1||N/A|N/A|(x)|(x)|
            ||6.6.0||N/A|N/A|N/A|(y)|
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}


            ---
            +Summary of affected versions+

            ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||6.0.4.||(/)|(x)|(x)|(x)|
            ||6.5.0||N/A|(x)|(x)|(x)|
            ||6.5.1||N/A|N/A|(x)|(x)|
            ||6.6.0||N/A|N/A|N/A|(y)|
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}


            ---
            +Summary of affected versions+

            ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||6.0.4.||(/)|(x)|(x)|(x)|
            ||6.5.0||N/A|(x)|(x)|(x)|
            ||6.5.1||N/A|N/A|(x)|(x)|
            ||6.6.0||N/A|N/A|N/A|(y)|
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}


            ---
            +Summary of affected versions+

            The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to.

            ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||6.0.4.||(/)|(x)|(x)|(x)|
            ||6.5.0||N/A|(x)|(x)|(x)|
            ||6.5.1||N/A|N/A|(x)|(x)|
            ||6.6.0||N/A|N/A|N/A|(y)|
            pvarley Patrick Varley made changes -
            Description +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}


            ---
            +Summary of affected versions+

            The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to.

            ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||6.0.4.||(/)|(x)|(x)|(x)|
            ||6.5.0||N/A|(x)|(x)|(x)|
            ||6.5.1||N/A|N/A|(x)|(x)|
            ||6.6.0||N/A|N/A|N/A|(y)|
            H1. TL;DR:

            In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail:
             
            +Affected versions+
            The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to.

            ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||6.0.4.||(/)|(x)|(x)|(x)|
            ||6.5.0||N/A|(x)|(x)|(x)|
            ||6.5.1||N/A|N/A|(x)|(x)|
            ||6.6.0||N/A|N/A|N/A|(y)|

            ---
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}
            pvarley Patrick Varley made changes -
            Description H1. TL;DR:

            In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail:
             
            +Affected versions+
            The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to.

            ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||6.0.4.||(/)|(x)|(x)|(x)|
            ||6.5.0||N/A|(x)|(x)|(x)|
            ||6.5.1||N/A|N/A|(x)|(x)|
            ||6.6.0||N/A|N/A|N/A|(y)|

            ---
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}
            H1. TL;DR:

            In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail:
             
            +Affected versions+
            The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to.

            ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||6.0.4.||(/)|(x)|(x)|(x)|
            ||6.5.0||N/A|(x)|(x)|(x)|
            ||6.5.1||N/A|N/A|(x)|(x)|
            ||6.6.0||N/A|N/A|N/A|(/)|

            ---
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}
            pvarley Patrick Varley made changes -
            Description H1. TL;DR:

            In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail:
             
            +Affected versions+
            The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to.

            ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||6.0.4.||(/)|(x)|(x)|(x)|
            ||6.5.0||N/A|(x)|(x)|(x)|
            ||6.5.1||N/A|N/A|(x)|(x)|
            ||6.6.0||N/A|N/A|N/A|(/)|

            ---
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}
            H1. TL;DR:

            In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail:
             
            +Affected versions+
            The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to.

            ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||6.0.4.||(/)|(x)|(x)|(x)*|
            ||6.5.0||N/A|(x)|(x)|(x*)|
            ||6.5.1||N/A|N/A|(x)|(x)*|
            ||6.6.0||N/A|N/A|N/A|(/)*|

            * When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below.

            ---
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}
            pvarley Patrick Varley made changes -
            Description H1. TL;DR:

            In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail:
             
            +Affected versions+
            The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to.

            ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||6.0.4.||(/)|(x)|(x)|(x)*|
            ||6.5.0||N/A|(x)|(x)|(x*)|
            ||6.5.1||N/A|N/A|(x)|(x)*|
            ||6.6.0||N/A|N/A|N/A|(/)*|

            * When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below.

            ---
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}
            H1. TL;DR:

            In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail:
             
            +Affected versions+
            The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to.

            ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||6.0.4||(/)|(x)|(x)|(x)*|
            ||6.5.0||N/A|(x)|(x)|(x*)|
            ||6.5.1||N/A|N/A|(x)|(x)*|
            ||6.6.0||N/A|N/A|N/A|(/)*|

            \* When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below.

            ---
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}
            drigby Dave Rigby made changes -
            Description H1. TL;DR:

            In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail:
             
            +Affected versions+
            The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to.

            ||Backup (rows) \ Restore (columns) Versions || 6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||6.0.4||(/)|(x)|(x)|(x)*|
            ||6.5.0||N/A|(x)|(x)|(x*)|
            ||6.5.1||N/A|N/A|(x)|(x)*|
            ||6.6.0||N/A|N/A|N/A|(/)*|

            \* When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below.

            ---
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}
            H1. TL;DR:

            In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail:
             
            +Affected versions+
            The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to.

            ||Backup (rows) \ Restore (columns) Versions || <=6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||<=6.0.4||(/)|(x)|(x)|(x)*|
            ||6.5.0||N/A|(x)|(x)|(x*)|
            ||6.5.1||N/A|N/A|(x)|(x)*|
            ||6.6.0||N/A|N/A|N/A|(/)*|

            \* When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below.

            ---
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}
            pvarley Patrick Varley made changes -
            Assignee Patrick Varley [ pvarley ] Paolo Cocchi [ paolo.cocchi ]
            owend Daniel Owen made changes -
            Fix Version/s 6.6.1 [ 17002 ]
            Hide
            owend Daniel Owen added a comment -

            Setting to 6.6.1 as I think we want to set ep_allow_del_with_meta_prune_user_data = true

            Show
            owend Daniel Owen added a comment - Setting to 6.6.1 as I think we want to set ep_allow_del_with_meta_prune_user_data = true
            owend Daniel Owen made changes -
            Description H1. TL;DR:

            In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail:
             
            +Affected versions+
            The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to.

            ||Backup (rows) \ Restore (columns) Versions || <=6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||<=6.0.4||(/)|(x)|(x)|(x)*|
            ||6.5.0||N/A|(x)|(x)|(x*)|
            ||6.5.1||N/A|N/A|(x)|(x)*|
            ||6.6.0||N/A|N/A|N/A|(/)*|

            \* When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below.

            ---
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}
            H1. TL;DR:

            In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail:
             
            +Affected versions+
            The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to.

            ||Backup (rows) \ Restore (columns) Versions || <=6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||<=6.0.4||(/)|(x)|(x)|(x)*|
            ||6.5.0||N/A|(x)|(x)|(x)*|
            ||6.5.1||N/A|N/A|(x)|(x)*|
            ||6.6.0||N/A|N/A|N/A|(/)*|

            \* When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below.

            ---
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}
            Hide
            paolo.cocchi Paolo Cocchi added a comment - - edited

            Patrick Varley I've reviewed all possible cases where we may generate an invalid payload at expiration (MB-40467, pre-6.6):

            Summary of when we do/don't hit the issue in pre-6.6:

            SysXattrs in the paylaod -> all good (VBucket::handlePreExpiry behaves correctly)
            No SysXattrs:

            • only Body: all good (VBucket::handlePreExpiry generates invalid payloads but VBucket::processExpiredItem rectifies)
            • only UserXattrs: we hit the issue
            • Body + UserXattrs: we hit the issue

            .. the final behaviour is that we end up with invalid payloads in expirations iff we don't have SysXattrs but we do have UserXattrs in the value.

            Show
            paolo.cocchi Paolo Cocchi added a comment - - edited Patrick Varley I've reviewed all possible cases where we may generate an invalid payload at expiration ( MB-40467 , pre-6.6): Summary of when we do/don't hit the issue in pre-6.6: SysXattrs in the paylaod -> all good (VBucket::handlePreExpiry behaves correctly) No SysXattrs: only Body: all good (VBucket::handlePreExpiry generates invalid payloads but VBucket::processExpiredItem rectifies) only UserXattrs: we hit the issue Body + UserXattrs: we hit the issue .. the final behaviour is that we end up with invalid payloads in expirations iff we don't have SysXattrs but we do have UserXattrs in the value.
            owend Daniel Owen made changes -
            Labels approved-for-6.6.1
            owend Daniel Owen made changes -
            Link This issue blocks MB-40528 [ MB-40528 ]
            arunkumar Arunkumar Senthilnathan made changes -
            Link This issue relates to CBQE-6179 [ CBQE-6179 ]
            pvarley Patrick Varley made changes -
            Description H1. TL;DR:

            In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail:
             
            +Affected versions+
            The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to.

            ||Backup (rows) \ Restore (columns) Versions || <=6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||<=6.0.4||(/)|(x)|(x)|(x)*|
            ||6.5.0||N/A|(x)|(x)|(x)*|
            ||6.5.1||N/A|N/A|(x)|(x)*|
            ||6.6.0||N/A|N/A|N/A|(/)*|

            \* When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below.

            ---
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}
            h1. TL;DR:
            h1.
            In some versions of Couchbase Server cbbackupmgr will backup tombstones that contain a body. Restoring these tombstone can fail:
             
            +Affected versions+
            The version of cbbackupmgr does not matter. What matters is the version of the cluster that was backed up and the version of the cluster being restored to.

            ||Backup (rows) \ Restore (columns) Versions || <=6.0.4|| 6.5.0||6.5.1||6.6.0||
            ||<=6.0.4||(/)|(x)|(x)|(x)*|
            ||6.5.0||N/A|(x)|(x)|(x)*|
            ||6.5.1||N/A|N/A|(x)|(x)*|
            ||6.6.0||N/A|N/A|N/A|(/)*|

            \* When restoring to Couchbase Server 6.6.0 there is a workaround, which is explained below.

            ---
            +Problem+
            {{cbbackupmgr restore}} fails against Couchbase Server 6.5.x and 6.6.0 as the delete_with_meta is reject as being invalided. On previous versions of Couchbase Server the delete_with_meta is accepted.

            +Notes+

            I can see the point that the delete_with_meta should never take a body, unfortunately DCP on previous versions of Couchbase Server does provide tombstone with bodies. {{cbbbackupmgr}} backs up everything it's given and when it does a restore it will give everything back. This also affects XDCR between Couchbase Server 6.0.4 to 6.6.0. Meaning two of our upgrade processes will not work for 6.6.0.

            Interestingly on 6.6.0 the following config is set:

            {noformat:title=6.6.0}
            # /opt/couchbase/bin/cbstats localhost config -b test -u Administrator -p password | grep allow_del
            ep_allow_del_with_meta_prune_user_data: false
            {noformat}

            There does not seem to be away to see what this is set to on old versions.

            +Steps to reproduce+
            1. Create a document with user xattrs and a 10 second TTL on Couchbase Server 6.0.3
            {noformat}
             /opt/couchbase/bin/cbc-subdoc -U couchbase://localhost/test -u Administrator -P password
             subdoc> set test value -x xattr=100 -e 10
            {noformat}
            2. Wait 10 seconds
            3. Take a backup
            {noformat}
            /opt/couchbase/bin/cbbackupmgr config -a backup -r zombie
            /opt/couchbase/bin/cbbackupmgr backup -a backup -r zombie -c 10.112.194.101 -u Administrator -p password
            {noformat}

            4. From the examine command we can see that the tombstone contains the xattr and a value:
            {noformat}
            /opt/couchbase/bin/cbbackupmgr examine -a backup -r zombie --bucket test --tombstones -k test
            Key: test
              SeqNo: 2
              Backup: 2020-10-29T15_05_45.745791997Z
              Deleted: true
              Size: 4B (key), 29B (meta), 3B (value)
              Meta: {"cas":1603983924472709120,"revseqno":2,"datatype":6}
              Xattrs: {"xattr":100}
              Value: value
            {noformat}
            5. Restore the backup to Couchbase Server 6.6.0
            {noformat}
             /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 2 items / 28.20KB
            [===============================================================================================================================================================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=memcached.log}
            2020-10-29T15:28:25.674599+00:00 INFO 44: HELO [{"a":"gocbcore/v9.0.4 cbbackupmgr-Unknown-5e545ae","i":"97f4cdb1b440b26d/f0ac4f557072da3e"}] XATTR, XERROR, Select bucket, Snappy, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ [::1]:57536 - [::1]:11210 (not authenticated) ]
            2020-10-29T15:28:25.684252+00:00 INFO 44: Client [::1]:57536 authenticated as <ud>Administrator</ud>
            {noformat}

            The restore on 6.5.1 and 6.5.0 also fails:

            {noformat:title=6.5.1 Restore}
            [root@node1-cb660-centos7 vagrant]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            {noformat}

            {noformat:title=6.5.0 Restore}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.200.101 -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 0B/s (about 0s remaining) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Error restoring cluster: invalid argument
            Restore bucket 'test' failed
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 0, Deletions failed to restore: 1
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

            {noformat}

            ---
            Restores to previous versions of Couchbase Server (6.0.X and lower) the restore works.

            The restore on 6.0.4:

            {noformat:title=6.0.4 Restore}
            /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.194.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 156.041335ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            ---

            The xdcr logs from Couchbase Server 6.0.4 where the target is 6.6.0 showing that it's affected:

            {noformat:title=6.0.4 goxdcr.log}
            2020-10-29T16:44:48.605Z ERRO GOXDCR.XmemNozzle: xmem_84576cb22c4dd3a9bd354e17b2186eec/test/test_10.112.205.101:11210_0 received error response from setMeta client. Repairing connection. response status=EINVAL, opcode=0xa8, seqno=1, req.Key=<ud>[116 101 115 116]</ud>, req.Cas=0, req.Cas=0, req.Extras=[0 0 0 0 95 154 227 170 0 0 0 0 0 0 0 2 22 66 126 225 139 162 0 0]
            {noformat}


            +Workaround+

            There is a work around but it only works for 6.6.0:

            {noformat:title=cbepctl workaround}
            # /opt/couchbase/bin/cbepctl localhost:11210 -b test -u Administrator -p password set flush_param allow_del_with_meta_prune_user_data true
            setting param: allow_del_with_meta_prune_user_data true
            set allow_del_with_meta_prune_user_data to true
            {noformat}

            This will allow the restore to work:

            {noformat:title=6.6.0 restore with workaround}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c localhost -u Administrator -p password
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Copied all data in 90.191456ms (Avg. 28.20KB/Sec) 1 items / 28.20KB
            [==================================================================================================================================] 100.00%
            Restore bucket 'test' succeeded
            Mutations restored: 0, Mutations failed to restore: 0
            Deletions restored: 1, Deletions failed to restore: 0
            Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
            Restore completed successfully
            {noformat}

            Unfortunately when the same work around is tried on 6.5.1 cbbackupmgr hangs and memcached produces an exception.

            {noformat:title=6.5.1 restore with workaround hangs}
            # /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.201.101 -u Administrator -p passwo
            rd
            (1/1) Restoring backup 2020-10-29T15_05_45.745791997Z '2020-10-29T15_05_45.745791997Z'
            Transferring key value data for 'test' at 192B/s (about 0s remaining) 1 items / 28.20KB
            [================================================================================================================================== ] 99.81%
            {noformat}

            {noformat:title=memcached exception}
            2020-10-29T18:16:55.484527+00:00 ERROR 45: exception occurred in runloop during packet execution. Cookie info: [{"aiostat":"success","connection":"[ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":57,"cas":0,"datatype":["Snappy","Xattr"],"extlen":30,"key":"<ud>test</ud>","keylen":4,"magic":"ClientRequest","opaque":134217728,"opcode":"DEL_WITH_META","vbucket":127},"refcount":0}] - closing connection ([ 10.112.201.1:51125 - 10.112.201.101:11210 (<ud>Administrator</ud>) ]): Blob::assign failed to inflate. buffer.size:21 uncompressedLength:0
            {noformat}
            Hide
            paolo.cocchi Paolo Cocchi added a comment -

            Patrick Varley allow_del_with_meta_prune_user_data enabled, assigning back to you. Thanks

            Show
            paolo.cocchi Paolo Cocchi added a comment - Patrick Varley allow_del_with_meta_prune_user_data enabled, assigning back to you. Thanks
            paolo.cocchi Paolo Cocchi made changes -
            Assignee Paolo Cocchi [ paolo.cocchi ] Patrick Varley [ pvarley ]
            paolo.cocchi Paolo Cocchi made changes -
            Resolution Done [ 6 ]
            Status Open [ 1 ] Resolved [ 5 ]
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.1-9175 contains kv_engine commit 54794ec with commit message:
            MB-42352: allow_del_with_meta_prune_user_data true by default

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9175 contains kv_engine commit 54794ec with commit message: MB-42352 : allow_del_with_meta_prune_user_data true by default
            Hide
            ashwin.govindarajulu Ashwin Govindarajulu added a comment -

            Seeing following error in memcached log while trying to restore backup on version 6.6.1-9182.

            [root@node1-cb601-centos7 ~]# grep -R "XATTR, XERROR" /opt/couchbase/
            bin/          etc/          lib/          LICENSE.txt   manifest.xml  README.txt    samples/      share/        var/          VARIANT.txt   VERSION.txt
            [root@node1-cb601-centos7 ~]# grep -R "XATTR, XERROR" /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt
            2020-11-12T22:34:34.205412-08:00 INFO 50: HELO [{"a":"java/3.0.8 (Mac OS X 10.15.6 x86_64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08)","i":"4DC2B38A00000001/0000000044555396"}] Mutation seqno, XATTR, XERROR, Select bucket, Snappy, Unordered execution, Tracing, AltRequestSupport, SyncReplication, VAttr, SubdocCreateAsDeleted [ {"ip":"10.112.191.1","port":53878} - {"ip":"10.112.191.101","port":11210} (not authenticated) ]
            2020-11-12T22:34:34.206164-08:00 INFO 55: HELO [{"a":"java/3.0.8 (Mac OS X 10.15.6 x86_64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08)","i":"4DC2B38A00000001/00000000844E2CDF"}] Mutation seqno, XATTR, XERROR, Select bucket, Snappy, Unordered execution, Tracing, AltRequestSupport, SyncReplication, VAttr, SubdocCreateAsDeleted [ {"ip":"10.112.191.1","port":53879} - {"ip":"10.112.191.101","port":11210} (not authenticated) ]
            2020-11-12T23:47:43.597165-08:00 INFO 49: HELO [{"a":"gocbcore/v9.0.7 cbbackupmgr-6.6.1-9182-305333a","i":"332db416ed6b1e65/1204d8b4fe5fcaa9"}] XATTR, XERROR, Select bucket, Snappy, JSON, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ {"ip":"10.112.191.101","port":56770} - {"ip":"10.112.191.101","port":11210} (not authenticated) ]

            Though the restore completed successfully.

            [root@node1-cb601-centos7 ~]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.191.101 -u Administrator -p password 
            (1/1) Restoring backup 2020-11-12T23_45_12.320791066-08_00 '2020-11-12T23_45_12.320791066-08_00' 
            Copied all data in 113.363318ms (Avg. 28.11KB/Sec) 1 items / 28.11KB 
            [====================================================================================================================================================================================================] 100.00% 
            Restore bucket 'test' succeeded 
            Mutations restored: 0, 
            Mutations failed to restore: 0 
            Deletions restored: 1, 
            Deletions failed to restore: 0 
            Skipped due to purge number or conflict resolution: 
            Mutations: 0 Deletions: 0 
            Restore completed successfully 

            Patrick Varley / Paolo Cocchi  is the mc_error log we see is expected ?

            Show
            ashwin.govindarajulu Ashwin Govindarajulu added a comment - Seeing following error in memcached log while trying to restore backup on version 6.6.1-9182. [root@node1-cb601-centos7 ~]# grep -R "XATTR, XERROR" /opt/couchbase/ bin/ etc/ lib/ LICENSE.txt manifest.xml README.txt samples/ share/ var/ VARIANT.txt VERSION.txt [root@node1-cb601-centos7 ~]# grep -R "XATTR, XERROR" /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2020-11-12T22:34:34.205412-08:00 INFO 50: HELO [{"a":"java/3.0.8 (Mac OS X 10.15.6 x86_64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08)","i":"4DC2B38A00000001/0000000044555396"}] Mutation seqno, XATTR, XERROR, Select bucket, Snappy, Unordered execution, Tracing, AltRequestSupport, SyncReplication, VAttr, SubdocCreateAsDeleted [ {"ip":"10.112.191.1","port":53878} - {"ip":"10.112.191.101","port":11210} (not authenticated) ] 2020-11-12T22:34:34.206164-08:00 INFO 55: HELO [{"a":"java/3.0.8 (Mac OS X 10.15.6 x86_64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08)","i":"4DC2B38A00000001/00000000844E2CDF"}] Mutation seqno, XATTR, XERROR, Select bucket, Snappy, Unordered execution, Tracing, AltRequestSupport, SyncReplication, VAttr, SubdocCreateAsDeleted [ {"ip":"10.112.191.1","port":53879} - {"ip":"10.112.191.101","port":11210} (not authenticated) ] 2020-11-12T23:47:43.597165-08:00 INFO 49: HELO [{"a":"gocbcore/v9.0.7 cbbackupmgr-6.6.1-9182-305333a","i":"332db416ed6b1e65/1204d8b4fe5fcaa9"}] XATTR, XERROR, Select bucket, Snappy, JSON, AltRequestSupport, SyncReplication, SubdocCreateAsDeleted [ {"ip":"10.112.191.101","port":56770} - {"ip":"10.112.191.101","port":11210} (not authenticated) ] Though the restore completed successfully. [root@node1-cb601-centos7 ~]# /opt/couchbase/bin/cbbackupmgr restore -a backup -r zombie -c 10.112.191.101 -u Administrator -p password (1/1) Restoring backup 2020-11-12T23_45_12.320791066-08_00 '2020-11-12T23_45_12.320791066-08_00' Copied all data in 113.363318ms (Avg. 28.11KB/Sec) 1 items / 28.11KB [====================================================================================================================================================================================================] 100.00% Restore bucket 'test' succeeded Mutations restored: 0, Mutations failed to restore: 0 Deletions restored: 1, Deletions failed to restore: 0 Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0 Restore completed successfully Patrick Varley / Paolo Cocchi   is the mc_error log we see is expected ?
            Hide
            paolo.cocchi Paolo Cocchi added a comment -

            Ashwin Govindarajulu XERROR (ie, Extended Error) is a memcached feature. HELO logs above mean that those connections support XERROR.

            Show
            paolo.cocchi Paolo Cocchi added a comment - Ashwin Govindarajulu XERROR (ie, Extended Error) is a memcached feature. HELO logs above mean that those connections support XERROR.
            Hide
            ashwin.govindarajulu Ashwin Govindarajulu added a comment -

            Validated using '6.6.1-9182'.

            Closing this ticket.

            Show
            ashwin.govindarajulu Ashwin Govindarajulu added a comment - Validated using '6.6.1-9182'. Closing this ticket.
            ashwin.govindarajulu Ashwin Govindarajulu made changes -
            VERIFICATION STEPS 1. Created doc with ttl=10 on Enterprise Edition 6.0.1 build 2037 cluster
            2. Took backup using 'cbbackupmgr backup' command
            3. Restored on ' 6.6.0 build 7897' node and backup failed with reason 'Deletions failed to restore: 1'
            4. Restored on '6.6.1-9182' and backup succeeded
            Assignee Patrick Varley [ pvarley ] Ashwin Govindarajulu [ ashwin.govindarajulu ]
            Status Resolved [ 5 ] Closed [ 6 ]
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-3860 contains kv_engine commit 54794ec with commit message:
            MB-42352: allow_del_with_meta_prune_user_data true by default

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3860 contains kv_engine commit 54794ec with commit message: MB-42352 : allow_del_with_meta_prune_user_data true by default

              People

              Assignee:
              ashwin.govindarajulu Ashwin Govindarajulu
              Reporter:
              pvarley Patrick Varley
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty