Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-41830

cbbackupmgr fails to wait until manifest update has propagated throughout the cluster

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Yes

    Description

      Last good build: 7.0.0-3265

      Issue found in: 7.0.0-3268

      Test:
      python3 testrunner.py -i ~/4node-sanity.ini -t ent_backup_restore.enterprise_bkrs_collection.EnterpriseBackupRestoreCollectionTest.test_backup_restore_collection_sanity,items=1000,map_data_collection=true,number_of_backups=2,load_to_collection=true,skip_cleanup=True

      Console error:

      2020-10-01 19:37:59,280 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7
      2020-10-01 19:37:59,280 - root - INFO - running command.raw on 172.23.104.136:   /opt/couchbase/bin//cbbackupmgr restore --archive /tmp/entbackup_172.23.104.105 --repo backup --cluster http://172.23.104.105:8091 --username Administrator  --password password  --start 2020-10-02T02_32_52.96737095Z --end 2020-10-02T02_33_17.196535998Z --no-progress-bar --map-data default.scope1=default.scope1,default.scope0=default.scope0
      2020-10-01 19:37:59,905 - root - INFO - command executed successfully with root
      2020-10-01 19:37:59,905 - root - INFO - Warning: `--map-data` option only works for the Data service. Will not restore the metadata for any other service
      2020-10-01 19:37:59,905 - root - INFO - Error restoring cluster: collection not found
      2020-10-01 19:37:59,905 - root - INFO - Restore bucket 'default' failed
      2020-10-01 19:37:59,905 - root - INFO - Mutations restored: 2, Mutations failed to restore: 1
      2020-10-01 19:37:59,905 - root - INFO - Deletions restored: 0, Deletions failed to restore: 0
      2020-10-01 19:37:59,905 - root - INFO - Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
      2020-10-01 19:37:59,964 - root - INFO - Node version in cluster 7.0.0-3268-enterprise
      

      backup.log:

      2020-10-02T02:37:59.668+00:00 (Plan) Transferring new key value data from default to default
      2020-10-02T02:37:59.668+00:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.104.105:8091/pools/default/buckets/default'
      2020-10-02T02:37:59.674+00:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.104.105:8091/pools/default/buckets/default'
      2020-10-02T02:37:59.738+00:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.104.105:8091/pools/default/buckets/default'
      2020-10-02T02:37:59.744+00:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.104.105:8091/pools/default/buckets/default'
      2020-10-02T02:37:59.744+00:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.104.105:8091/pools/default/buckets/default/collections'
      2020-10-02T02:37:59.745+00:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.104.105:8091/pools/default/buckets/default/collections'
      2020-10-02T02:37:59.785+00:00 (default) (vb 191) Skipping byseq start: 0 end: 0 last
      2020-10-02T02:37:59.785+00:00 (Scanworker) (vb 47) Transferring data seqno start 0 seqno end 9
      2020-10-02T02:37:59.790+00:00 (Scanworker) (vb 973) Transferring data seqno start 0 seqno end 7
      2020-10-02T02:37:59.793+00:00 (Scanworker) (vb 16) Transferring data seqno start 0 seqno end 7
      2020-10-02T02:37:59.794+00:00 (default) (vb 512) Skipping byseq start: 0 end: 0 last
      2020-10-02T02:37:59.795+00:00 WARN: (Pool) (default) Failed to send document with key '<ud>pymc658</ud>' because collection with id 9 does not exist -- couchbase.(*MemcachedWorker).processOperation() at pool_worker.go:346
      2020-10-02T02:37:59.796+00:00 ERRO: (Scanworker) (vb 16) Failed to perform 'FullFileScan': failed to perform full scan: failed to perform full scan on chunk 0 for vBucket 16: mutation callback failed: collection not found | {"status_code":136,"bucket":"default","collection_id":9,"error_name":"UNKNOWN_COLLECTION","error_description":"Operation specified an unknown collection.","opaque":7,"context":"Unknown scope or collection in operation","last_dispatched_to":"172.23.104.107:11210","last_dispatched_from":"172.23.104.136:40316","last_connection_id":"82f9131a6e022f56/84c835312b65e526"} -- archive.scanWorker.func1() at source.go:314
      2020-10-02T02:37:59.797+00:00 (Stats) Stopping stat collection
      2020-10-02T02:37:59.800+00:00 (Cmd) Error restoring cluster: failed to execute cluster operations: failed to execute bucket operations: failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to perform full scan: failed to perform full scan on chunk 0 for vBucket 16: mutation callback failed: collection not found | {"status_code":136,"bucket":"default","collection_id":9,"error_name":"UNKNOWN_COLLECTION","error_description":"Operation specified an unknown collection.","opaque":7,"context":"Unknown scope or collection in operation","last_dispatched_to":"172.23.104.107:11210","last_dispatched_from":"172.23.104.136:40316","last_connection_id":"82f9131a6e022f56/84c835312b65e526"}
      2020-10-02T02:37:59.800+00:00 (Cmd)  Restore bucket 'default' failed
      2020-10-02T02:37:59.800+00:00 (Cmd)  Mutations restored: 2, Mutations failed to restore: 1
      2020-10-02T02:37:59.800+00:00 (Cmd)  Deletions restored: 0, Deletions failed to restore: 0
      2020-10-02T02:37:59.800+00:00 (Cmd)  Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
      

      Change log:
      MB-38316 Remove old collection create/delete functionality
      MB-38316 Blackhole sink will now bulk update the manifest
      MB-38316 Refactor how we wait for manifest changes to propagate
      MB-38316 Use the bulk update API to auto create/delete collections

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty