Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46638

[System Test][Backup Service] restore of longevity backups to another cluster failed with error - validOnUid doesn't match the current manifest Uid

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • No

    Description

      7.0.0-3219

      Test:
      -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
      Scale 3
      Iteration 3

      Test has been taking multiple backups since the beginning - multiple merges have gone through successfully as well - I did the following:

      1. Setup another cluster with same config
      2. Created the buckets and scopes in that cluster
      3. Tried to restore backups from .108.103 to this cluster - failed with following error:

      2021-05-28T11:04:48.685-07:00 INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr restore -c http://172.23.104.232:8091/ -a /data/archive -r ed601e0c-fafe-45d1-87df-553bf9936783 --auto-select-threads --json-progress 15456001-190c-48ec-ba72-36bbdca300d3 --start 2021-05-27T10_00_10.505185696-07_00 --end 2021-05-28T10_00_12.536138636-07_00
      		 
      2021-05-28T11:04:48.876-07:00 DEBUG (Leader Manager) Received store event {"eventType": 3}
      		 
      2021-05-28T11:04:48.876-07:00 DEBUG (Leader Manager) Received store event {"eventType": 3}
      		 
      2021-05-28T11:04:58.688-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:04:58.688-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-865e9620-6d9e-4caf-8936-23876d765819", "status": "running"}
      		 
      2021-05-28T11:05:08.688-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:05:08.688-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-865e9620-6d9e-4caf-8936-23876d765819", "status": "running"}
      		 
      2021-05-28T11:05:18.688-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:05:18.689-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-865e9620-6d9e-4caf-8936-23876d765819", "status": "running"}
      		 
      2021-05-28T11:05:28.688-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:05:28.688-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-865e9620-6d9e-4caf-8936-23876d765819", "status": "running"}
      		 
      2021-05-28T11:05:38.689-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:05:38.689-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-865e9620-6d9e-4caf-8936-23876d765819", "status": "running"}
      		 
      2021-05-28T11:05:48.688-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:05:48.688-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-865e9620-6d9e-4caf-8936-23876d765819", "status": "running"}
      		 
      2021-05-28T11:05:58.695-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:05:58.695-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-865e9620-6d9e-4caf-8936-23876d765819", "status": "running"}
      		 
      2021-05-28T11:06:00.776-07:00 INFO (Worker) Command exited {"err": "exit status 1", "cluster": "self", "repositoryID": "my_repo", "state": "active", "taskName": "restore-865e9620-6d9e-4caf-8936-23876d765819"}
      		 
      2021-05-28T11:06:00.777-07:00 WARN (Worker) Task failed {"cluster": "self", "repositoryID": "my_repo", "state": "active", "taskName": "restore-865e9620-6d9e-4caf-8936-23876d765819", "err": "exit status 1", "cbmErr": "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'NEW_ORDER': failed to automatically create collections: failed to update scopes/collections: failed to update cluster collection manifest: failed to execute request: {\"errors\":{\"_\":\"validOnUid doesn't match the current manifest Uid\"}}"}

      4. Dropped all the buckets in target cluster - tried to restore with auto-create buckets option - again failed with error:

      2021-05-28T11:09:52.903-07:00 INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr restore -c http://172.23.104.232:8091/ -a /data/archive -r ed601e0c-fafe-45d1-87df-553bf9936783 --auto-select-threads --json-progress 2d6434ab-ffe4-40a2-84f2-db821ff98b17 --start 2021-05-27T10_00_10.505185696-07_00 --end 2021-05-28T10_00_12.536138636-07_00 --auto-create-buckets
      		 
      2021-05-28T11:09:53.025-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:09:53.025-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "waiting"}
      		 
      2021-05-28T11:09:53.025-07:00 DEBUG (Leader Manager) Received store event {"eventType": 3}
      		 
      2021-05-28T11:09:53.026-07:00 DEBUG (Leader Manager) Received store event {"eventType": 3}
      		 
      2021-05-28T11:10:02.909-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:10:02.909-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:10:12.906-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:10:12.907-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:10:22.906-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:10:22.906-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:10:32.906-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:10:32.906-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:10:42.906-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:10:42.906-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:10:52.906-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:10:52.906-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:11:02.906-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:11:02.906-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:11:12.907-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:11:12.907-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:11:22.912-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:11:22.912-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:11:32.920-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:11:32.920-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:11:37.146-07:00 INFO (Stats) Start repositories data size collection
      		 
      2021-05-28T11:11:37.239-07:00 INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info -j -a /data/archive --offset 0 --limit 0 -r ed601e0c-fafe-45d1-87df-553bf9936783
      		 
      2021-05-28T11:11:38.435-07:00 INFO (Stats) Stop repositories data size collection
      		 
      2021-05-28T11:11:42.906-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:11:42.906-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:11:52.907-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:11:52.907-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:12:02.907-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:12:02.907-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:12:12.907-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:12:12.907-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:12:22.907-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:12:22.907-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:12:32.908-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:12:32.908-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:12:42.908-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:12:42.908-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:12:52.910-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:12:52.910-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:13:02.908-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:13:02.908-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:13:12.909-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:13:12.909-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:13:22.909-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:13:22.909-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:13:32.909-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:13:32.910-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:13:42.909-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:13:42.909-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:13:52.908-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:13:52.909-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:14:02.909-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:14:02.909-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:14:12.909-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:14:12.909-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:14:22.909-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-28T11:14:22.909-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "status": "running"}
      		 
      2021-05-28T11:14:29.788-07:00 INFO (Worker) Command exited {"err": "exit status 1", "cluster": "self", "repositoryID": "my_repo", "state": "active", "taskName": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be"}
      		 
      2021-05-28T11:14:29.789-07:00 WARN (Worker) Task failed {"cluster": "self", "repositoryID": "my_repo", "state": "active", "taskName": "restore-b19a059a-bb60-40d6-bd1d-ec83ad7c21be", "err": "exit status 1", "cbmErr": "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'NEW_ORDER': failed to automatically create collections: failed to update scopes/collections: failed to update cluster collection manifest: failed to execute request: {\"errors\":{\"_\":\"validOnUid doesn't match the current manifest Uid\"}}"}
      		 
      2021-05-28T11:14:29.791-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}

      Logs:
      source:
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.104.155.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.104.157.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.104.5.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.104.69.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.104.70.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.105.107.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.106.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.106.188.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.108.103.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.120.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.121.117.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.121.3.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.123.27.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.123.28.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.96.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.96.251.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.96.252.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.96.253.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.97.119.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.97.121.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.97.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.97.239.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.97.242.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.98.135.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.99.11.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1622227009/collectinfo-2021-05-28T183651-ns_1%40172.23.99.20.zip

      Attaching backup logs and target logs

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            arunkumar Arunkumar Senthilnathan (Inactive)
            arunkumar Arunkumar Senthilnathan (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty