Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Untriaged
-
1
-
No
-
CX Sprint 248, CX Sprint 249
Description
7.0.0-3219
Test:
-test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
Scale 3
Iteration 3
Test has been taking multiple backups since the beginning - multiple merges have gone through successfully as well - I did the following:
1. Aborted the test - stopped a rebalance that was in progress manually
2. Killed all the data loading docker containers created by the test
3. Tried to restore the backups taken on to the same cluster - failed with error:
2021-05-28T11:54:58.382-07:00 INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr restore -c http://172.23.108.103/ -a /data/archive -r ed601e0c-fafe-45d1-87df-553bf9936783 --auto-select-threads --json-progress 7b17a6d1-5ac3-4f6d-9d15-4f847bd5230b --start 2021-05-27T10_00_10.505185696-07_00 --end 2021-05-28T10_00_12.536138636-07_00
|
2021-05-28T11:54:58.541-07:00 DEBUG (Leader Manager) Received store event {"eventType": 3}
|
2021-05-28T11:54:58.541-07:00 DEBUG (Leader Manager) Received store event {"eventType": 3}
|
2021-05-28T11:55:08.386-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
|
2021-05-28T11:55:08.386-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-c55f0fd2-508a-42d4-9fc2-7a93e234a5fa", "status": "running"}
|
2021-05-28T11:55:18.385-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
|
2021-05-28T11:55:18.385-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-c55f0fd2-508a-42d4-9fc2-7a93e234a5fa", "status": "running"}
|
2021-05-28T11:55:28.385-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
|
2021-05-28T11:55:28.386-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-c55f0fd2-508a-42d4-9fc2-7a93e234a5fa", "status": "running"}
|
2021-05-28T11:55:38.399-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
|
2021-05-28T11:55:38.399-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-c55f0fd2-508a-42d4-9fc2-7a93e234a5fa", "status": "running"}
|
2021-05-28T11:55:48.397-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
|
2021-05-28T11:55:48.397-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-c55f0fd2-508a-42d4-9fc2-7a93e234a5fa", "status": "running"}
|
2021-05-28T11:55:58.386-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
|
2021-05-28T11:55:58.386-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo", "task": "restore-c55f0fd2-508a-42d4-9fc2-7a93e234a5fa", "status": "running"}
|
2021-05-28T11:56:03.816-07:00 INFO (Worker) Command exited {"err": "exit status 1", "cluster": "self", "repositoryID": "my_repo", "state": "active", "taskName": "restore-c55f0fd2-508a-42d4-9fc2-7a93e234a5fa"}
|
2021-05-28T11:56:03.816-07:00 WARN (Worker) Task failed {"cluster": "self", "repositoryID": "my_repo", "state": "active", "taskName": "restore-c55f0fd2-508a-42d4-9fc2-7a93e234a5fa", "err": "exit status 1", "cbmErr": "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket6': failed to transfer Analytics for bucket 'bucket6': failed to set Analytics metadata: failed to post analytics metadata: failed to execute request: http client failed to dispatch/receive request/response: Post \"http://172.23.105.107:8095/api/v1/bucket/bucket6/backup\": context deadline exceeded"}
|
2021-05-28T11:56:03.818-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
|
Attaching backup logs - will post the server logs once collected
Attachments
Issue Links
| For Gerrit Dashboard: MB-46639 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 154730,6 | MB-46639: only suspend aggregated listener for DDL on active link | 7.0.0 | cbas-core | Status: MERGED | +2 | +1 |
| 154731,1 | MB-46639: Disconnect only the bucket being restored | 7.0.0 | cbas-core | Status: ABANDONED | 0 | -1 |
| 154900,3 | MB-46639 Increase analytics POST timeout | cheshire-cat | backup | Status: MERGED | +2 | +1 |
| 154913,1 | MB-46639 Increase analytics POST timeout | mad-hatter | backup | Status: ABANDONED | 0 | 0 |
| 154951,2 | Merge branch 'cheshire-cat' into master | master | backup | Status: MERGED | +2 | +1 |
| 155277,2 | MB-46639 Increase analytics timeout | master | backup | Status: ABANDONED | 0 | +1 |
| 155401,4 | MB-46639 Increase analytics POST timeout | cheshire-cat | backup | Status: MERGED | +2 | +1 |
| 155406,3 | Merge branch 'cheshire-cat' into master | master | backup | Status: MERGED | +2 | +1 |