Details
-
Bug
-
Resolution: Duplicate
-
Major
-
None
-
7.1.0
-
Untriaged
-
1
-
No
Description
7.1.0-1701
Test:
-test tests/integration/neo/test_neo_couchstore_milestone3.yml -scope tests/integration/neo/scope_couchstore.yml
Scale 3
Iteration 2
Backup task failed with error:
{
|
"task_name": "backup-1",
|
"status": "failed",
|
"start": "2021-11-14T18:00:46.227721654-08:00",
|
"end": "2021-11-14T18:01:48.763172845-08:00",
|
"node_runs": [
|
{
|
"node_id": "d46cdb27f27c27a7560c1f113f512dc5",
|
"status": "failed",
|
"start": "2021-11-14T18:00:46.332055828-08:00",
|
"end": "2021-11-14T18:01:48.700037387-08:00",
|
"error": "exit status 1: failed to get backup transferable: failed to create backup: failed to get source collection manifest: failed to wait for manifest id 1468: timeout after scopes/collections (for manifest id 5bc) to be ready for use waiting for 1m0s, please try again later",
|
"progress": 0,
|
"stats": {
|
"error": "failed to get backup transferable: failed to create backup: failed to get source collection manifest: failed to wait for manifest id 1468: timeout after scopes/collections (for manifest id 5bc) to be ready for use waiting for 1m0s, please try again later"
|
},
|
"error_code": 2
|
}
|
],
|
"error": "exit status 1: failed to get backup transferable: failed to create backup: failed to get source collection manifest: failed to wait for manifest id 1468: timeout after scopes/collections (for manifest id 5bc) to be ready for use waiting for 1m0s, please try again later",
|
"error_code": 2,
|
"type": "BACKUP",
|
"show": true
|
}
|
During this time, a kv rebalance was stuck in the test:
[2021-11-13T13:26:35-08:00, sequoiatools/couchbase-cli:7.1:c2fa80] server-add -c 172.23.108.103:8091 --server-add https://172.23.104.157 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
[2021-11-13T13:26:49-08:00, sequoiatools/couchbase-cli:7.1:480316] rebalance -c 172.23.108.103:8091 -u Administrator -p password
We have filed MB-49542 for the rebalance stuck issue but filing this to make sure the error is due to this
Cluster config:
########## Cluster config ##################
|
###### backup : 1 ===== > [172.23.123.28:8091] ###########
|
###### n1ql : 2 ===== > [172.23.96.251:8091 172.23.99.11:8091] ###########
|
###### eventing : 3 ===== > [172.23.104.137:8091 172.23.123.27:8091 172.23.98.135:8091] ###########
|
###### fts : 2 ===== > [172.23.104.155:8091 172.23.104.157:8091] ###########
|
###### kv : 11 ===== > [172.23.104.5:8091 172.23.106.100:8091 172.23.108.103:8091 172.23.96.148:8091 172.23.97.119:8091 172.23.97.121:8091 172.23.97.122:8091 172.23.97.239:8091 172.23.99.20:8091 172.23.99.21:8091 172.23.99.25:8091] ###########
|
###### index : 6 ===== > [172.23.104.67:8091 172.23.104.69:8091 172.23.120.245:8091 172.23.121.117:8091 172.23.96.252:8091 172.23.96.253:8091] ###########
|
###### cbas : 3 ===== > [172.23.106.188:8091 172.23.121.3:8091 172.23.97.242:8091] ###########
|
Attaching cbm collect logs - server logs here:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.104.137.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.104.155.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.104.157.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.104.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.104.67.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.104.69.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.104.70.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.106.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.106.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.108.103.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.120.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.121.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.121.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.123.27.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.123.28.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.96.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.96.251.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.96.252.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.96.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.97.119.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.97.121.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.97.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.97.239.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.97.242.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.98.135.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.99.11.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.99.20.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.99.21.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.99.25.zip
Attachments
Issue Links
- duplicates
-
MB-48637 [System Test][CBM] backup task failed with error - timeout reached whilst waiting for manifest id xxx to have propagated throughout the cluster, waited 1m0s
- Closed