Details
-
Bug
-
Resolution: Duplicate
-
Major
-
None
-
7.1.0
-
Untriaged
-
1
-
No
Description
While I was running toy build as requested in MB-48856, observed that backup tasks were failing with the following error:
{
|
"task_name": "backup-1",
|
"status": "failed",
|
"start": "2021-11-04T04:00:49.708150308-07:00",
|
"end": "2021-11-04T04:01:52.251689926-07:00",
|
"node_runs": [
|
{
|
"node_id": "765727ae7d9be639ca3a78ef66fbf697",
|
"status": "failed",
|
"start": "2021-11-04T04:00:49.733209487-07:00",
|
"end": "2021-11-04T04:01:52.233147064-07:00",
|
"error": "exit status 1: failed to get backup transferable: failed to create backup: failed to get source collection manifest: failed to wait for manifest id 291: timeout reached whilst waiting for manifest id 123 to have propagated throughout the cluster, waited 1m0s",
|
"progress": 0,
|
"stats": {
|
"error": "failed to get backup transferable: failed to create backup: failed to get source collection manifest: failed to wait for manifest id 291: timeout reached whilst waiting for manifest id 123 to have propagated throughout the cluster, waited 1m0s"
|
},
|
"error_code": 2
|
}
|
],
|
"error": "exit status 1: failed to get backup transferable: failed to create backup: failed to get source collection manifest: failed to wait for manifest id 291: timeout reached whilst waiting for manifest id 123 to have propagated throughout the cluster, waited 1m0s",
|
"error_code": 2,
|
"type": "BACKUP",
|
"show": true
|
}
|
Cluster config:
########## Cluster config ##################
|
###### kv : 12 ===== > [172.23.120.73:8091 172.23.120.77:8091 172.23.120.86:8091 172.23.121.77:8091 172.23.123.24:8091 172.23.123.25:8091 172.23.123.26:8091 172.23.96.122:8091 172.23.96.14:8091 172.23.96.48:8091 172.23.97.241:8091 172.23.97.74:8091] ###########
|
###### backup : 1 ===== > [172.23.120.74:8091] ###########
|
Error in backup service node 172.23.120.74:
2021-11-04T04:00:59.736-07:00 WARN (Worker) No progress given by cbbackupmgr {"cluster": "self", "repositoryID": "my_repo", "state": "active", "taskName": "backup-1"}
|
2021-11-04T04:01:09.736-07:00 WARN (Worker) No progress given by cbbackupmgr {"cluster": "self", "repositoryID": "my_repo", "state": "active", "taskName": "backup-1"}
|
2021-11-04T04:01:19.736-07:00 WARN (Worker) No progress given by cbbackupmgr {"cluster": "self", "repositoryID": "my_repo", "state": "active", "taskName": "backup-1"}
|
2021-11-04T04:01:29.736-07:00 WARN (Worker) No progress given by cbbackupmgr {"cluster": "self", "repositoryID": "my_repo", "state": "active", "taskName": "backup-1"}
|
2021-11-04T04:01:39.736-07:00 WARN (Worker) No progress given by cbbackupmgr {"cluster": "self", "repositoryID": "my_repo", "state": "active", "taskName": "backup-1"}
|
2021-11-04T04:01:49.736-07:00 WARN (Worker) No progress given by cbbackupmgr {"cluster": "self", "repositoryID": "my_repo", "state": "active", "taskName": "backup-1"}
|
2021-11-04T04:01:52.233-07:00 WARN (Worker) Task failed {"cluster": "self", "repositoryID": "my_repo", "state": "active", "taskName": "backup-1", "err": "exit status 1", "cbmErr": "exit status 1: failed to get backup transferable: failed to create backup: failed to get source collection manifest: failed to wait for manifest id 291: timeout reached whilst waiting for manifest id 123 to have propagated throughout the cluster, waited 1m0s"}
|
Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636027174/collectinfo-2021-11-04T115935-ns_1%40172.23.120.74.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636027174/collectinfo-2021-11-04T115935-ns_1%40172.23.120.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636027174/collectinfo-2021-11-04T115935-ns_1%40172.23.120.86.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636027174/collectinfo-2021-11-04T115935-ns_1%40172.23.121.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636027174/collectinfo-2021-11-04T115935-ns_1%40172.23.123.24.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636027174/collectinfo-2021-11-04T115935-ns_1%40172.23.123.25.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636027174/collectinfo-2021-11-04T115935-ns_1%40172.23.123.26.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636027174/collectinfo-2021-11-04T115935-ns_1%40172.23.96.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636027174/collectinfo-2021-11-04T115935-ns_1%40172.23.97.241.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636027174/collectinfo-2021-11-04T115935-ns_1%40172.23.97.74.zip
