[System Test][CBM] backup task failed with error - internal server error executing 'POST' request to '/pools/default/buckets/bucket1/scopes/@ensureManifest/5': ["Unexpected server error, request logged."]

7.1.1-3166

Test:
-test tests/integration/neo/test_neo_couchstore_milestone4.yml -scope tests/integration/neo/scope_couchstore.yml
Iteration 4

Backup task failed with error:

{
"task_name": "backup-1",
"status": "failed",
"start": "2022-06-19T17:00:09.584920925-07:00",
"end": "2022-06-19T17:02:41.452765401-07:00",
"node_runs": [
{
"node_id": "e675e09af6be26f9a63f2e4490d3ec11",
"status": "failed",
"start": "2022-06-19T17:00:09.675589238-07:00",
"end": "2022-06-19T17:02:41.38737722-07:00",
"error": "exit status 1: failed to get backup transferable: failed to create backup: failed to get source collection manifest: failed to wait for manifest id 5: failed to execute request: internal server error executing 'POST' request to '/pools/default/buckets/bucket1/scopes/@ensureManifest/5': [\"Unexpected server error, request logged.\"]",
"progress": 0,
"stats":

{ "error": "failed to get backup transferable: failed to create backup: failed to get source collection manifest: failed to wait for manifest id 5: failed to execute request: internal server error executing 'POST' request to '/pools/default/buckets/bucket1/scopes/@ensureManifest/5': [\"Unexpected server error, request logged.\"]" }

,
"error_code": 2
}
],
"error": "exit status 1: failed to get backup transferable: failed to create backup: failed to get source collection manifest: failed to wait for manifest id 5: failed to execute request: internal server error executing 'POST' request to '/pools/default/buckets/bucket1/scopes/@ensureManifest/5': [\"Unexpected server error, request logged.\"]",
"error_code": 2,
"type": "BACKUP",
"show": true
}

During this time, test was creating some fts indexes:

[2022-06-20T17:00:44-07:00, sequoiatools/indexmanager:7529e9] -n 172.23.108.103 -o 8091 -u Administrator -p password -b bucket7 -a enable_cbo --cbo_enable_ratio=50
[2022-06-20T17:00:49-07:00, sequoiatools/cmd:78801b] 60
[2022-06-20T17:01:55-07:00, sequoiatools/ftsindexmanager:79ebf6] -n 172.23.96.148 -o 8091 -u Administrator -p password -b bucket4 -m 1:1:2 -s 3 -a create_index_from_map
[2022-06-20T17:02:04-07:00, sequoiatools/cmd:abe3d7] 300
[2022-06-20T17:07:14-07:00, sequoiatools/ftsindexmanager:bbb037] -n 172.23.96.148 -o 8091 -u Administrator -p password -b bucket5 -m 1:0:5 -s 3 -a create_index_from_map
[2022-06-20T17:07:23-07:00, sequoiatools/cmd:0ae245] 300

Attaching backup logs

Server logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.104.137.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.104.155.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.104.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.104.67.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.104.69.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.104.70.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.105.107.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.105.111.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.105.168.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.106.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.106.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.108.103.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.120.107.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.120.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.121.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.123.28.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.96.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.96.192.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.96.251.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.96.252.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.96.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.97.121.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.97.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.97.239.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.99.11.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.99.21.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1655687500/collectinfo-2022-06-20T011142-ns_1%40172.23.99.25.zip