[System Test][CBM] Backup task failed: failed to get sequence numbers: exhausted retry count after 5 retries

7.1.0-1361

Test:
-test tests/integration/neo/test_neo_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3_magma.yml -scope tests/integration/neo/scope_neo_with_backup_magma.yml
Scale 3
Iteration 1

Backup task failed with error:

172.23.123.28 : backup

2021-09-27T23:51:33.886-07:00 WARN (Worker) Task failed {"cluster": "self", "repositoryID": "my_repo", "state": "active", "taskName": "backup-1", "err": "exit status 1", "cbmErr": "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'NEW_ORDER': failed to transfer bucket data for bucket 'NEW_ORDER': failed to transfer data ranges: failed to get source data ranges: failed to get sequence numbers: exhausted retry count after 5 retries"}

Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.104.137.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.104.155.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.104.157.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.104.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.104.67.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.104.69.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.104.70.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.105.107.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.106.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.106.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.108.103.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.120.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.121.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.121.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.123.27.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.123.28.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.96.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.96.251.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.96.252.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.96.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.97.119.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.97.121.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.97.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.97.239.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.97.242.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.98.135.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.99.11.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.99.20.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.99.21.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1632815976/collectinfo-2021-09-28T075938-ns_1%40172.23.99.25.zip

Attaching cbm logs