[System Test][CBM] backup task failed with error - Get "http://172.23.105.168:8095/api/v1/backup": net/http: timeout awaiting response headers

7.1.1-3094

Test:
-test tests/integration/neo/test_neo.yml -scope tests/integration/neo/scope_couchstore.yml
Iteration 2

Backup task failed with error:

{
"task_name": "backup-1",
"status": "failed",
"start": "2022-05-27T15:00:28.525258766-07:00",
"end": "2022-05-27T15:00:41.469807054-07:00",
"node_runs": [
{
"node_id": "2a8c16feb35d1fd887bdfd6abdfee75c",
"status": "failed",
"start": "2022-05-27T15:00:28.643577077-07:00",
"end": "2022-05-27T15:00:41.370344548-07:00",
"error": "exit status 1: failed to execute cluster operations: failed to transfer Analytics metadata: failed to get Analytics metadata: failed to get analytics metadata: failed to execute request: failed to execute request: failed to perform request: Get \"http://172.23.105.168:8095/api/v1/backup\": net/http: timeout awaiting response headers",
"progress": 0,
"stats": {
"error": "failed to execute cluster operations: failed to transfer Analytics metadata: failed to get Analytics metadata: failed to get analytics metadata: failed to execute request: failed to execute request: failed to perform request: Get \"http://172.23.105.168:8095/api/v1/backup\": net/http: timeout awaiting response headers",
"stats":

{ "started_at": 1653688829546841300 }

},
"error_code": 2
}
],
"error": "exit status 1: failed to execute cluster operations: failed to transfer Analytics metadata: failed to get Analytics metadata: failed to get analytics metadata: failed to execute request: failed to execute request: failed to perform request: Get \"http://172.23.105.168:8095/api/v1/backup\": net/http: timeout awaiting response headers",
"error_code": 2,
"type": "BACKUP",
"show": true
}

During this time, the test was dropping some cbas indexes:

[2022-05-27T15:24:58-07:00, sequoiatools/analyticsmanager:1.0:9d7c0b] -i 172.23.106.188 -b default,bucket1,bucket4,bucket5,bucket6,bucket7 -o drop_cbas_infra --api_timeout 3600
[2022-05-27T17:28:48-07:00, sequoiatools/couchbase-cli:7.1:6c0351] server-add -c 172.23.108.103:8091 --server-add https://172.23.104.70 -u Administrator -p password --server-add-username Administrator --server-add-password password --services analytics

CBM logs:

2022-05-27T15:00:31.171-07:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.104.69:8091/pools/default/buckets/bucket9'
2022-05-27T15:00:31.244-07:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.104.69:8091/pools/default/buckets/bucket9'
2022-05-27T15:00:31.245-07:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.104.69:8091/pools/default/buckets/bucket9/scopes'
2022-05-27T15:00:31.246-07:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.104.69:8091/pools/default/buckets/bucket9/scopes'
2022-05-27T15:00:31.247-07:00 (REST) (Attempt 1) (POST) Dispatching request to 'http://172.23.104.69:8091/pools/default/buckets/bucket9/scopes/@ensureManifest/15b'
2022-05-27T15:00:31.349-07:00 (REST) (Attempt 1) (POST) (200) Received response from 'http://172.23.104.69:8091/pools/default/buckets/bucket9/scopes/@ensureManifest/15b'
2022-05-27T15:00:31.350-07:00 (Archive) (Repo) Creating new backup '2022-05-27T15_00_31.350446065-07_00'
2022-05-27T15:00:31.360-07:00 (Stats) Starting stat gathering - stat timestamp: 1653688831
2022-05-27T15:00:31.360-07:00 (Plan) (Cluster) Transferring cluster configuration
2022-05-27T15:00:31.361-07:00 (Plan) (Cluster) Successfully transferred cluster configuration |

{"number":1,"duration":"965.233µs"}

2022-05-27T15:00:31.364-07:00 (Plan) (Analytics) Transferring Analytics metadata
2022-05-27T15:00:31.365-07:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.105.168:8095/api/v1/backup'
2022-05-27T15:00:41.367-07:00 ERRO: (REST) (Attempt 1) (GET) Failed to perform request to 'http://172.23.105.168:8095/api/v1/backup': Get "http://172.23.105.168:8095/api/v1/backup": net/http: timeout awaiting response headers – logging.(*ToolsCommonLogger).Log() at tools_common.go:30
2022-05-27T15:00:41.367-07:00 WARN: (REST) (Attempt 1) (GET) Request to endpoint '/api/v1/backup' failed due to error: failed to perform request: Get "http://172.23.105.168:8095/api/v1/backup": net/http: timeout awaiting response headers – logging.(*ToolsCommonLogger).Log() at tools_common.go:28
2022-05-27T15:00:41.367-07:00 (Cmd) Error backing up cluster: failed to execute cluster operations: failed to transfer Analytics metadata: failed to get Analytics metadata: failed to get analytics metadata: failed to execute request: failed to execute request: failed to perform request: Get "http://172.23.105.168:8095/api/v1/backup": net/http: timeout awaiting response headers
2022-05-27T21:00:32.339-07:00 (Cmd) cbbackupmgr version 7.1.1-3094 Hostname: cnt7-qe1 OS: linux Version: 3.10.0-693.5.2.el7.x86_64 Arch: amd64 vCPU: 8 Memory: 25107591168 (23.38GiB)
2022-05-27T21:00:32.339-07:00 (Cmd) backup --cluster http://172.23.104.69:8091 --archive /data/archive --repo 2b93f7e0-f20f-4bbc-8be5-f77b0244f4ab --purge --auto-select-threads --json-progress 00e1f76b-6537-4c5b-8463-0f2dabe48377 --full-backup
2022-05-27T21:00:32.339-07:00 (Cmd) mounted archive with id: b9258679-c56a-4524-8ca7-0a45f6b63aeb
2022-05-27T21:00:32.339-07:00 (Cmd) Automatically set the number of threads to 6

Server logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.104.137.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.104.155.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.104.157.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.104.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.104.67.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.104.69.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.104.70.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.105.107.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.105.111.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.105.168.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.106.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.106.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.108.103.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.120.107.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.120.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.121.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.123.28.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.96.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.96.192.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.96.251.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.96.252.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.96.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.97.119.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.97.121.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.97.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.97.239.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.99.20.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.99.21.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.99.25.zip