Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-52401

[System Test][CBM] backup task failed with error - Get "http://172.23.105.168:8095/api/v1/backup": net/http: timeout awaiting response headers

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Yes

    Description

      7.1.1-3094

      Test:
      -test tests/integration/neo/test_neo.yml -scope tests/integration/neo/scope_couchstore.yml
      Iteration 2

      Backup task failed with error:

      {
      "task_name": "backup-1",
      "status": "failed",
      "start": "2022-05-27T15:00:28.525258766-07:00",
      "end": "2022-05-27T15:00:41.469807054-07:00",
      "node_runs": [
      {
      "node_id": "2a8c16feb35d1fd887bdfd6abdfee75c",
      "status": "failed",
      "start": "2022-05-27T15:00:28.643577077-07:00",
      "end": "2022-05-27T15:00:41.370344548-07:00",
      "error": "exit status 1: failed to execute cluster operations: failed to transfer Analytics metadata: failed to get Analytics metadata: failed to get analytics metadata: failed to execute request: failed to execute request: failed to perform request: Get \"http://172.23.105.168:8095/api/v1/backup\": net/http: timeout awaiting response headers",
      "progress": 0,
      "stats": {
      "error": "failed to execute cluster operations: failed to transfer Analytics metadata: failed to get Analytics metadata: failed to get analytics metadata: failed to execute request: failed to execute request: failed to perform request: Get \"http://172.23.105.168:8095/api/v1/backup\": net/http: timeout awaiting response headers",
      "stats":

      { "started_at": 1653688829546841300 }

      },
      "error_code": 2
      }
      ],
      "error": "exit status 1: failed to execute cluster operations: failed to transfer Analytics metadata: failed to get Analytics metadata: failed to get analytics metadata: failed to execute request: failed to execute request: failed to perform request: Get \"http://172.23.105.168:8095/api/v1/backup\": net/http: timeout awaiting response headers",
      "error_code": 2,
      "type": "BACKUP",
      "show": true
      }

      During this time, the test was dropping some cbas indexes:

      [2022-05-27T15:24:58-07:00, sequoiatools/analyticsmanager:1.0:9d7c0b] -i 172.23.106.188 -b default,bucket1,bucket4,bucket5,bucket6,bucket7 -o drop_cbas_infra --api_timeout 3600
      [2022-05-27T17:28:48-07:00, sequoiatools/couchbase-cli:7.1:6c0351] server-add -c 172.23.108.103:8091 --server-add https://172.23.104.70 -u Administrator -p password --server-add-username Administrator --server-add-password password --services analytics

      CBM logs:

      2022-05-27T15:00:31.171-07:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.104.69:8091/pools/default/buckets/bucket9'
      2022-05-27T15:00:31.244-07:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.104.69:8091/pools/default/buckets/bucket9'
      2022-05-27T15:00:31.245-07:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.104.69:8091/pools/default/buckets/bucket9/scopes'
      2022-05-27T15:00:31.246-07:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.104.69:8091/pools/default/buckets/bucket9/scopes'
      2022-05-27T15:00:31.247-07:00 (REST) (Attempt 1) (POST) Dispatching request to 'http://172.23.104.69:8091/pools/default/buckets/bucket9/scopes/@ensureManifest/15b'
      2022-05-27T15:00:31.349-07:00 (REST) (Attempt 1) (POST) (200) Received response from 'http://172.23.104.69:8091/pools/default/buckets/bucket9/scopes/@ensureManifest/15b'
      2022-05-27T15:00:31.350-07:00 (Archive) (Repo) Creating new backup '2022-05-27T15_00_31.350446065-07_00'
      2022-05-27T15:00:31.360-07:00 (Stats) Starting stat gathering - stat timestamp: 1653688831
      2022-05-27T15:00:31.360-07:00 (Plan) (Cluster) Transferring cluster configuration
      2022-05-27T15:00:31.361-07:00 (Plan) (Cluster) Successfully transferred cluster configuration |

      {"number":1,"duration":"965.233┬Ás"}

      2022-05-27T15:00:31.364-07:00 (Plan) (Analytics) Transferring Analytics metadata
      2022-05-27T15:00:31.365-07:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.105.168:8095/api/v1/backup'
      2022-05-27T15:00:41.367-07:00 ERRO: (REST) (Attempt 1) (GET) Failed to perform request to 'http://172.23.105.168:8095/api/v1/backup': Get "http://172.23.105.168:8095/api/v1/backup": net/http: timeout awaiting response headers – logging.(*ToolsCommonLogger).Log() at tools_common.go:30
      2022-05-27T15:00:41.367-07:00 WARN: (REST) (Attempt 1) (GET) Request to endpoint '/api/v1/backup' failed due to error: failed to perform request: Get "http://172.23.105.168:8095/api/v1/backup": net/http: timeout awaiting response headers – logging.(*ToolsCommonLogger).Log() at tools_common.go:28
      2022-05-27T15:00:41.367-07:00 (Cmd) Error backing up cluster: failed to execute cluster operations: failed to transfer Analytics metadata: failed to get Analytics metadata: failed to get analytics metadata: failed to execute request: failed to execute request: failed to perform request: Get "http://172.23.105.168:8095/api/v1/backup": net/http: timeout awaiting response headers
      2022-05-27T21:00:32.339-07:00 (Cmd) cbbackupmgr version 7.1.1-3094 Hostname: cnt7-qe1 OS: linux Version: 3.10.0-693.5.2.el7.x86_64 Arch: amd64 vCPU: 8 Memory: 25107591168 (23.38GiB)
      2022-05-27T21:00:32.339-07:00 (Cmd) backup --cluster http://172.23.104.69:8091 --archive /data/archive --repo 2b93f7e0-f20f-4bbc-8be5-f77b0244f4ab --purge --auto-select-threads --json-progress 00e1f76b-6537-4c5b-8463-0f2dabe48377 --full-backup
      2022-05-27T21:00:32.339-07:00 (Cmd) mounted archive with id: b9258679-c56a-4524-8ca7-0a45f6b63aeb
      2022-05-27T21:00:32.339-07:00 (Cmd) Automatically set the number of threads to 6

      Server logs:
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.104.137.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.104.155.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.104.157.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.104.5.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.104.67.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.104.69.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.104.70.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.105.107.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.105.111.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.105.168.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.106.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.106.188.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.108.103.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.120.107.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.120.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.121.117.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.123.28.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.96.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.96.192.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.96.251.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.96.252.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.96.253.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.97.119.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.97.121.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.97.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.97.239.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.99.20.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.99.21.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1653715753/collectinfo-2022-05-28T052915-ns_1%40172.23.99.25.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Matt.Hall Matt Hall
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty