Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49554

[System Test][Backup Service] backup task failed with error - timeout after scopes/collections (for manifest id 5bc) to be ready for use waiting for 1m0s, please try again later

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • None
    • 7.1.0
    • tools
    • Untriaged
    • 1
    • No

    Description

      7.1.0-1701

      Test:
      -test tests/integration/neo/test_neo_couchstore_milestone3.yml -scope tests/integration/neo/scope_couchstore.yml
      Scale 3
      Iteration 2

      Backup task failed with error:

      {
        "task_name": "backup-1",
        "status": "failed",
        "start": "2021-11-14T18:00:46.227721654-08:00",
        "end": "2021-11-14T18:01:48.763172845-08:00",
        "node_runs": [
          {
            "node_id": "d46cdb27f27c27a7560c1f113f512dc5",
            "status": "failed",
            "start": "2021-11-14T18:00:46.332055828-08:00",
            "end": "2021-11-14T18:01:48.700037387-08:00",
            "error": "exit status 1: failed to get backup transferable: failed to create backup: failed to get source collection manifest: failed to wait for manifest id 1468: timeout after scopes/collections (for manifest id 5bc) to be ready for use waiting for 1m0s, please try again later",
            "progress": 0,
            "stats": {
              "error": "failed to get backup transferable: failed to create backup: failed to get source collection manifest: failed to wait for manifest id 1468: timeout after scopes/collections (for manifest id 5bc) to be ready for use waiting for 1m0s, please try again later"
            },
            "error_code": 2
          }
        ],
        "error": "exit status 1: failed to get backup transferable: failed to create backup: failed to get source collection manifest: failed to wait for manifest id 1468: timeout after scopes/collections (for manifest id 5bc) to be ready for use waiting for 1m0s, please try again later",
        "error_code": 2,
        "type": "BACKUP",
        "show": true
      }
      

      During this time, a kv rebalance was stuck in the test:

      [2021-11-13T13:26:35-08:00, sequoiatools/couchbase-cli:7.1:c2fa80] server-add -c 172.23.108.103:8091 --server-add https://172.23.104.157 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      [2021-11-13T13:26:49-08:00, sequoiatools/couchbase-cli:7.1:480316] rebalance -c 172.23.108.103:8091 -u Administrator -p password

      We have filed MB-49542 for the rebalance stuck issue but filing this to make sure the error is due to this

      Cluster config:

      ########## Cluster config ##################
      ######  backup : 1 ===== > [172.23.123.28:8091]  ###########
      ######  n1ql : 2 ===== > [172.23.96.251:8091 172.23.99.11:8091]  ###########
      ######  eventing : 3 ===== > [172.23.104.137:8091 172.23.123.27:8091 172.23.98.135:8091]  ###########
      ######  fts : 2 ===== > [172.23.104.155:8091 172.23.104.157:8091]  ###########
      ######  kv : 11 ===== > [172.23.104.5:8091 172.23.106.100:8091 172.23.108.103:8091 172.23.96.148:8091 172.23.97.119:8091 172.23.97.121:8091 172.23.97.122:8091 172.23.97.239:8091 172.23.99.20:8091 172.23.99.21:8091 172.23.99.25:8091]  ###########
      ######  index : 6 ===== > [172.23.104.67:8091 172.23.104.69:8091 172.23.120.245:8091 172.23.121.117:8091 172.23.96.252:8091 172.23.96.253:8091]  ###########
      ######  cbas : 3 ===== > [172.23.106.188:8091 172.23.121.3:8091 172.23.97.242:8091]  ###########
      

      Attaching cbm collect logs - server logs here:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.104.137.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.104.155.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.104.157.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.104.5.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.104.67.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.104.69.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.104.70.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.106.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.106.188.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.108.103.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.120.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.121.117.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.121.3.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.123.27.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.123.28.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.96.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.96.251.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.96.252.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.96.253.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.97.119.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.97.121.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.97.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.97.239.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.97.242.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.98.135.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.99.11.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.99.20.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.99.21.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636945325/collectinfo-2021-11-15T030207-ns_1%40172.23.99.25.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty