Uploaded image for project: 'Couchbase Go SDK'
  1. Couchbase Go SDK
  2. GOCBC-1072

gocbcore attempts to execute 'SelectBucket' on a KV node which contains no buckets (due to a stopped/failed rebalance)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • None
    • core-9.1.2
    • core-library
    • 1

    Description

      7.0.0-4626

      Test:
      -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
      2nd iteration
      Scale 2

      Backup task failed with error:

      {
        "task_name": "backup-1",
        "status": "failed",
        "start": "2021-03-10T04:00:22.70680266-08:00",
        "end": "2021-03-10T04:00:49.991647042-08:00",
        "node_runs": [
          {
            "node_id": "0e6f9f3a49f42fe5f42d95a4e8e8bc6b",
            "status": "failed",
            "start": "2021-03-10T04:00:22.836914313-08:00",
            "end": "2021-03-10T04:00:49.852280959-08:00",
            "error": "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket6': failed to transfer bucket data for bucket 'bucket6': failed to transfer key value data: failed to transfer key value data: failed to initilise worker 2: failed to get gocbcore DCP agent: agent failed to connect to the cluster: bucket not found",
            "progress": 0,
            "stats": {
              "error": "failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket6': failed to transfer bucket data for bucket 'bucket6': failed to transfer key value data: failed to transfer key value data: failed to initilise worker 2: failed to get gocbcore DCP agent: agent failed to connect to the cluster: bucket not found",
              "stats": {
                "started_at": 1615377623033792300,
                "buckets": {
                  "bucket6": {
                    "estimated_total_items": 9057547,
                    "total_vbuckets": 1024,
                    "started_at": 1615377638386640600,
                    "complete": false,
                    "errored": false
                  }
                },
                "complete": false
              }
            },
            "error_code": 2
          }
        ],
        "error": "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket6': failed to transfer bucket data for bucket 'bucket6': failed to transfer key value data: failed to transfer key value data: failed to initilise worker 2: failed to get gocbcore DCP agent: agent failed to connect to the cluster: bucket not found",
        "error_code": 2,
        "type": "BACKUP",
        "show": true
      }
      

      From backup.log:

      2021-03-10T04:00:38.282-08:00 (Plan) (Data) Transferring new key value data for bucket 'bucket6'
      2021-03-10T04:00:38.282-08:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.104.137:8091/pools/default/buckets/bucket6'
      2021-03-10T04:00:38.340-08:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.104.137:8091/pools/default/buckets/bucket6'
      2021-03-10T04:00:38.341-08:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.104.137:8091/pools/default/buckets/bucket6/scopes'
      2021-03-10T04:00:38.342-08:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.104.137:8091/pools/default/buckets/bucket6/scopes'
      2021-03-10T04:00:38.342-08:00 (REST) (Attempt 1) (POST) Dispatching request to 'http://172.23.104.137:8091/pools/default/buckets/bucket6/scopes/%40ensureManifest/22'
      2021-03-10T04:00:38.349-08:00 (REST) (Attempt 1) (POST) (200) Received response from 'http://172.23.104.137:8091/pools/default/buckets/bucket6/scopes/%40ensureManifest/22'
      2021-03-10T04:00:49.835-08:00 (Cmd) Error backing up cluster: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket6': failed to transfer bucket data for bucket 'bucket6': failed to transfer key value data: failed to transfer key value data: failed to initilise worker 2: failed to get gocbcore DCP agent: agent failed to connect to the cluster: bucket not found
      

      During this time, the test was building indexes:

      2021-03-10T03:47:10-08:00, sequoiatools/cbq:e08555] -e=http://172.23.96.251:8093 -u=Administrator -p=password -script=create primary index on `default` using GSI
      [2021-03-10T03:47:16-08:00, sequoiatools/indexmanager:6ec292] -n 172.23.108.103 -o 8091 -u Administrator -p password -b bucket4 -i 4 -a create_index
      [2021-03-10T04:39:53-08:00, sequoiatools/indexmanager:83e1ac] -n 172.23.108.103 -o 8091 -u Administrator -p password -b bucket5 -i 4 -a create_index
      [2021-03-10T05:17:16-08:00, sequoiatools/indexmanager:228e26] -n 172.23.108.103 -o 8091 -u Administrator -p password -b bucket6 -i 4 -a create_index
      [2021-03-10T06:07:44-08:00, sequoiatools/indexmanager:b04b4c] -n 172.23.108.103 -o 8091 -u Administrator -p password -b bucket7 -i 4 -a create_index
      

      Server logs collected around this time:
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.104.137.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.104.155.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.104.157.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.104.5.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.104.67.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.104.69.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.104.70.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.105.107.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.105.111.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.106.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.106.188.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.108.103.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.120.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.121.117.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.121.3.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.123.27.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.123.28.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.96.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.96.251.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.96.252.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.96.253.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.97.119.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.97.121.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.97.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.97.239.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.97.242.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.98.135.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.99.11.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.99.20.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.99.21.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615382242/collectinfo-2021-03-10T131724-ns_1%40172.23.99.25.zip

      Attaching backup log

      Attachments

        1. backup-0.log
          16.72 MB
        2. backup-0-1.log
          57.28 MB
        3. backup-0-apr17.log
          24.07 MB
        4. reproduction.pcap
          6.38 MB
        5. reproductions.tar.gz
          1.11 MB
        6. Screen Shot 2021-04-22 at 3.40.39 PM.png
          Screen Shot 2021-04-22 at 3.40.39 PM.png
          47 kB

        Issue Links

          For Gerrit Dashboard: GOCBC-1072
          # Subject Branch Project Status CR V

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty