Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49527

CBBM restore fails with "file does not exist" when restoring ranges

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Yes

    Description

      What's happening?

      When trying to perform a restore of a range of backups, CBBM is failing with "file does not exist".

      When looking at the logs, it seems that the file for a vbucket is missing, causing this failure.

      Relevant log snippet:

      '2021-11-12T06:44:50.644-08:00 (Archive) (default) (vb 1023) Transferring data | {"start":0,"end":7}', 
      '2021-11-12T06:44:50.646-08:00 (Archive) (default) (vb 1023) Completed data transfer | {"start":0,"end":7}', 
      '2021-11-12T06:44:50.646-08:00 (Archive) (default) (vb 510) Transferring data | {"start":0,"end":7}', 
      '2021-11-12T06:44:50.647-08:00 (Archive) (default) (vb 510) Completed data transfer | {"start":0,"end":7}', 
      '2021-11-12T06:44:50.647-08:00 (Archive) (default) (vb 1022) Transferring data | {"start":0,"end":5}', 
      "2021-11-12T06:44:50.647-08:00 (Cmd) Error restoring cluster: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to open source vBucket 1022: failed to open vBucket 1022: failed to open index: failed to open read only index: file does not exist"
      

      What is expected?

      I'd expect the restore to go through successfully, or for a more useful error message to be shown.

      Steps to reproduce

      1. Take many backups of a cluster
      2. Perform restores of random ranges
      3. Eventually a failure will occur

      It's worth noting that this is consistently reproducible on our jenkins machines, but not on my local mac machine.

      This suggests it could be linked to machine performance.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            arunkumar Arunkumar Senthilnathan added a comment - - edited

            This is observed in longevity against 1701 as well:

            {
              "task_name": "merge",
              "status": "failed",
              "start": "2021-11-13T12:00:03.391420056-08:00",
              "end": "2021-11-13T12:04:47.514909333-08:00",
              "node_runs": [
                {
                  "node_id": "f2a5753e6728fe13ff76ca78912f644c",
                  "status": "failed",
                  "start": "2021-11-13T12:00:03.468026758-08:00",
                  "end": "2021-11-13T12:04:47.476018885-08:00",
                  "error": "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket9': failed to transfer bucket data for bucket 'bucket9': failed to transfer key value data: failed to transfer key value data: failed to open source vBucket 1022: failed to open vBucket 1022: failed to open index: failed to open read only index: file does not exist",
                  "progress": 0,
                  "stats": {
                    "error": "failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket9': failed to transfer bucket data for bucket 'bucket9': failed to transfer key value data: failed to transfer key value data: failed to open source vBucket 1022: failed to open vBucket 1022: failed to open index: failed to open read only index: file does not exist"
                  },
                  "error_code": 2
                }
              ],
              "error": "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket9': failed to transfer bucket data for bucket 'bucket9': failed to transfer key value data: failed to transfer key value data: failed to open source vBucket 1022: failed to open vBucket 1022: failed to open index: failed to open read only index: file does not exist",
              "error_code": 2,
              "type": "MERGE",
              "show": true
            }
            

            Attaching cbm collect-logs

            arunkumar Arunkumar Senthilnathan added a comment - - edited This is observed in longevity against 1701 as well: { "task_name": "merge", "status": "failed", "start": "2021-11-13T12:00:03.391420056-08:00", "end": "2021-11-13T12:04:47.514909333-08:00", "node_runs": [ { "node_id": "f2a5753e6728fe13ff76ca78912f644c", "status": "failed", "start": "2021-11-13T12:00:03.468026758-08:00", "end": "2021-11-13T12:04:47.476018885-08:00", "error": "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket9': failed to transfer bucket data for bucket 'bucket9': failed to transfer key value data: failed to transfer key value data: failed to open source vBucket 1022: failed to open vBucket 1022: failed to open index: failed to open read only index: file does not exist", "progress": 0, "stats": { "error": "failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket9': failed to transfer bucket data for bucket 'bucket9': failed to transfer key value data: failed to transfer key value data: failed to open source vBucket 1022: failed to open vBucket 1022: failed to open index: failed to open read only index: file does not exist" }, "error_code": 2 } ], "error": "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket9': failed to transfer bucket data for bucket 'bucket9': failed to transfer key value data: failed to transfer key value data: failed to open source vBucket 1022: failed to open vBucket 1022: failed to open index: failed to open read only index: file does not exist", "error_code": 2, "type": "MERGE", "show": true } Attaching cbm collect-logs
            arunkumar Arunkumar Senthilnathan added a comment - - edited

            Server logs:

            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.106.134.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.106.136.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.120.58.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.120.73.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.120.74.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.120.75.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.120.77.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.120.81.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.120.86.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.121.77.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.123.24.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.123.25.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.123.26.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.123.31.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.123.32.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.123.33.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.96.122.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.96.14.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.96.243.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.96.254.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.96.48.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.97.105.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.97.110.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.97.112.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.97.148.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.97.149.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.97.150.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.97.151.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.97.241.zip
            url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.97.74.zip

            arunkumar Arunkumar Senthilnathan added a comment - - edited Server logs: url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.106.134.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.106.136.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.120.58.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.120.73.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.120.74.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.120.75.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.120.77.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.120.81.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.120.86.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.121.77.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.123.24.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.123.25.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.123.26.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.123.31.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.123.32.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.123.33.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.96.122.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.96.14.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.96.243.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.96.254.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.96.48.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.97.105.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.97.110.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.97.112.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.97.148.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.97.149.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.97.150.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.97.151.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.97.241.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636903495/collectinfo-2021-11-14T152457-ns_1%40172.23.97.74.zip
            james.lee James Lee added a comment -

            Marking as a duplicate, solely because the code is already up against MB-49530.

            james.lee James Lee added a comment - Marking as a duplicate, solely because the code is already up against MB-49530 .

            People

              joe.mitchelljones Joe Mitchell Jones
              joe.mitchelljones Joe Mitchell Jones
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty