Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49527

CBBM restore fails with "file does not exist" when restoring ranges

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Yes

    Description

      What's happening?

      When trying to perform a restore of a range of backups, CBBM is failing with "file does not exist".

      When looking at the logs, it seems that the file for a vbucket is missing, causing this failure.

      Relevant log snippet:

      '2021-11-12T06:44:50.644-08:00 (Archive) (default) (vb 1023) Transferring data | {"start":0,"end":7}', 
      '2021-11-12T06:44:50.646-08:00 (Archive) (default) (vb 1023) Completed data transfer | {"start":0,"end":7}', 
      '2021-11-12T06:44:50.646-08:00 (Archive) (default) (vb 510) Transferring data | {"start":0,"end":7}', 
      '2021-11-12T06:44:50.647-08:00 (Archive) (default) (vb 510) Completed data transfer | {"start":0,"end":7}', 
      '2021-11-12T06:44:50.647-08:00 (Archive) (default) (vb 1022) Transferring data | {"start":0,"end":5}', 
      "2021-11-12T06:44:50.647-08:00 (Cmd) Error restoring cluster: failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: failed to open source vBucket 1022: failed to open vBucket 1022: failed to open index: failed to open read only index: file does not exist"
      

      What is expected?

      I'd expect the restore to go through successfully, or for a more useful error message to be shown.

      Steps to reproduce

      1. Take many backups of a cluster
      2. Perform restores of random ranges
      3. Eventually a failure will occur

      It's worth noting that this is consistently reproducible on our jenkins machines, but not on my local mac machine.

      This suggests it could be linked to machine performance.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              joe.mitchelljones Joe Mitchell Jones
              joe.mitchelljones Joe Mitchell Jones
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty