Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48485

[CBM] Failed to resume backup with error ' (bucket0) (vb 867) Received an unexpected error from the sink callback, beginning teardown'

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Duplicate
    • 7.0.2
    • None
    • tools
    • Ubuntu 20.04
    • Untriaged
    • Ubuntu 64-bit
    • 1
    • Yes

    Description

      Install Couchbase server 7.0.2-6643 on 4 ubuntu 20.04 servers
      Create bucket bucket0
      Load 1 billion docs with size from 512 to 1024 bytes to bucket0
      Stop all loads
      At node 172.31.19.224 of cluster, create backup repo and run backup.
      During backup, ssh session is out and cbbackupmgr killed.
      Login again and run resume backup, failed to run backup with error:

      2021-09-16T02:35:01.823+00:00 WARN: (DCP) (bucket0) (vb 867) Received an unexpected error from the sink callback, beginning teardown: failed to process mutation for sequence number 594291: failed verify and rollback data store: failed to rollback index to sequence number 594291: failed to rollback to sequence number 594291: database disk image is malformed -- couchbase.(*DCPAsyncWorker).handleDCPError() at dcp_async_worker.go:568
      2021-09-16T02:35:01.827+00:00 (Cmd) Error backing up cluster: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket0': failed to transfer bucket data for bucket 'bucket0': failed to transfer key value data: failed to transfer key value data: failed to process mutation for sequence number 594291: failed verify and rollback data store: failed to rollback index to sequence number 594291: failed to rollback to sequence number 594291: database disk image is malformed
      2021-09-16T02:46:52.748+00:00 (Cmd) cbbackupmgr version 7.0.2-6643 Hostname: ip-172-31-19-224 OS: linux Version: 5.4.0-1045-aws Arch: amd64 vCPU: 16 Memory: 515474309120 (480.07GiB)
      2021-09-16T02:46:52.748+00:00 (Cmd) collect-logs -a /data/bk/ -o /root/
      2021-09-16T02:46:52.748+00:00 (Cmd) mounted archive with id: fd738579-f97c-44aa-9224-8eb8db616461
      
      

      This test run to verify MB-47144

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Backup and restore without resume work as expected. All data are backup and restored with repo bk1.

            thuan Thuan Nguyen added a comment - Backup and restore without resume work as expected. All data are backup and restored with repo bk1.
            james.lee James Lee added a comment - - edited

            Hi Thuan Nguyen,

            Thanks for raising this issue, it's an interesting behaviour; I had a look at 'testrunner' and it looks like we're killing 'cbbackupmgr' using 'kill -9' (please correct me if I'm wrong).

            This is not a signal that can be caught, and may cause in-progress operations to fail (causing corruption).

            With that being said, I think we should handle these cases slightly better if possible (I've got a couple of MBs open regarding this which I've linked to this MB).

            Thanks,
            James

            james.lee James Lee added a comment - - edited Hi Thuan Nguyen , Thanks for raising this issue, it's an interesting behaviour; I had a look at ' testrunner ' and it looks like we're killing ' cbbackupmgr ' using ' kill -9 ' (please correct me if I'm wrong). This is not a signal that can be caught, and may cause in-progress operations to fail (causing corruption). With that being said, I think we should handle these cases slightly better if possible (I've got a couple of MBs open regarding this which I've linked to this MB). Thanks, James
            thuan Thuan Nguyen added a comment -

            This is manual test. No testrunner is used in this bug.

            thuan Thuan Nguyen added a comment - This is manual test. No testrunner is used in this bug.
            james.lee James Lee added a comment -

            Ah my mistake, thanks for highlighting that Thuan Nguyen, I think the same still applies in this case though.

            james.lee James Lee added a comment - Ah my mistake, thanks for highlighting that Thuan Nguyen , I think the same still applies in this case though.

            People

              thuan Thuan Nguyen
              thuan Thuan Nguyen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty