Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-24952

cbbackupmgr merge fails with error Backup Meta `/disk3/backup/backup/.merge/other-3ebf39ba0b75e24c10024712fd58c61e/range.json` not found

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      Build : 5.0.0-3104

      This is a system test where we generate a lot of incremental backups with constant KV ops, and merge these backups. When merging a merged backup with several incremental backups, merge fails with the error :
      Backup Meta `/disk3/backup/backup/.merge/other-3ebf39ba0b75e24c10024712fd58c61e/range.json` not found

      No. of nodes in src cluster : 3 (64GB RAM x 32 cores)
      No. of Items in cluster : ~166M (some items expired)
      No. of buckets : 2

      Steps :
      1. Start data loading on 2 buckets (65% creates, 25% updates, 10% deletes, 15% items have TTL set to 10 mins). This will run throughout the test
      2. After 5 mins, take 1st backup
      3. Rebalance out a node
      4. After 5 mins, take 2nd backup
      5. Rebalance in a node
      6. After 5 mins, take 3rd backup
      7. Failover and add back a node
      8. After 5 mins, take 4th backup
      9. Failover and rebalance out a node
      10. After 5 mins, take 5th backup
      11. Flush bucket
      12. After 5 mins, take 6th backup
      13. Merge all backups
      14. Repeat Steps 2-12
      15. At this point, there will be 7 backups (1 merged, 6 incremental)
      16. Merge all backups

      Merge fails with the above mentioned error. backup.log attached.

      Attachments

        1. backup_5.0.0-3161.log
          671 kB
        2. backup.log
          11.84 MB

        Issue Links

          For Gerrit Dashboard: MB-24952
          # Subject Branch Project Status CR V

          Activity

            Not seeing this issue on 4.6.3-4047. Ran merge around 25+ times.

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - Not seeing this issue on 4.6.3-4047. Ran merge around 25+ times.

            Build 5.0.0-3272 contains backup commit 5aeb9f819a39e5683fbcb59ec147fed1dc27c013 with commit message:
            MB-24952: [Backport] Fix deadlock when an unexpected DCP error is received
            https://github.com/couchbase/backup/commit/5aeb9f819a39e5683fbcb59ec147fed1dc27c013

            build-team Couchbase Build Team added a comment - Build 5.0.0-3272 contains backup commit 5aeb9f819a39e5683fbcb59ec147fed1dc27c013 with commit message: MB-24952 : [Backport] Fix deadlock when an unexpected DCP error is received https://github.com/couchbase/backup/commit/5aeb9f819a39e5683fbcb59ec147fed1dc27c013

            Build 5.0.0-3272 contains backup commit c2efca5f9c1b9fddc4d60d595ca4bb6ca33b14c7 with commit message:
            MB-24952: [Backport] Use the snap end seqno as the end seqno if it's larger
            https://github.com/couchbase/backup/commit/c2efca5f9c1b9fddc4d60d595ca4bb6ca33b14c7

            build-team Couchbase Build Team added a comment - Build 5.0.0-3272 contains backup commit c2efca5f9c1b9fddc4d60d595ca4bb6ca33b14c7 with commit message: MB-24952 : [Backport] Use the snap end seqno as the end seqno if it's larger https://github.com/couchbase/backup/commit/c2efca5f9c1b9fddc4d60d595ca4bb6ca33b14c7

            Build 5.1.0-1013 contains backup commit 5aeb9f819a39e5683fbcb59ec147fed1dc27c013 with commit message:
            MB-24952: [Backport] Fix deadlock when an unexpected DCP error is received
            https://github.com/couchbase/backup/commit/5aeb9f819a39e5683fbcb59ec147fed1dc27c013

            build-team Couchbase Build Team added a comment - Build 5.1.0-1013 contains backup commit 5aeb9f819a39e5683fbcb59ec147fed1dc27c013 with commit message: MB-24952 : [Backport] Fix deadlock when an unexpected DCP error is received https://github.com/couchbase/backup/commit/5aeb9f819a39e5683fbcb59ec147fed1dc27c013

            Build 5.1.0-1013 contains backup commit c2efca5f9c1b9fddc4d60d595ca4bb6ca33b14c7 with commit message:
            MB-24952: [Backport] Use the snap end seqno as the end seqno if it's larger
            https://github.com/couchbase/backup/commit/c2efca5f9c1b9fddc4d60d595ca4bb6ca33b14c7

            build-team Couchbase Build Team added a comment - Build 5.1.0-1013 contains backup commit c2efca5f9c1b9fddc4d60d595ca4bb6ca33b14c7 with commit message: MB-24952 : [Backport] Use the snap end seqno as the end seqno if it's larger https://github.com/couchbase/backup/commit/c2efca5f9c1b9fddc4d60d595ca4bb6ca33b14c7

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty