Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-55561

[CBM] Data range backwards compatability issue with empty backups (x.x.x -> 7.1.x)

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Yes

    Description

      What's the issue?
      When merging an empty backup created before 7.1.0 cbbackupmgr will create an empty data range.

      {"version":1,"source":{},"sink":{"initial":{},"current":{}}}
      

      If merged later on, a panic will occur trying to access the source.failover_log attribute which is empty.

      panic: runtime error: index out of range [-1]
       
      goroutine 1 [running]:
      github.com/couchbase/backup/archive.(*repoDir).shouldRollback(0x0?, 0x0?, 0x0?)
      	/home/james/Projects/couchbase-build/backup/archive/dir_repo.go:761 +0xf0
      github.com/couchbase/backup/archive.(*repoDir).updateSourceRanges(_, {0xc00097a6f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
      	/home/james/Projects/couchbase-build/backup/archive/dir_repo.go:751 +0x7a
      github.com/couchbase/backup/archive.(*repoDir).getBackupSeqNoRange(_, {_, _}, {_, _}, {_, _}, _)
      	/home/james/Projects/couchbase-build/backup/archive/dir_repo.go:709 +0x4f3
      github.com/couchbase/backup/archive.(*Source).GetDataRanges(_, {_, _}, {_, _})
      	/home/james/Projects/couchbase-build/backup/archive/source.go:166 +0x8c
      github.com/couchbase/backup/plan/services/data.(*dataRange).Execute(0xc0000bc5a0, {0x7f44540a73d8, 0xc00054c000}, {0x7f44540a7420, 0xc000152000}, 0xc0009be3b0)
      	/home/james/Projects/couchbase-build/backup/plan/services/data/range.go:45 +0x169
      github.com/couchbase/backup/plan/services/data.(*Data).Execute(0xc0000138a8, {0x7f44540a73d8, 0xc00054c000}, {0x7f44540a7420, 0xc000152000}, 0xc0009be3b0?)
      	/home/james/Projects/couchbase-build/backup/plan/services/data/data.go:53 +0xca
      github.com/couchbase/backup/plan/cluster/bucket.(*Bucket).Execute(0xc00054a3f0, {0x7f44540a5968?, 0xc00054c000}, {0x7f44540a5a08?, 0xc000152000}, 0x144aca0?)
      	/home/james/Projects/couchbase-build/backup/plan/cluster/bucket/bucket.go:205 +0xb5a
      github.com/couchbase/backup/plan/cluster.(*Cluster).Execute.func1(0x0?)
      	/home/james/Projects/couchbase-build/backup/plan/cluster/cluster.go:91 +0x125
      github.com/couchbase/backup/plan/cluster.(*Cluster).Execute(0xc0000bc550, {0x7f44540a5680?, 0xc00054c000}, {0x7f44540a5730?, 0xc000152000}, 0xc0009be3b0)
      	/home/james/Projects/couchbase-build/backup/plan/cluster/cluster.go:130 +0x431
      github.com/couchbase/backup/plan.(*Plan).execute(0xc00041b760, {0x1ee7ac8?, 0xc00054c000}, {0x1eeaf08?, 0xc000152000}, {0x1eddb40, 0xc00050f050})
      	/home/james/Projects/couchbase-build/backup/plan/plan.go:126 +0x13a
      github.com/couchbase/backup/plan.(*Plan).Execute(0x1a96081?, {0x1ee7ac8?, 0xc00054c000?}, {0x1eeaf08?, 0xc000152000?}, {0x1eddb40?, 0xc00050f050?})
      	/home/james/Projects/couchbase-build/backup/plan/plan.go:60 +0x51
      github.com/couchbase/backup/archive.(*Archive).MergeIncrementalBackups(0xc00041a500, {{0x7fff045b0f75, 0x4}, {0x7fff045b0f82, 0x5}, {0x7fff045b0f8e, 0x3}, 0x3, {0x1eddb40, 0xc00050f050}, ...})
      	/home/james/Projects/couchbase-build/backup/archive/archive.go:1068 +0xde5
      main.(*MergeContext).Run(0xc000116400)
      	/home/james/Projects/couchbase-build/backup/cmd/cbbackupmgr/merge.go:147 +0xac5
      github.com/couchbase/cbflag.(*Command).parseFlags(0xc000945f10, 0xc0009bf218, {0xc000000200, 0x8, 0xc000014638?})
      	/home/james/.local/lib/golang/pkg/mod/github.com/couchbase/cbflag@v0.0.0-20220325172535-f0f642f857ed/command.go:283 +0x111e
      github.com/couchbase/cbflag.(*Command).parse(0xc000945f10, 0xc0009bf218, {0xc000000200?, 0x8, 0x8})
      	/home/james/.local/lib/golang/pkg/mod/github.com/couchbase/cbflag@v0.0.0-20220325172535-f0f642f857ed/command.go:111 +0x12f
      github.com/couchbase/cbflag.(*Command).parseCommands(0xc00079a0e0, 0xc0009bf218, {0xc0000001f0, 0x9, 0x9})
      	/home/james/.local/lib/golang/pkg/mod/github.com/couchbase/cbflag@v0.0.0-20220325172535-f0f642f857ed/command.go:124 +0x1aa
      github.com/couchbase/cbflag.(*Command).parse(0xc00079a0e0, 0xc0009bf218, {0xc0000001f0?, 0x9, 0x9})
      	/home/james/.local/lib/golang/pkg/mod/github.com/couchbase/cbflag@v0.0.0-20220325172535-f0f642f857ed/command.go:109 +0x146
      github.com/couchbase/cbflag.(*CLI).Parse(0xc0003dab00, {0xc0000001e0, 0xa, 0xa})
      	/home/james/.local/lib/golang/pkg/mod/github.com/couchbase/cbflag@v0.0.0-20220325172535-f0f642f857ed/cli.go:70 +0x199
      main.main()
      	/home/james/Projects/couchbase-build/backup/cmd/cbbackupmgr/main.go:4081 +0xdc1c
      

      The reason for cbbackupmgr creating an empty data range is due to backwards compatibility handling (hence the issue initially being seen when moving from 7.0.3 to 7.1.3).

      In theory, this issue will affect reading any backup created prior to 7.1.0 using a 7.1.x version of cbbackupmgr.

      What's the fix?
      We should disregard the failover log stored in range.json and use the one stored in the data directory, which should always be persisted, whether or not any data is transferred.

      Is there a workaround?
      No, however, this failure case is non-destructive; cbbackupmgr will fail part way through the merge when trying to read the source data range.

      1. Don't include the affected backups in the merge
      2. Create a new archive/repository with 7.1.x, future backups/merges should work as expected

      Attachments

        1. archive.tar.gz
          21.20 MB
          Gilad Kalchheim

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              gilad.kalchheim Gilad Kalchheim
              james.lee James Lee
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty