Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40235

[CBM] cbbackupmgr in upgrade version (6.6.0) throws segmentation violation error when merging old backup repo (5.5.0)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 6.6.0
    • Fix Version/s: 6.6.0
    • Component/s: tools
    • Environment:
      Centos 7.6
    • Triage:
      Untriaged
    • Operating System:
      Centos 64-bit
    • Story Points:
      1
    • Is this a Regression?:
      Unknown

      Description

      Install Couchbase server  5.5.0 on 4 centos 7.6 servers.

      Create 2 nodes cluster (224 and 225) and default bucket.

      Load data to default bucket.

      Run several backups with cbbackupmgr (5.5.0) to backup data

      Fresh install Couchbase server 6.6.0 on 2 centos servers (226 and 227)

      Add 2 6.6.0-7846 nodes to cluster above and remove 2 nodes 224 and 225

      Cluster becomes 6.6.0.

      Run some backups with cbbackupmgr (6.6.0-7846)

      Run merge with cbbackupmgr (6.6.0-7846), merge failed with segfault error

      2020-06-30T19:57:36.189-07:00 (Cmd) merge --archive /tmp/entbackup_172.23.121.224 --repo backup --start 2020-06-30T19_44_37.802124479-07_00 --end 2020-06-30T19_56_43.262610544-07_00 
      2020-06-30T19:57:36.189-07:00 (Cmd) mounted archive with id: unknown
      2020-06-30T19:57:36.191-07:00 (Stats) Starting stat gathering - stat timestamp: 1593572256
      2020-06-30T19:57:36.191-07:00 Merging backups [2020-06-30T19_44_37.802124479-07_00 2020-06-30T19_45_12.203175007-07_00 2020-06-30T19_45_48.336465942-07_00 2020-06-30T19_53_57.816275267-07_00 2020-06-30T19_54_54.873738339-07_00 2020-06-30T19_55_51.398617954-07_00 2020-06-30T19_56_43.262610544-07_00]
      2020-06-30T19:57:36.192-07:00 Creating the merge directory for backup backup
      panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0xab14ae]
       
       
      goroutine 1 [running]:
      github.com/couchbase/backup/archive.(*Archive).MergeIncrementalBackups(0xc0002807c0, 0x7ffcbe78edc3, 0x6, 0x7ffcbe78edd2, 0x23, 0x7ffcbe78edfc, 0x23, 0x3, 0x0, 0x0, ...)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/backup/archive/archive.go:747 +0x1d4e
      main.(*MergeContext).Run(0xc0002b8000)
      	goproj/src/github.com/couchbase/backup/cmd/cbbackupmgr/merge.go:111 +0x520
      github.com/couchbase/cbflag.(*Command).parseFlags(0xc0002c2a80, 0xc000207490, 0xc0000c4020, 0x8, 0x8)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbflag/command.go:241 +0x17bf
      github.com/couchbase/cbflag.(*Command).parse(0xc0002c2a80, 0xc000207490, 0xc0000c4020, 0x8, 0x8)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbflag/command.go:92 +0xd2
      github.com/couchbase/cbflag.(*Command).parseCommands(0xc0002c2c40, 0xc000207490, 0xc0000c4010, 0x9, 0x9)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbflag/command.go:104 +0x12c
      github.com/couchbase/cbflag.(*Command).parse(0xc0002c2c40, 0xc000207490, 0xc0000c4010, 0x9, 0x9)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbflag/command.go:90 +0x101
      github.com/couchbase/cbflag.(*CLI).Parse(0xc0002b8090, 0xc0000c4000, 0xa, 0xa)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbflag/cli.go:59 +0x18a
      main 

      I will upload logs soon

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            thuan Thuan Nguyen created issue -
            thuan Thuan Nguyen made changes -
            Field Original Value New Value
            Summary [CBM] cbbackupmgr in upgrade version (6.6.0) throws segmentation violation error when merging old backup repo (5.0.0) [CBM] cbbackupmgr in upgrade version (6.6.0) throws segmentation violation error when merging old backup repo (5.5.0)
            thuan Thuan Nguyen made changes -
            thuan Thuan Nguyen made changes -
            thuan Thuan Nguyen made changes -
            thuan Thuan Nguyen made changes -
            thuan Thuan Nguyen made changes -
            pvarley Patrick Varley made changes -
            Assignee Patrick Varley [ pvarley ] James Lee [ james.lee ]
            james.lee James Lee made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            james.lee James Lee made changes -
            Link This issue relates to MB-40246 [ MB-40246 ]
            till Till Westmann made changes -
            Link This issue blocks MB-38724 [ MB-38724 ]
            till Till Westmann made changes -
            Labels approved-for-6.6.0
            james.lee James Lee made changes -
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            james.lee James Lee made changes -
            Assignee James Lee [ james.lee ] Thuan Nguyen [ thuan ]
            Hide
            thuan Thuan Nguyen added a comment -

            I will verify it when the build with this fixes is ready

            Show
            thuan Thuan Nguyen added a comment - I will verify it when the build with this fixes is ready
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.0-7859 contains backup commit 9a9c4cc with commit message:
            MB-40235 Avoid segfault when merging 5.5.x backups

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.0-7859 contains backup commit 9a9c4cc with commit message: MB-40235 Avoid segfault when merging 5.5.x backups
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-2533 contains backup commit 9a9c4cc with commit message:
            MB-40235 Avoid segfault when merging 5.5.x backups

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-2533 contains backup commit 9a9c4cc with commit message: MB-40235 Avoid segfault when merging 5.5.x backups
            Hide
            arunkumar Arunkumar Senthilnathan added a comment -

            Verified with steps in description in 6.6.0-7861:

            [root@node1-mad-hatter-testing-centos7 bin]# ./cbbackupmgr merge -a /tmp/entbackup/ -r backup --start 2020-07-09T11_13_30.250764423-07_00 --end 2020-07-09T11_27_58.415507775-07_00
            (1/6) Merging backup '2020-07-09T11_13_30.250764423-07_00'
            Copied all data in 4.922372899s (Avg. 195.95KB/Sec) 10000 items / 783.81KB
            [======================================================================================================================================================================================================================================================================] 100.00%
            Merge bucket 'default' succeeded
            Mutations merged: 10000, Mutations failed to merged: 0
            Deletions merged: 0, Deletions failed to merged: 0

            (2/6) Merging backup '2020-07-09T11_13_52.941198062-07_00'
            Copied all data in 4.908067918s (Avg. 286.28KB/Sec) 15000 items / 1.12MB
            [======================================================================================================================================================================================================================================================================] 100.00%
            Merge bucket 'default' succeeded
            Mutations merged: 15000, Mutations failed to merged: 0
            Deletions merged: 0, Deletions failed to merged: 0

            (3/6) Merging backup '2020-07-09T11_14_09.662222502-07_00'
            Copied all data in 5.059106021s (Avg. 301.29KB/Sec) 20000 items / 1.47MB
            [======================================================================================================================================================================================================================================================================] 100.00%
            Merge bucket 'default' succeeded
            Mutations merged: 20000, Mutations failed to merged: 0
            Deletions merged: 0, Deletions failed to merged: 0

            (4/6) Merging backup '2020-07-09T11_22_45.949655018-07_00'
            Copied all data in 8.025484135s (Avg. 218.21KB/Sec) 25000 items / 1.70MB
            [======================================================================================================================================================================================================================================================================] 100.00%
            Merge bucket 'default' succeeded
            Mutations merged: 25000, Mutations failed to merged: 0
            Deletions merged: 0, Deletions failed to merged: 0

            (5/6) Merging backup '2020-07-09T11_27_10.003238822-07_00'
            Copied all data in 8.545139194s (Avg. 260.33KB/Sec) 30000 items / 2.03MB
            [======================================================================================================================================================================================================================================================================] 100.00%
            Merge bucket 'default' succeeded
            Mutations merged: 30000, Mutations failed to merged: 0
            Deletions merged: 0, Deletions failed to merged: 0

            (6/6) Merging backup '2020-07-09T11_27_58.415507775-07_00'
            Copied all data in 8.492812354s (Avg. 302.44KB/Sec) 35000 items / 2.36MB
            [======================================================================================================================================================================================================================================================================] 100.00%
            Merge bucket 'default' succeeded
            Mutations merged: 35000, Mutations failed to merged: 0
            Deletions merged: 0, Deletions failed to merged: 0
            Merge completed successfully

            Show
            arunkumar Arunkumar Senthilnathan added a comment - Verified with steps in description in 6.6.0-7861: [root@node1-mad-hatter-testing-centos7 bin] # ./cbbackupmgr merge -a /tmp/entbackup/ -r backup --start 2020-07-09T11_13_30.250764423-07_00 --end 2020-07-09T11_27_58.415507775-07_00 (1/6) Merging backup '2020-07-09T11_13_30.250764423-07_00' Copied all data in 4.922372899s (Avg. 195.95KB/Sec) 10000 items / 783.81KB [======================================================================================================================================================================================================================================================================] 100.00% Merge bucket 'default' succeeded Mutations merged: 10000, Mutations failed to merged: 0 Deletions merged: 0, Deletions failed to merged: 0 (2/6) Merging backup '2020-07-09T11_13_52.941198062-07_00' Copied all data in 4.908067918s (Avg. 286.28KB/Sec) 15000 items / 1.12MB [======================================================================================================================================================================================================================================================================] 100.00% Merge bucket 'default' succeeded Mutations merged: 15000, Mutations failed to merged: 0 Deletions merged: 0, Deletions failed to merged: 0 (3/6) Merging backup '2020-07-09T11_14_09.662222502-07_00' Copied all data in 5.059106021s (Avg. 301.29KB/Sec) 20000 items / 1.47MB [======================================================================================================================================================================================================================================================================] 100.00% Merge bucket 'default' succeeded Mutations merged: 20000, Mutations failed to merged: 0 Deletions merged: 0, Deletions failed to merged: 0 (4/6) Merging backup '2020-07-09T11_22_45.949655018-07_00' Copied all data in 8.025484135s (Avg. 218.21KB/Sec) 25000 items / 1.70MB [======================================================================================================================================================================================================================================================================] 100.00% Merge bucket 'default' succeeded Mutations merged: 25000, Mutations failed to merged: 0 Deletions merged: 0, Deletions failed to merged: 0 (5/6) Merging backup '2020-07-09T11_27_10.003238822-07_00' Copied all data in 8.545139194s (Avg. 260.33KB/Sec) 30000 items / 2.03MB [======================================================================================================================================================================================================================================================================] 100.00% Merge bucket 'default' succeeded Mutations merged: 30000, Mutations failed to merged: 0 Deletions merged: 0, Deletions failed to merged: 0 (6/6) Merging backup '2020-07-09T11_27_58.415507775-07_00' Copied all data in 8.492812354s (Avg. 302.44KB/Sec) 35000 items / 2.36MB [======================================================================================================================================================================================================================================================================] 100.00% Merge bucket 'default' succeeded Mutations merged: 35000, Mutations failed to merged: 0 Deletions merged: 0, Deletions failed to merged: 0 Merge completed successfully
            arunkumar Arunkumar Senthilnathan made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

              People

              Assignee:
              thuan Thuan Nguyen
              Reporter:
              thuan Thuan Nguyen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty