[MB-40235] [CBM] cbbackupmgr in upgrade version (6.6.0) throws segmentation violation error when merging old backup repo (5.5.0) - Couchbase database

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: 6.6.0
Fix Version/s: 6.6.0
Component/s: tools
Labels:
- approved-for-6.6.0
Environment:
Centos 7.6

Triage:
Untriaged
Operating System:
Centos 64-bit
Story Points:
1
Is this a Regression?:
Unknown

Description

Install Couchbase server 5.5.0 on 4 centos 7.6 servers.

Create 2 nodes cluster (224 and 225) and default bucket.

Load data to default bucket.

Run several backups with cbbackupmgr (5.5.0) to backup data

Fresh install Couchbase server 6.6.0 on 2 centos servers (226 and 227)

Add 2 6.6.0-7846 nodes to cluster above and remove 2 nodes 224 and 225

Cluster becomes 6.6.0.

Run some backups with cbbackupmgr (6.6.0-7846)

Run merge with cbbackupmgr (6.6.0-7846), merge failed with segfault error

2020-06-30T19:57:36.189-07:00 (Cmd) merge --archive /tmp/entbackup_172.23.121.224 --repo backup --start 2020-06-30T19_44_37.802124479-07_00 --end 2020-06-30T19_56_43.262610544-07_00

2020-06-30T19:57:36.189-07:00 (Cmd) mounted archive with id: unknown

2020-06-30T19:57:36.191-07:00 (Stats) Starting stat gathering - stat timestamp: 1593572256

2020-06-30T19:57:36.191-07:00 Merging backups [2020-06-30T19_44_37.802124479-07_00 2020-06-30T19_45_12.203175007-07_00 2020-06-30T19_45_48.336465942-07_00 2020-06-30T19_53_57.816275267-07_00 2020-06-30T19_54_54.873738339-07_00 2020-06-30T19_55_51.398617954-07_00 2020-06-30T19_56_43.262610544-07_00]

2020-06-30T19:57:36.192-07:00 Creating the merge directory for backup backup

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0xab14ae]

goroutine 1 [running]:

github.com/couchbase/backup/archive.(*Archive).MergeIncrementalBackups(0xc0002807c0, 0x7ffcbe78edc3, 0x6, 0x7ffcbe78edd2, 0x23, 0x7ffcbe78edfc, 0x23, 0x3, 0x0, 0x0, ...)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/backup/archive/archive.go:747 +0x1d4e

main.(*MergeContext).Run(0xc0002b8000)

	goproj/src/github.com/couchbase/backup/cmd/cbbackupmgr/merge.go:111 +0x520

github.com/couchbase/cbflag.(*Command).parseFlags(0xc0002c2a80, 0xc000207490, 0xc0000c4020, 0x8, 0x8)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbflag/command.go:241 +0x17bf

github.com/couchbase/cbflag.(*Command).parse(0xc0002c2a80, 0xc000207490, 0xc0000c4020, 0x8, 0x8)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbflag/command.go:92 +0xd2

github.com/couchbase/cbflag.(*Command).parseCommands(0xc0002c2c40, 0xc000207490, 0xc0000c4010, 0x9, 0x9)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbflag/command.go:104 +0x12c

github.com/couchbase/cbflag.(*Command).parse(0xc0002c2c40, 0xc000207490, 0xc0000c4010, 0x9, 0x9)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbflag/command.go:90 +0x101

github.com/couchbase/cbflag.(*CLI).Parse(0xc0002b8090, 0xc0000c4000, 0xa, 0xa)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbflag/cli.go:59 +0x18a

main

I will upload logs soon

Attachments

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

Attachments

172.23.121.224-20200630-2024-diag.zip
2.04 MB
30/Jun/20 8:27 PM
172.23.121.225-20200630-2026-diag.zip
1.77 MB
30/Jun/20 8:31 PM
172.23.121.226-20200630-2028-diag.zip
9.49 MB
30/Jun/20 8:32 PM
172.23.121.227-20200630-2032-diag.zip
6.23 MB
30/Jun/20 8:36 PM
cbbackupmgr-collectinfo-entbackup_172.23.121.224-2020-06-30T202213.zip
3.77 MB
30/Jun/20 8:23 PM

Issue Links

relates to

MB-40246 Have CV run some backwards compatible archive sanity testing

Open

Options
- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

Gerrit Reviews

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

Ascending order - Click to sort in descending order

Thuan Nguyen created issue - 30/Jun/20 8:20 PM

Thuan Nguyen made changes - 30/Jun/20 8:23 PM

Field	Original Value	New Value
Summary	[CBM] cbbackupmgr in upgrade version (6.6.0) throws segmentation violation error when merging old backup repo (5.0.0)	[CBM] cbbackupmgr in upgrade version (6.6.0) throws segmentation violation error when merging old backup repo (5.5.0)

Thuan Nguyen made changes - 30/Jun/20 8:23 PM

Attachment

cbbackupmgr-collectinfo-entbackup_172.23.121.224-2020-06-30T202213.zip [ 99475 ]

Thuan Nguyen made changes - 30/Jun/20 8:27 PM

Attachment

172.23.121.224-20200630-2024-diag.zip [ 99476 ]

Thuan Nguyen made changes - 30/Jun/20 8:31 PM

Attachment

172.23.121.225-20200630-2026-diag.zip [ 99477 ]

Thuan Nguyen made changes - 30/Jun/20 8:32 PM

Attachment

172.23.121.226-20200630-2028-diag.zip [ 99478 ]

Thuan Nguyen made changes - 30/Jun/20 8:36 PM

Attachment

172.23.121.227-20200630-2032-diag.zip [ 99479 ]

Patrick Varley made changes - 01/Jul/20 2:35 AM

Assignee

Patrick Varley [ pvarley ]

James Lee [ james.lee ]

James Lee made changes - 01/Jul/20 4:40 AM

Status

Open [ 1 ]

In Progress [ 3 ]

James Lee made changes - 01/Jul/20 6:47 AM

Link

This issue relates to MB-40246 [ MB-40246 ]

Till Westmann made changes - 01/Jul/20 8:49 AM

Link

This issue blocks MB-38724 [ MB-38724 ]

Till Westmann made changes - 01/Jul/20 8:50 AM

Labels

approved-for-6.6.0

James Lee made changes - 01/Jul/20 8:51 AM

Resolution		Fixed [ 1 ]
Status	In Progress [ 3 ]	Resolved [ 5 ]

James Lee made changes - 01/Jul/20 8:51 AM

Assignee

James Lee [ james.lee ]

Thuan Nguyen [ thuan ]

Hide

Permalink

Thuan Nguyen added a comment - 01/Jul/20 9:10 AM

I will verify it when the build with this fixes is ready

Show

Thuan Nguyen added a comment - 01/Jul/20 9:10 AM I will verify it when the build with this fixes is ready

Hide

Permalink

Couchbase Build Team added a comment - 01/Jul/20 9:51 AM

Build couchbase-server-6.6.0-7859 contains backup commit 9a9c4cc with commit message:
~~MB-40235~~ Avoid segfault when merging 5.5.x backups

Show

Couchbase Build Team added a comment - 01/Jul/20 9:51 AM Build couchbase-server-6.6.0-7859 contains backup commit 9a9c4cc with commit message: MB-40235 Avoid segfault when merging 5.5.x backups

Hide

Permalink

Couchbase Build Team added a comment - 06/Jul/20 1:50 AM

Build couchbase-server-7.0.0-2533 contains backup commit 9a9c4cc with commit message:
~~MB-40235~~ Avoid segfault when merging 5.5.x backups

Show

Couchbase Build Team added a comment - 06/Jul/20 1:50 AM Build couchbase-server-7.0.0-2533 contains backup commit 9a9c4cc with commit message: MB-40235 Avoid segfault when merging 5.5.x backups

Hide

Permalink

Arunkumar Senthilnathan added a comment - 09/Jul/20 11:32 AM

Verified with steps in description in 6.6.0-7861:

[root@node1-mad-hatter-testing-centos7 bin]# ./cbbackupmgr merge -a /tmp/entbackup/ -r backup --start 2020-07-09T11_13_30.250764423-07_00 --end 2020-07-09T11_27_58.415507775-07_00
(1/6) Merging backup '2020-07-09T11_13_30.250764423-07_00'
Copied all data in 4.922372899s (Avg. 195.95KB/Sec) 10000 items / 783.81KB
[======================================================================================================================================================================================================================================================================] 100.00%
Merge bucket 'default' succeeded
Mutations merged: 10000, Mutations failed to merged: 0
Deletions merged: 0, Deletions failed to merged: 0

(2/6) Merging backup '2020-07-09T11_13_52.941198062-07_00'
Copied all data in 4.908067918s (Avg. 286.28KB/Sec) 15000 items / 1.12MB
[======================================================================================================================================================================================================================================================================] 100.00%
Merge bucket 'default' succeeded
Mutations merged: 15000, Mutations failed to merged: 0
Deletions merged: 0, Deletions failed to merged: 0

(3/6) Merging backup '2020-07-09T11_14_09.662222502-07_00'
Copied all data in 5.059106021s (Avg. 301.29KB/Sec) 20000 items / 1.47MB
[======================================================================================================================================================================================================================================================================] 100.00%
Merge bucket 'default' succeeded
Mutations merged: 20000, Mutations failed to merged: 0
Deletions merged: 0, Deletions failed to merged: 0

(4/6) Merging backup '2020-07-09T11_22_45.949655018-07_00'
Copied all data in 8.025484135s (Avg. 218.21KB/Sec) 25000 items / 1.70MB
[======================================================================================================================================================================================================================================================================] 100.00%
Merge bucket 'default' succeeded
Mutations merged: 25000, Mutations failed to merged: 0
Deletions merged: 0, Deletions failed to merged: 0

(5/6) Merging backup '2020-07-09T11_27_10.003238822-07_00'
Copied all data in 8.545139194s (Avg. 260.33KB/Sec) 30000 items / 2.03MB
[======================================================================================================================================================================================================================================================================] 100.00%
Merge bucket 'default' succeeded
Mutations merged: 30000, Mutations failed to merged: 0
Deletions merged: 0, Deletions failed to merged: 0

(6/6) Merging backup '2020-07-09T11_27_58.415507775-07_00'
Copied all data in 8.492812354s (Avg. 302.44KB/Sec) 35000 items / 2.36MB
[======================================================================================================================================================================================================================================================================] 100.00%
Merge bucket 'default' succeeded
Mutations merged: 35000, Mutations failed to merged: 0
Deletions merged: 0, Deletions failed to merged: 0
Merge completed successfully

Show

Arunkumar Senthilnathan added a comment - 09/Jul/20 11:32 AM Verified with steps in description in 6.6.0-7861: [root@node1-mad-hatter-testing-centos7 bin] # ./cbbackupmgr merge -a /tmp/entbackup/ -r backup --start 2020-07-09T11_13_30.250764423-07_00 --end 2020-07-09T11_27_58.415507775-07_00 (1/6) Merging backup '2020-07-09T11_13_30.250764423-07_00' Copied all data in 4.922372899s (Avg. 195.95KB/Sec) 10000 items / 783.81KB [======================================================================================================================================================================================================================================================================] 100.00% Merge bucket 'default' succeeded Mutations merged: 10000, Mutations failed to merged: 0 Deletions merged: 0, Deletions failed to merged: 0 (2/6) Merging backup '2020-07-09T11_13_52.941198062-07_00' Copied all data in 4.908067918s (Avg. 286.28KB/Sec) 15000 items / 1.12MB [======================================================================================================================================================================================================================================================================] 100.00% Merge bucket 'default' succeeded Mutations merged: 15000, Mutations failed to merged: 0 Deletions merged: 0, Deletions failed to merged: 0 (3/6) Merging backup '2020-07-09T11_14_09.662222502-07_00' Copied all data in 5.059106021s (Avg. 301.29KB/Sec) 20000 items / 1.47MB [======================================================================================================================================================================================================================================================================] 100.00% Merge bucket 'default' succeeded Mutations merged: 20000, Mutations failed to merged: 0 Deletions merged: 0, Deletions failed to merged: 0 (4/6) Merging backup '2020-07-09T11_22_45.949655018-07_00' Copied all data in 8.025484135s (Avg. 218.21KB/Sec) 25000 items / 1.70MB [======================================================================================================================================================================================================================================================================] 100.00% Merge bucket 'default' succeeded Mutations merged: 25000, Mutations failed to merged: 0 Deletions merged: 0, Deletions failed to merged: 0 (5/6) Merging backup '2020-07-09T11_27_10.003238822-07_00' Copied all data in 8.545139194s (Avg. 260.33KB/Sec) 30000 items / 2.03MB [======================================================================================================================================================================================================================================================================] 100.00% Merge bucket 'default' succeeded Mutations merged: 30000, Mutations failed to merged: 0 Deletions merged: 0, Deletions failed to merged: 0 (6/6) Merging backup '2020-07-09T11_27_58.415507775-07_00' Copied all data in 8.492812354s (Avg. 302.44KB/Sec) 35000 items / 2.36MB [======================================================================================================================================================================================================================================================================] 100.00% Merge bucket 'default' succeeded Mutations merged: 35000, Mutations failed to merged: 0 Deletions merged: 0, Deletions failed to merged: 0 Merge completed successfully

Arunkumar Senthilnathan made changes - 09/Jul/20 11:32 AM

Status

Resolved [ 5 ]

Closed [ 6 ]

People

Assignee:: Thuan Nguyen

Reporter:: Thuan Nguyen

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 30/Jun/20 8:20 PM

Updated:: 09/Jul/20 11:32 AM

Resolved:: 01/Jul/20 8:51 AM

Gerrit Reviews

There are no open Gerrit changes

Show There are 2 closed Gerrit changes

Hide There are 2 closed Gerrit changes

MB-40235 Avoid segfault when merging 5.5.x backups: Gerrit Review:

Merge branch 'mad-hatter' into master: Gerrit Review:

PagerDuty

Couchbase Server

Details

Description

Attachments

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty