Details
-
Bug
-
Resolution: Fixed
-
Critical
-
5.1.2
-
Triaged
-
-
No
-
Storage-Sprint-End-Dec-07-2018
Description
Build : 5.1.2-6016
Iteration : 1
Scale : 3
Test : -test tests/2i/test_idx_rebalance_replica_spock.yml -scope tests/2i/scope_idx_rebalance_replica_12nodes.yml
Cluster : Centos2
Seeing plasma crash on an indexer node while another indexer node failed over and removed from the cluster. The stack trace is similar to what was seen in MB-30492. This bug could also be related to MB-29952
Here is the panic msg seen on 172.23.96.254 :
2018-08-07T07:49:17.227-07:00 [Info] default/default_result/Backstore Plasma: logCleaner: starting... frag 31, data: 119312344, used: 174093758 log:(1535029530 - 1710063119)
|
2018-08-07T07:49:17.234-07:00 [Info] plasmaSlice::OpenSnapshot SliceId 0 IndexInstId 364261172130525603 Creating New Snapshot SnapshotInfo: count:2600642 committed:false
|
2018-08-07T07:49:17.234-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 364261172130525603 PartitionId: 0 SliceId: 0 Crc64: 12939052796789740186 (SnapshotInfo: count:2600642 committed:false) SnapCreateDur 8.521µs SnapOpenDur 46.944µs
|
2018-08-07T07:49:17.245-07:00 [Info] plasmaSlice::OpenSnapshot SliceId 0 IndexInstId 10856253011618663302 Creating New Snapshot SnapshotInfo: count:707118 committed:false
|
2018-08-07T07:49:17.245-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 10856253011618663302 PartitionId: 0 SliceId: 0 Crc64: 11492456263764330149 (SnapshotInfo: count:707118 committed:false) SnapCreateDur 8.272µs SnapOpenDur 58.486µs
|
2018-08-07T07:49:17.247-07:00 [Info] default/default_result/Mainstore Plasma: logCleaner: completed... frag 30, data: 151747691, used: 219922561, relocated: 69, retries: 0, skipped: 335 log:(881737763 - 1102335312)
|
2018-08-07T07:49:17.255-07:00 [Info] plasmaSlice::OpenSnapshot SliceId 0 IndexInstId 364261172130525603 Creating New Snapshot SnapshotInfo: count:2600645 committed:false
|
2018-08-07T07:49:17.255-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 364261172130525603 PartitionId: 0 SliceId: 0 Crc64: 12939052796789740186 (SnapshotInfo: count:2600645 committed:false) SnapCreateDur 8.473µs SnapOpenDur 58.681µs
|
panic: Corrupted item detected:
|
00000000 00 19 00 be 03 25 00 00 c0 13 00 00 00 30 33 44 |.....%.......03D|
|
|
|
goroutine 4482806 [running]:
|
panic(0xd06640, 0xc4400e5760)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/panic.go:500 +0x1a1 fp=0xc463f05788 sp=0xc463f056f8
|
github.com/couchbase/plasma.(*item).checkInvalidHeader(0xc43af32003)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/item.go:214 +0x188 fp=0xc463f05818 sp=0xc463f05788
|
github.com/couchbase/plasma.(*item).Key(0xc43af32003, 0x0, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/item.go:98 +0x5c fp=0xc463f05830 sp=0xc463f05818
|
github.com/couchbase/plasma.cmpItem(0x7fee46d06c60, 0xc43af32003, 0x7fee9a7d29c000)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/item.go:179 +0xaa fp=0xc463f05890 sp=0xc463f05830
|
github.com/couchbase/nitro/skiplist.compare(0xf36d20, 0x7fee46d06c60, 0xc43af32003, 0xc44ba21800)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/item.go:32 +0x88 fp=0xc463f058b8 sp=0xc463f05890
|
github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0xc42c52d860, 0xc43af32003, 0xf36d20, 0xc42ad6adb0, 0xc42c52d878, 0xc44ba21880)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:232 +0x190 fp=0xc463f05930 sp=0xc463f058b8
|
github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0xc42c52d860, 0xc43af32003, 0xf36d20, 0xc42ad6adb0, 0xc42c52d878, 0xc44ba21880, 0x7feec5bf5bc0, 0xc4306f3720)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x57 fp=0xc463f05970 sp=0xc463f05930
|
github.com/couchbase/plasma.newStoreContext.func1(0xc43af32003, 0xc42fb12600, 0x7fec21a383c0, 0x16a0120)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/pagetable.go:88 +0xae fp=0xc463f059c0 sp=0xc463f05970
|
github.com/couchbase/plasma.(*Plasma).CleanLSS.func1(0x5b940faa, 0x5b9415c8, 0xc44d356000, 0x61a, 0x21bf8, 0xd93d00, 0x0, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/lss_cleaner.go:87 +0x297 fp=0xc463f05ae0 sp=0xc463f059c0
|
github.com/couchbase/plasma.(*lsStore).RunCleaner.func1(0x5b940faa, 0xc44d356000, 0x61a, 0x21bf8, 0x0, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/lss.go:320 +0x6d fp=0xc463f05b50 sp=0xc463f05ae0
|
github.com/couchbase/plasma.(*lsStore).visitor(0xc429a401e0, 0x5b8d0851, 0x65ed7e0f, 0xc44c1b05a0, 0xc43ab6ae60, 0xc44c1b0540, 0x2)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/lss.go:350 +0x167 fp=0xc463f05bc0 sp=0xc463f05b50
|
github.com/couchbase/plasma.(*lsStore).RunCleaner(0xc429a401e0, 0xc44e086500, 0xc43ab6ae60, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/lss.go:334 +0x133 fp=0xc463f05c28 sp=0xc463f05bc0
|
github.com/couchbase/plasma.(*Plasma).CleanLSS(0xc423672580, 0xc442457a30, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/lss_cleaner.go:147 +0x57e fp=0xc463f05e50 sp=0xc463f05c28
|
github.com/couchbase/plasma.(*Plasma).lssCleanerDaemon(0xc423672580)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/lss_cleaner.go:192 +0x179 fp=0xc463f05f48 sp=0xc463f05e50
|
runtime.goexit()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc463f05f50 sp=0xc463f05f48
|
created by github.com/couchbase/plasma.New
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:439 +0x1115
|