Details
-
Bug
-
Resolution: Duplicate
-
Major
-
None
-
5.5.0
-
centos longevity cluster
-
Untriaged
-
Unknown
Description
Build : 5.5.0-2958 (RC4)
Test : -test tests/integration/test_allFeatures_vulcan.yml -scope tests/integration/scope_Xattrs_Vulcan.yml
Iteration : 8th (3d 10h)
Scale : 3
Seeing plasma crash on an indexer node while another indexer node was being added to the cluster. Here is the panic msg seen on 172.23.96.148 :
2018-07-16T08:03:50.013-07:00 [Info] default/default_result_ratings_claims_pindex/Mainstore#2205988093120078337:2 Plasma: logCleaner: completed... frag 30, data: 56230393, used: 81245282, relocated: 41974, retries: 2, skipped: 24787 log:(327591395 - 409776128)
|
2018-07-16T08:03:50.014-07:00 [Info] DISTRICT/o2_claims/Mainstore#1269962683878245180:0 Plasma: logCleaner: starting... frag 31, data: 184459859, used: 267594849 log:(779277710 - 1046900736)
|
2018-07-16T08:03:50.025-07:00 [Info] PrunePartitions: bucket DISTRICT stream MAINT_STREAM
|
2018-07-16T08:03:50.025-07:00 [Info] PrunePartition: Prune instance 2205988093120078337 partitions [2]
|
2018-07-16T08:03:50.025-07:00 [Warn] PurnePartition: Index Instance 2205988093120078337 is not in bucket DISTRICT. Do not prune now.
|
2018-07-16T08:03:50.031-07:00 [Info] DISTRICT/o2_claims/Backstore#1269962683878245180:0 Plasma: logCleaner: starting... frag 31, data: 141122673, used: 205101810 log:(1483347137 - 1688621056)
|
2018-07-16T08:03:50.031-07:00 [Info] PrunePartitions: bucket DISTRICT stream MAINT_STREAM
|
2018-07-16T08:03:50.031-07:00 [Info] PrunePartition: Prune instance 2205988093120078337 partitions [2]
|
2018-07-16T08:03:50.031-07:00 [Warn] PurnePartition: Index Instance 2205988093120078337 is not in bucket DISTRICT. Do not prune now.
|
2018-07-16T08:03:50.046-07:00 [Info] DISTRICT/o2_claims/Mainstore#1269962683878245180:0 Plasma: logCleaner: completed... frag 30, data: 184433435, used: 266633209, relocated: 129804, retries: 9, skipped: 63049 log:(779277710 - 1046900736)
|
2018-07-16T08:03:50.063-07:00 [Info] DISTRICT/o2_claims/Backstore#1269962683878245180:0 Plasma: logCleaner: completed... frag 30, data: 141230293, used: 204625176, relocated: 511389, retries: 34, skipped: 130496 log:(1483347137 - 1688621056)
|
panic: Corrupted item detected:
|
00000000 7b 22 74 6f 70 6f 6c 6f 67 79 4b 65 79 73 22 3a |{"topologyKeys":|
|
|
|
goroutine 71863328 [running]:
|
panic(0xded7a0, 0xc4fe52d730)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1 fp=0xc4f2e27708 sp=0xc4f2e27678
|
github.com/couchbase/plasma.(*item).checkInvalidHeader(0x7fb6a790e640)
|
goproj/src/github.com/couchbase/plasma/item.go:214 +0x188 fp=0xc4f2e27798 sp=0xc4f2e27708
|
github.com/couchbase/plasma.(*item).Key(0x7fb6a790e640, 0x0, 0x0, 0x0)
|
goproj/src/github.com/couchbase/plasma/item.go:98 +0x5c fp=0xc4f2e277b0 sp=0xc4f2e27798
|
github.com/couchbase/plasma.cmpItem(0x7fb89761b558, 0x7fb6a790e640, 0x7fb89761b558)
|
goproj/src/github.com/couchbase/plasma/item.go:179 +0xaa fp=0xc4f2e27810 sp=0xc4f2e277b0
|
github.com/couchbase/plasma.(*page).equal(0xc4c2a60100, 0x7fb89761b558, 0x7fb6a790e640, 0x7fba0f62b668, 0x7fb89761b558)
|
goproj/src/github.com/couchbase/plasma/page.go:496 +0x42 fp=0xc4f2e27838 sp=0xc4f2e27810
|
github.com/couchbase/plasma.(*page).Lookup(0xc4c2a60100, 0x7fb6a790e640, 0x0)
|
goproj/src/github.com/couchbase/plasma/page.go:515 +0x4d0 fp=0xc4f2e27a78 sp=0xc4f2e27838
|
github.com/couchbase/plasma.(*Writer).Lookup(0xc446a7d000, 0x7fb6a790e640, 0x13, 0x13, 0x0)
|
goproj/src/github.com/couchbase/plasma/plasma.go:1733 +0xcf fp=0xc4f2e27b00 sp=0xc4f2e27a78
|
github.com/couchbase/plasma.(*Writer).LookupKV(0xc446a7d000, 0xc44844f220, 0x13, 0x13, 0xc4f2e27f26, 0x3, 0x3, 0xc4f2e27f20, 0x3)
|
goproj/src/github.com/couchbase/plasma/mvcc.go:332 +0x109 fp=0xc4f2e27b98 sp=0xc4f2e27b00
|
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).deleteSecIndex(0xc426943200, 0xc44844f220, 0x13, 0x13, 0x0, 0x0, 0x0, 0x1, 0x0, 0x5b4cb300)
|
goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:839 +0x122 fp=0xc4f2e27c60 sp=0xc4f2e27b98
|
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).delete(0xc426943200, 0xc44844f220, 0x13, 0x13, 0x1, 0x1)
|
goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:797 +0xf8 fp=0xc4f2e27cc8 sp=0xc4f2e27c60
|
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker(0xc426943200, 0x1)
|
goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:487 +0x419 fp=0xc4f2e27f40 sp=0xc4f2e27cc8
|
runtime.goexit()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4f2e27f48 sp=0xc4f2e27f40
|
created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initWriters
|
goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:2149 +0x3b3
|
Supportal Link : http://supportal.couchbase.com/snapshot/85277d8dde1234d3615f37137f90624c::2
Index data backup is stored at /index/idxbackup.zip on 172.23.96.148
Indexer recovered after the crash and the subsequent rebalance operation went fine.