Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
6.5.0
-
Untriaged
-
-
Unknown
Description
Build : 6.5.0-4380
Test : -test tests/integration/test_allFeatures_madhatter.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
Scale : 3
Issue seen on Day 3
During a swap rebalance operation of indexer nodes, indexer process on another indexer node - 172.23.104.69 - crashed with the following stack trace. This was around 5-6 mins after starting the rebalance.
panic: default/default_result_ratings_claims_pindex/Mainstore#17069669666856553551:3 : fatal: Empty read at offset 12807805592
|
|
goroutine 460319682 [running]:
|
panic(0xf72ba0, 0xc42ebeb1b0)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1
|
github.com/couchbase/plasma.(*Plasma).fatalPanic(0xc42851d400, 0xc4427f0d80, 0x20)
|
goproj/src/github.com/couchbase/plasma/plasma.go:1947 +0x310
|
github.com/couchbase/plasma.(*Plasma).CleanLSS.func1(0x2fb679a98, 0x2fb679a9c, 0x7f2b7b9bea9c, 0x0, 0xff564, 0x0, 0x0, 0x0, 0x0)
|
goproj/src/github.com/couchbase/plasma/lss_cleaner.go:251 +0xc24
|
github.com/couchbase/plasma.(*lsStore).RunCleaner.func1(0x2fb679a98, 0x7f2b7b9bea9c, 0x0, 0xff564, 0x0, 0xff564, 0x100000)
|
goproj/src/github.com/couchbase/plasma/lss.go:408 +0x6d
|
github.com/couchbase/plasma.(*lsStore).Visitor2(0xc4392ab970, 0x2fb679a98, 0x2fd553000, 0xc4427f0d60, 0xc430a7d080, 0xd, 0xc4427f0d20, 0x2)
|
goproj/src/github.com/couchbase/plasma/lss.go:443 +0x16e
|
github.com/couchbase/plasma.(*lsStore).RunCleaner(0xc4392ab970, 0xc443e91530, 0xc430a7d080, 0x0, 0x0, 0x0)
|
goproj/src/github.com/couchbase/plasma/lss.go:422 +0x142
|
github.com/couchbase/plasma.(*Plasma).CleanLSS(0xc42851d400, 0xc435a61f00, 0x0, 0x0)
|
goproj/src/github.com/couchbase/plasma/lss_cleaner.go:332 +0x465
|
github.com/couchbase/plasma.(*Plasma).lssCleanerDaemon(0xc42851d400)
|
goproj/src/github.com/couchbase/plasma/lss_cleaner.go:383 +0x179
|
created by github.com/couchbase/plasma.New
|
goproj/src/github.com/couchbase/plasma/plasma.go:629 +0x1335
|
Index Nodes in the cluster at the time of crash : 172.23.104.164:8091, 172.23.104.69:8091 (crash here) 172.23.104.70:8091 172.23.96.95:8091 172.23.99.11:8091 (rebalance out)