Details
-
Task
-
Resolution: Duplicate
-
Major
-
5.0.0
-
Enterprise Edition 5.0.0 build 3083
Description
Steps to Repro(It must be noted tests were run on a vagrant VM).
1) Created a 3 node cluster - kv-index1-n1ql
2) Pumped in lot of data and created multiple secondary indexes.
3) added one more indexer to the cluster and did a gsi rebalance(lets say indexer2). All the indexes will still be in indexer1 as per gsi rebalance design.
4) started creating docs on the bucket until indexer1 went into paused state.
5) Now tried to rebalance out indexer2.
6) GSI rebalance failed.(which is expected)
However all the indexes on indexer1 got stuck in ready-warmup-ready loop after gsi rebalance. This happened maybe around 10 times. I was also seeing projector crashes along with indexer crashes. At this point my laptop hung and I had to restart it. After restart I manually restarted the vm's from virtual box. I started seeing the following panic in indexer
panic: snappy: corrupt input
goroutine 1238 [running]:
panic(0xcf0060, 0xc42012f020)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/panic.go:500 +0x1a1 fp=0xc445dc3598 sp=0xc445dc3508
github.com/couchbase/nitro/plasma.(*wCtx).decompress(0xc42aa88680, 0xc447a94002, 0x2810, 0x338da, 0xc43fcc67c0, 0x2812, 0x338dc, 0x119d0f1)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/plasma/plasma.go:695 +0x20c fp=0xc445dc3620 sp=0xc445dc3598
github.com/couchbase/nitro/plasma.(*page).unmarshalDelta(0xc444f7f000, 0xc447a94002, 0x2810, 0x338da, 0xc42aa88680, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/plasma/page.go:962 +0xaf fp=0xc445dc3728 sp=0xc445dc3620
github.com/couchbase/nitro/plasma.(*page).Unmarshal(0xc444f7f000, 0xc447a94002, 0x2810, 0x338da, 0xc42aa88680)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/plasma/page.go:958 +0x53 fp=0xc445dc3770 sp=0xc445dc3728
github.com/couchbase/nitro/plasma.(*Plasma).doRecovery.func1(0x119d0ed, 0xc447a94000, 0x2812, 0x338dc, 0x0, 0x0, 0x0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/plasma/plasma.go:511 +0xda fp=0xc445dc3878 sp=0xc445dc3770
github.com/couchbase/nitro/plasma.(*lsStore).visitor(0xc43fbb7200, 0x1017d0, 0x129b394, 0xc43fcc67a0, 0xc43fcc6780, 0xd4bc40, 0xc41ed8a601)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/plasma/lss.go:309 +0x167 fp=0xc445dc38e8 sp=0xc445dc3878
github.com/couchbase/nitro/plasma.(*lsStore).Visitor(0xc43fbb7200, 0xc43fcc67a0, 0xc43fcc6780, 0xc4476eb000, 0x1000)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/plasma/lss.go:298 +0x9d fp=0xc445dc3938 sp=0xc445dc38e8
github.com/couchbase/nitro/plasma.(*Plasma).doRecovery(0xc43cd07b80, 0xc444eb30e0, 0x21)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/plasma/plasma.go:557 +0x231 fp=0xc445dc3a00 sp=0xc445dc3938
github.com/couchbase/nitro/plasma.New(0x1e, 0x12c, 0x5, 0x4, 0xede030, 0xede000, 0xede038, 0xc445122a00, 0xc445122a10, 0xede000, ...)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/plasma/plasma.go:366 +0xe25 fp=0xc445dc3e08 sp=0xc445dc3a00
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func2(0xc4451229d0, 0xc440f6ec00, 0xc4451229c0, 0xc43e246700, 0xc4451229f0)
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:238 +0x91 fp=0xc445dc3f68 sp=0xc445dc3e08
runtime.goexit()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc445dc3f70 sp=0xc445dc3f68
created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:250 +0x14dc
Logs attached.
Attachments
Issue Links
- depends on
-
MB-25440 panic when indexer file is manually corrupted - handle data corruption
- Closed