Details
-
Bug
-
Resolution: Fixed
-
Critical
-
5.0.0, 5.1.1
-
Untriaged
-
-
Unknown
-
Storage-Sprint-End-Jun-15-2018
Description
Build : 5.1.1-5720
The indexer logs on one of the index nodes - 172.23.96.148 shows a lot of panic messages like the following couple of them :
2018-05-22T18:09:35.033-07:00 [Info] Indexer::initPartnInstance Initialized Partition:
|
Index: 17689712609126445135 Partition: PartitionId: 0 Endpoints: [:9105]
|
2018-05-22T18:09:36.266-07:00 [Info] plasmaSlice::doRecovery SliceId 0 IndexInstId 17689712609126445135 Recovering from recovery point ..
|
2018-05-22T18:09:36.303-07:00 [Info] plasmaSlice::doRecovery SliceId 0 IndexInstId 17689712609126445135 Warmup took 1.106382471s
|
2018-05-22T18:09:36.303-07:00 [Info] plasmaSlice:NewplasmaSlice Created New Slice Id 0 IndexInstId 17689712609126445135 WriterThreads 8
|
2018-05-22T18:09:36.303-07:00 [Info] Indexer::initPartnInstance Initialized Slice:
|
Index: 17689712609126445135 Slice: SliceId: 0 File: /data/@2i/CUSTOMER_o1_result_17689712609126445135_0.index Index: 17689712609126445135
|
2018-05-22T18:09:36.303-07:00 [Info] Indexer::initPartnInstance Initialized Partition:
|
Index: 2308736630918829209 Partition: PartitionId: 0 Endpoints: [:9105]
|
panic: found missing page
|
|
goroutine 800 [running]:
|
panic(0xd01400, 0xc4255894a0)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/panic.go:500 +0x1a1 fp=0xc423639df8 sp=0xc423639d68
|
github.com/couchbase/plasma.(*Plasma).doRecovery.func2(0xdd11c0, 0x7fc000f9ed40, 0x0, 0x0, 0xffffffffffffffff, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:646 +0x220 fp=0xc423639e98 sp=0xc423639df8
|
github.com/couchbase/plasma.(*Plasma).VisitPartition(0xc4237f8580, 0x0, 0x0, 0xffffffffffffffff, 0xc42a635920, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/page_visitor.go:71 +0x2e7 fp=0xc423639f20 sp=0xc423639e98
|
github.com/couchbase/plasma.(*Plasma).PageVisitor.func1(0xc425589480, 0xc425589490, 0x1, 0x1, 0xc4237f8580, 0xc42a635920, 0x0, 0x0, 0xffffffffffffffff)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/page_visitor.go:40 +0x89 fp=0xc423639f68 sp=0xc423639f20
|
runtime.goexit()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc423639f70 sp=0xc423639f68
|
created by github.com/couchbase/plasma.(*Plasma).PageVisitor
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/page_visitor.go:41 +0x1ae
|
panic: invalid last page
|
|
goroutine 823 [running]:
|
panic(0xd01400, 0xc42ed521d0)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/panic.go:500 +0x1a1 fp=0xc42f68b870 sp=0xc42f68b7e0
|
github.com/couchbase/plasma.(*Plasma).doRecovery(0xc4237f8580, 0xc4306c9360, 0x21)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:662 +0x41d fp=0xc42f68b938 sp=0xc42f68b870
|
github.com/couchbase/plasma.New(0xc8, 0x190, 0x19, 0x4, 0xf30198, 0xf30160, 0xf301a0, 0xc42b4e7d10, 0xc42b4e7d20, 0xf30160, ...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:429 +0xf9a fp=0xc42f68bde8 sp=0xc42f68b938
|
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0xc42a004850, 0xc429fec600, 0xc42a004840, 0xc426d6bb00, 0xc42a004860)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:233 +0x91 fp=0xc42f68bf78 sp=0xc42f68bde8
|
runtime.goexit()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42f68bf80 sp=0xc42f68bf78
|
created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:249 +0x15e2
|
[goport(/opt/couchbase/bin/indexer)] 2018/05/22 18:09:40 child process exited with status 134
|
The indexer process continously crashes on this node.
2i data backup : /root/backup_2i_data on 172.23.96.148
Note : We havent seen this issue during the system test runs done in 5.0.0 or 5.1.0, or even 5.5.0
Attachments
Issue Links
- relates to
-
MB-30242 CLONE (Backport MB-29800) - Rebalance Failures & Plasma panics seen in 5.1.1 system test run - "panic: duplicate index node"
- Closed