Details
-
Bug
-
Resolution: Fixed
-
Critical
-
5.5.0
-
centos2 cluster
-
Untriaged
-
No
Description
Build : 5.5.0-2958 (RC4)
Test : -test tests/2i/test_idx_rebalance_replica_vulcan_kv_opt.yml -scope tests/2i/scope_idx_rebalance_replica_vulcan_new.yml
Iteration : 2 (after ~18.5 hrs of test run)
At the end of the iteration, all indexes are dropped, so that new indexes can be created at the start of the next iteration. While indexes were being dropped, on one of the nodes, indexer process crashed with status 134. The following panic was seen in the logs:
Message: MsgStreamUpdate
|
Type: REMOVE_BUCKET_FROM_STREAM
|
Stream: MAINT_STREAM
|
Bucket: default
|
BuildTS: []
|
IndexList: []
|
RestartTs: <nil>
|
2018-07-10T16:33:58.888-07:00 [Error] MutationMgr::handleRemoveIndexListFromStream Stream Already Closed MAINT_STREAM
|
2018-07-10T16:33:58.888-07:00 [Error] Indexer::sendStreamUpdateToWorker - Error received from MutationMgr processing Msg
|
Message: MsgStreamUpdate
|
Type: REMOVE_BUCKET_FROM_STREAM
|
Stream: MAINT_STREAM
|
Bucket: default
|
BuildTS: []
|
IndexList: []
|
RestartTs: <nil> Err <nil>. Aborted.
|
panic: Inconsistent Internal State
|
|
|
goroutine 1 [running]:
|
panic(0xe22440, 0xc42014df80)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1 fp=0xc4cf4ba098 sp=0xc4cf4ba008
|
github.com/couchbase/indexing/secondary/common.CrashOnError(0x1826aa0, 0xc42014df80)
|
goproj/src/github.com/couchbase/indexing/secondary/common/util.go:404 +0x5c fp=0xc4cf4ba0c8 sp=0xc4cf4ba098
|
github.com/couchbase/indexing/secondary/indexer.(*indexer).removeIndexesFromStream(0xc42009a500, 0xc48e593440, 0x1, 0x1, 0xc42356ce06, 0x7, 0xc4235655c0, 0x20, 0x1, 0x5, ...)
|
goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:3424 +0x401 fp=0xc4cf4ba310 sp=0xc4cf4ba0c8
|
github.com/couchbase/indexing/secondary/indexer.(*indexer).sendStreamUpdateForDropIndex(0xc42009a500, 0xd96b88492b918839, 0xc2dc7b23d7db90dd, 0xc423567170, 0x24, 0xc42356ce00, 0x6, 0xc42356ce06, 0x7, 0xc4235655c0, ...)
|
goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:3367 +0x12c fp=0xc4cf4ba398 sp=0xc4cf4ba310
|
github.com/couchbase/indexing/secondary/indexer.(*indexer).cleanupIndex(0xc42009a500, 0xd96b88492b918839, 0xc2dc7b23d7db90dd, 0xc423567170, 0x24, 0xc42356ce00, 0x6, 0xc42356ce06, 0x7, 0xc4235655c0, ...)
|
goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:3031 +0xfb fp=0xc4cf4baa50 sp=0xc4cf4ba398
|
github.com/couchbase/indexing/secondary/indexer.(*indexer).handleDropIndex(0xc42009a500, 0x1827520, 0xc4aa7df110)
|
goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:2603 +0x9a6 fp=0xc4cf4bb570 sp=0xc4cf4baa50
|
github.com/couchbase/indexing/secondary/indexer.(*indexer).handleAdminMsgs(0xc42009a500, 0x1827520, 0xc4aa7df110)
|
goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1067 +0x355 fp=0xc4cf4bb620 sp=0xc4cf4bb570
|
github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0xc42009a500)
|
goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:577 +0x1e8 fp=0xc4cf4bb750 sp=0xc4cf4bb620
|
github.com/couchbase/indexing/secondary/indexer.NewIndexer(0xc42014e6c0, 0x24, 0xc420139de8, 0x1, 0x1)
|
goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:327 +0x1f2c fp=0xc4cf4bbc78 sp=0xc4cf4bb750
|
main.main()
|
goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:165 +0x12e1 fp=0xc4cf4bbf38 sp=0xc4cf4bbc78
|
runtime.main()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/proc.go:183 +0x1f4 fp=0xc4cf4bbf90 sp=0xc4cf4bbf38
|
runtime.goexit()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4cf4bbf98 sp=0xc4cf4bbf90
|
The continues to run after this and the indexer node had recovered.
This issue isnt seen consistently - it wasnt seen in iteration 1, but seen in iteration 2. Also, this issue wasnt seen in the previous runs as well. Not sure if this is a regression, or a race condition which never surfaced.
Logs on Supportal : http://supportal.couchbase.com/snapshot/dad95d23905d9834b7b373d62c7535cb::0
Attachments
Issue Links
- relates to
-
-
- Closed
-