Description
Build: 7.1.0 build 1624
- Cluster with 3 nodes - kv, fts, index and n1ql
- Created bucket and 100 scopes and overall 750 collections
- Created a primary index with below:
CREATE PRIMARY INDEX `idx1` ON `bucket1`.`scope_0`.`coll_0`
Index not online. Seeing indexer crashing continuously and seeing below panic in the indexer log of node: 172.23.96.119
2021-12-02T12:32:10.397-08:00 [Fatal] StreamReader Panic Err runtime error: index out of range [4] with length 4
|
2021-12-02T12:32:10.397-08:00 [Info] TK SystemEvent INIT_STREAM bucket1:scope_0:coll_0 3 131860626422156 2 1 9 2
|
2021-12-02T12:32:10.397-08:00 [Fatal] /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:312 +0x25
|
github.com/couchbase/indexing/secondary/indexer.(*mutationStreamReader).panicHandler(0xc0072e0c30)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_reader.go:391 +0x21d
|
panic(0x1268d80, 0xc0061dd2c0)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/runtime/panic.go:965 +0x1b9
|
github.com/couchbase/indexing/secondary/indexer.(*streamWorker).updateVbuuidInFilter(0xc0059ab970, 0xc0072f8040)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_reader.go:1403 +0x46a
|
github.com/couchbase/indexing/secondary/indexer.(*streamWorker).handleSingleKeyVersion(0xc0059ab970, 0xc0061dd290, 0x16, 0x4, 0x5a2d5f29d263, 0x1, 0xc007f81bc0, 0xc006342905)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_reader.go:820 +0xb9f
|
github.com/couchbase/indexing/secondary/indexer.(*streamWorker).handleKeyVersions(0xc0059ab970, 0xc0061dd290, 0x16, 0x4, 0x5a2d5f29d263, 0x1, 0xc0006afba0, 0x3, 0x4, 0x448b05)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_reader.go:697 +0xae
|
github.com/couchbase/indexing/secondary/indexer.(*streamWorker).start(0xc0059ab970)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_reader.go:680 +0xa8
|
created by github.com/couchbase/indexing/secondary/indexer.CreateMutationStreamReader
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_reader.go:133 +0x60a
|
|
2021-12-02T12:32:10.397-08:00 [Error] MutationMgr::listenWorkerMsgs Stream Reader Received for StreamId: INIT_STREAM, Err: runtime error: index out of range [4] with length 4
|
2021-12-02T12:32:10.397-08:00 [Fatal] Indexer::handleWorkerMsgs Fatal Error On Worker Channel &{streamId:3 err:{code:9 severity:0 category:4 cause:{x:4 y:4 signed:true code:0} msg:}}
|
panic: interface conversion: indexer.Message is *indexer.MsgStreamError, not *indexer.MsgError
|
|
goroutine 1 [running]:
|
github.com/couchbase/indexing/secondary/indexer.(*indexer).handleWorkerMsgs(0xc0001ac000, 0x15304a0, 0xc00734a480)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1382 +0x1dd7
|
github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0xc0001ac000)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1061 +0x773
|
github.com/couchbase/indexing/secondary/indexer.NewIndexer(0xc0000df1d0, 0x4, 0x1381543, 0xf, 0xc0001d3e78)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:492 +0x2528
|
main.main()
|
goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:182 +0x1565
|
Initializing write barrier = 8000
|
Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1638478288/collectinfo-2021-12-02T205130-ns_1%40172.23.96.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1638478288/collectinfo-2021-12-02T205130-ns_1%40172.23.96.118.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1638478288/collectinfo-2021-12-02T205130-ns_1%40172.23.96.119.zip