Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
Cheshire-Cat
-
Untriaged
-
-
1
-
Unknown
-
Plasma-Sprint-May-30-2021
Description
Build : 7.0.0-4603
Test : Longevity (-test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml)
Scale : 2
Iteration : 1st
On 172.23.96.252, seeing the following errors leading to indexer crash while rebalance is going on. In-memory compression is not turned on in this test.
2021-03-04T10:06:09.702-08:00 [Fatal] plasmaSlice::handleCommandsWorker: panic detected while processing mutation for operation 1 key = <ud>(^H^F_sync:unusedSeq:56555095^@^@^@)</ud> docid = <ud>(_sync:unusedSeq:56555095)</ud> Index sg_syncDocs_x1, Bucket default, IndexInstId 13320080993727085165, PartitionId 0
|
...
|
...
|
2021-03-04T10:06:09.725-08:00 [Fatal] plasmaSlice::handleCommandsWorker: panic detected while processing mutation for operation 1 key = <ud>(^H^F_sync:unusedSeq:56555067^@^@^@)</ud> docid = <ud>(_sync:unusedSeq:56555067)</ud> Index sg_syncDocs_x1, Bucket default, IndexInstId 13320080993727085165, PartitionId 0
|
2021-03-04T10:06:09.744-08:00 [Fatal] plasmaSlice::handleCommandsWorker: panic detected while processing mutation for operation 1 key = <ud>(^H^F_sync:unusedSeq:56555090^@^@^@)</ud> docid = <ud>(_sync:unusedSeq:56555090)</ud> Index sg_syncDocs_x1, Bucket default, IndexInstId 13320080993727085165, PartitionId 0
|
2021-03-04T10:06:09.767-08:00 [Fatal] goroutine 176366914 [running]:
|
github.com/couchbase/indexing/secondary/logging.(*destination).StackTraceAll(...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:175
|
github.com/couchbase/indexing/secondary/logging.StackTraceAll(0x20ff5f0, 0x2120001)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:317 +0x6d
|
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker.func1(0xc03c7aded0, 0xc012e01700)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:596 +0x424
|
panic(0x10a1e00, 0xc03c787c70)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
|
github.com/couchbase/plasma.(*Shard).raisePanic(...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1011
|
github.com/couchbase/plasma.(*LSSCtx).raiseCorrupted(0xc0062c3040, 0x1488ee0, 0xc03c787c70)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/lssctx.go:179 +0x206
|
github.com/couchbase/plasma.(*Plasma).fatalPanic(0xc00a9b0580, 0xc03c7822a0, 0x21)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1354 +0x293
|
github.com/couchbase/plasma.(*page).lookup(0xc01e86af40, 0xc03c7ad730, 0x7f0c5629c520, 0x0, 0xcbd800)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/page.go:766 +0x1f4
|
github.com/couchbase/plasma.(*page).Lookup(0xc01e86af40, 0x7f0c5629c520, 0x0, 0x7f0c53ae8800)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/page.go:667 +0x105
|
github.com/couchbase/plasma.(*Writer).Lookup(0xc0193a47b0, 0x7f0c5629c520, 0x18, 0x20, 0x0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/writer.go:135 +0x103
|
github.com/couchbase/plasma.(*Writer).LookupKV(0xc0193a47b0, 0xc00be266c0, 0x18, 0x20, 0xc0073ef940, 0xc0073ef978, 0x434e81, 0x12f0150, 0xc0073ef988)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mvcc.go:395 +0x10c
|
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).deleteSecIndex(0xc012e01700, 0xc00be266c0, 0x18, 0x20, 0xc00be267c0, 0x1d, 0x20, 0x2, 0x0, 0x100000000000000)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:1141 +0x13b
|
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insertSecIndex(0xc012e01700, 0xc00be267c0, 0x1d, 0x20, 0xc00be266c0, 0x18, 0x20, 0x2, 0x4c2500, 0xc01c295480, ...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:775 +0xb18
|
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insert(0xc012e01700, 0xc00be267c0, 0x1d, 0x20, 0xc00be266c0, 0x18, 0x20, 0x2, 0xc01ea76200, 0xc01c295480, ...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:733 +0x164
|
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker(0xc012e01700, 0x2)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:610 +0x58a
|
created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initWriters
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:2980 +0x521
|
Attachments
Issue Links
| For Gerrit Dashboard: MB-44749 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 148985,1 | MB-44749: Add test for gorountine leak | unstable | plasma | Status: NEW | 0 | -1 |
| 149526,2 | MB-44749: Add unit test for concurrent writer and snapshot | unstable | plasma | Status: NEW | 0 | -1 |