Details
-
Bug
-
Resolution: Fixed
-
Critical
-
5.5.0
-
Untriaged
-
-
Unknown
Description
Build : 5.5.0-2473
Indexer crashes while processing scans and building index at the same time along with ongoing kv mutations. Mostly this is because of computing aggregates. This happens when some indexes are dropped and then attempted to recreate. Building the first index being recreated doesnt complete for more than 2 hrs and then indexer crashes. All index nodes are running ~100% cpu during these 2 hrs or so.
Panic seen in indexer logs:
2018-04-13T03:22:56.248-07:00 [Info] Indexer::ReadMemstats Time Taken 4.493647ms
|
panic: runtime error: slice bounds out of range
|
|
|
goroutine 5611 [running]:
|
panic(0xe0d580, 0xc4200160d0)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1 fp=0xc47c7738f8 sp=0xc47c773868
|
runtime.panicslice()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:34 +0x6d fp=0xc47c773928 sp=0xc47c7738f8
|
github.com/couchbase/indexing/secondary/indexer.distinctCompare(0xc4422b4000, 0x2b, 0x3800, 0x7f512199f2d1, 0x2b, 0x2b, 0xc46cd2ce01)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_pipeline.go:654 +0x12a fp=0xc47c7739a8 sp=0xc47c773928
|
github.com/couchbase/indexing/secondary/indexer.(*entryCache).EqualsEntry(0xc489a40500, 0x7f512199f2d1, 0x2b, 0x2b, 0xc487760ac0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_pipeline.go:1262 +0x5e fp=0xc47c7739f0 sp=0xc47c7739a8
|
github.com/couchbase/indexing/secondary/indexer.computeGroupAggr(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x0, ...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_pipeline.go:794 +0x54b fp=0xc47c773b30 sp=0xc47c7739f0
|
github.com/couchbase/indexing/secondary/indexer.(*IndexScanSource).Routine.func2(0x7f512199f2d1, 0x2b, 0x2b, 0x6, 0xc481e842d0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_pipeline.go:237 +0xb9d fp=0xc47c773d98 sp=0xc47c773b30
|
github.com/couchbase/indexing/secondary/indexer.gather(0xc46bca8480, 0xc4610e42d0, 0x6, 0x6, 0xc45abd6d90, 0xc45abd6cb0, 0xc45abd6d20, 0xc45fb84fc0, 0xc43c58c2c0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_scatter.go:473 +0x319 fp=0xc47c773f58 sp=0xc47c773d98
|
runtime.goexit()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc47c773f60 sp=0xc47c773f58
|
created by github.com/couchbase/indexing/secondary/indexer.scanMultiple
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_scatter.go:65 +0x439
|
All indexer archived logs from the offending index node are attached along with cbcollectinfo from the cluster.