Details
Description
Build
4.5.0-2047,2083,2100
Testcase
./testrunner -i INI_FILE.ini -p get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,cluster=D:F:F,GROUP=P1 -t fts.stable_topology_fts.StableTopFTS.index_wiki,items=20000,lang=ES,analyzer=es,GROUP=P1
Multiple crashes seen in a matter of few mins. Breakpad dmp generated for every crash. With every crash, memory usage increases and at some point cbft gets killed by OOM killer.
gdb on dmp generated core produces -
#0 0x00007f476d9e9bcb in avl_first (tree=0x7f47602fd468, node=0x7f4760333890) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/avltree.cc:222
|
#1 avl_remove (tree=0x7f47602fd468, node=0x7f4760333890) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/avltree.cc:549
|
#2 0x00007f476da2b586 in wal_commit (txn=Unhandled dwarf expression opcode 0xf3
|
) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/wal.cc:1280
|
#3 0x00007f476da0d2d9 in _fdb_commit (handle=0x7f474c51bc00, opt=Unhandled dwarf expression opcode 0xf3
|
) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/forestdb.cc:4075
|
#4 0x00007f476da292a8 in fdb_end_transaction (fhandle=Unhandled dwarf expression opcode 0xf3
|
) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/transaction.cc:182
|
#5 0x0000000000dc0ebb in _cgo_3fe29c8698dc_Cfunc_fdb_end_transaction (v=0xc820693938) at /home/couchbase/jenkins/workspace/watson-unix/godeps/src/github.com/couchbase/goforestdb/tx.go:80
|
#6 0x0000000000471ba0 in runtime.asmcgocall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:582
|
#7 0x0000000000000008 in ?? ()
|
#8 0x000000c8206938b8 in ?? ()
|
#9 0x000000000041529a in runtime.cgocall (fn=0xc820020a00, arg=0x0, ~r2=538273664) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/cgocall.go:110
|
#10 0x00000000004449a0 in runtime.startTheWorldWithSema () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:983
|
#11 0x000000c820020a00 in ?? ()
|
#12 0x0000000000000000 in ?? ()
|
In cbft log -
fatal error: unexpected signal during runtime execution
|
[signal 0xb code=0x80 addr=0x0 pc=0x7f1608b0b9cb]
|
|
|
runtime stack:
|
runtime.throw(0x11f7c00, 0x2a)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/panic.go:530 +0x90 fp=0x7f15db3fdac0 sp=0x7f15db3fdaa8
|
runtime.sigpanic()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sigpanic_unix.go:12 +0x5a fp=0x7f15db3fdb10 sp=0x7f15db3fdac0
|
|
|
goroutine 124 [syscall, locked to thread]:
|
runtime.cgocall(0xdbdd40, 0xc8205de938, 0xc800000000)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/cgocall.go:123 +0x11b fp=0xc8205de8e8 sp=0xc8205de8b8
|
github.com/couchbase/goforestdb._Cfunc_fdb_end_transaction(0x7f15d8ac2240, 0x7f15d8ac2200, 0x0)
|
??:0 +0x41 fp=0xc8205de938 sp=0xc8205de8e8
|
github.com/couchbase/goforestdb.(*File).EndTransaction(0xc820208a50, 0x1286a00, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/watson-unix/godeps/src/github.com/couchbase/goforestdb/tx.go:37 +0x1ef fp=0xc8205de9c0 sp=0xc8205de938
|
github.com/couchbase/goforestdb.(*KVStore).ExecuteBatch.func1(0xc8205deae8, 0xc820aedf00, 0xc820419700)
|
/home/couchbase/jenkins/workspace/watson-unix/godeps/src/github.com/couchbase/goforestdb/kvbatch.go:89 +0x3f fp=0xc8205de9f8 sp=0xc8205de9c0
|
github.com/couchbase/goforestdb.(*KVStore).ExecuteBatch(0xc820aedf00, 0xc820aedf60, 0xc8205dec00, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/watson-unix/godeps/src/github.com/couchbase/goforestdb/kvbatch.go:116 +0x52e fp=0xc8205dead0 sp=0xc8205de9f8
|
github.com/blevesearch/blevex/forestdb.(*Writer).ExecuteBatch(0xc82047e780, 0x7f16065dee40, 0xc820aedf20, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/blevex/forestdb/writer.go:74 +0x580 fp=0xc8205dec80 sp=0xc8205dead0
|
github.com/blevesearch/bleve/index/store/moss.(*llStore).update(0xc82012a680, 0x7f16065d3a20, 0xc820c28800, 0x0, 0x0, 0x0, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/store/moss/lower.go:256 +0x76c fp=0xc8205dee58 sp=0xc8205dec80
|
github.com/blevesearch/bleve/index/store/moss.initLowerLevelStore.func1(0x7f16065d3a20, 0xc820c28800, 0x0, 0x0, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/watson-unix/godeps/src/github.com/blevesearch/bleve/index/store/moss/lower.go:66 +0x59 fp=0xc8205deea0 sp=0xc8205dee58
|
github.com/couchbase/moss.(*collection).runPersister(0xc8205ca0f0)
|
/home/couchbase/jenkins/workspace/watson-unix/godeps/src/github.com/couchbase/moss/persister.go:69 +0x2a7 fp=0xc8205defa8 sp=0xc8205deea0
|
runtime.goexit()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc8205defb0 sp=0xc8205defa8
|
created by github.com/couchbase/moss.(*collection).Start
|
/home/couchbase/jenkins/workspace/watson-unix/godeps/src/github.com/couchbase/moss/collection.go:88 +0x63
|
Attaching the very first breakpad dump created and its core -
https://s3.amazonaws.com/bugdb/jira/MB-19179/4616759a-1109-88d8-1e0924de-7fc79306.dmp
https://s3.amazonaws.com/bugdb/jira/MB-19179/mycore
Attachments
Issue Links
- is triggered by
-
MB-19192 cbft test framework must cleanup old files before new run
-
- Closed
-