Details
Description
4.5 GA, 12 days up, indexer crashed 3 hours after encountering MB-19975.
Unfortunately logs rolled over before I was able to back them up. From ui:
ervice 'indexer' exited with status 1. Restarting. Messages: runtime.goexit()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc8a7472fb0 sp=0xc8a7472fa8
|
created by net/http.(*Transport).dialConn
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/net/http/transport.go:853 +0x10a6
|
[goport] 2016/06/20 11:21:48 /opt/couchbase/bin/indexer terminated: signal: aborted (core dumped)
|
|
Service 'indexer' exited with status 1. Restarting. Messages: 2016-06-20T11:26:55.528-07:00 [Info] SCAN##294930 RESPONSE status:(error = Indexer In Bootstrap State. Please retry the request later.), requestId: c4cc7fa2-54e6-45aa-a7b5-b9aae7f807dd
|
2016-06-20T11:26:55.528-07:00 [Info] SCAN##294931 REQUEST defnId:3960761121304801551, index:/, type:scan, span:range (%!s(<nil>),%!s(<nil>) incl:both), limit:9223372036854775807, requestId:6766cd7d-5eda-4266-8c1b-6b6a38ad496d
|
2016-06-20T11:26:55.528-07:00 [Info] SCAN##294931 RESPONSE status:(error = Indexer In Bootstrap State. Please retry the request later.), requestId: 6766cd7d-5eda-4266-8c1b-6b6a38ad496d
|
2016-06-20T11:26:55.531-07:00 [Fatal] Indexer::validateIndexInstMap Bucket CUSTOMER not ready even after max retries. Restarting indexer.
|
[goport] 2016/06/20 11:26:55 /opt/couchbase/bin/indexer terminated: exit status 1
|
Looks like forestdb was doing a rollback at time of exiting.
172.23.105.60:/data/core.indexer.12929
Thread 12 (Thread 0x7f00357fa700 (LWP 12981)):
|
#0 je_free (ptr=0x7efd07667e60) at src/jemalloc.c:1837
|
#1 0x00007f0335eb1223 in _flush_dirty_blocks (fname_item=0x7efd073b2280, flush_all=true, immutables_only=false, sync=true)
|
at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/blockcache.cc:620
|
#2 0x00007f0335ec982a in filemgr_commit_bid (file=0x7efd08a0ef00, bid=18446744073709551615, bmp_revnum=5, sync=<optimized out>, log_callback=0x7f0084a834b0)
|
at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/filemgr.cc:2321
|
#3 0x00007f0335eea8ef in _fdb_kvs_remove (fhandle=fhandle@entry=0x7f0084a18580, kvs_name=<optimized out>, rollback_recreate=true)
|
at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/kv_instance.cc:2058
|
#4 0x00007f0335eecf4f in fdb_kvs_rollback (handle_ptr=0xc82509d668, seqnum=0) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/kv_instance.cc:2168
|
#5 0x00007f0335ed699d in fdb_rollback (handle_ptr=0xc82509d668, seqnum=0) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/forestdb.cc:1194
|
#6 0x0000000000c8a257 in _cgo_ea7381d1a5e0_Cfunc_fdb_rollback (v=0xcafe84f820)
|
at /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/fdb/commit.go:50
|
#7 0x0000000000464010 in runtime.asmcgocall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:582
|
#8 0x0000000000000008 in ?? ()
|
#9 0x000000cafe84f7a0 in ?? ()
|
#10 0x00000000004073ba in runtime.cgocall (fn=0xc820022a00, arg=0x0, ~r2=629030016) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/cgocall.go:110
|
#11 0x0000000000436a30 in runtime.startTheWorldWithSema () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:983
|
#12 0x000000c820022a00 in ?? ()
|
#13 0x0000000000000000 in ?? ()
|
Attachments
Issue Links
- relates to
-
MB-19975 Index scan timed out on 10 row query
- Closed