Indexer stuck with n1ql_thr_array_indexing_Q3_20M_between_gsi_false

cat /opt/couchbase/var/lib/couchbase/logs/indexer.log '|' grep -i hanging -A 10 -B 10

cat: invalid option -- 'i'

Try `cat --help' for more information.

[root@hadesindex-s522 ~]# cat /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep -i hanging -A 10 -B 10

2016-05-01T19:33:11.710-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 38400 Milliseconds Vbucket 716

2016-05-01T19:33:11.740-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 38430 Milliseconds Vbucket 716

2016-05-01T19:33:11.770-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 38460 Milliseconds Vbucket 716

2016-05-01T19:33:11.800-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 38490 Milliseconds Vbucket 716

2016-05-01T19:33:11.830-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 38520 Milliseconds Vbucket 716

2016-05-01T19:33:11.860-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 38550 Milliseconds Vbucket 716

2016-05-01T19:33:11.890-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 38580 Milliseconds Vbucket 716

2016-05-01T19:33:11.920-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 38610 Milliseconds Vbucket 716

2016-05-01T19:33:11.950-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 38640 Milliseconds Vbucket 716

Caught SIGSEGV in libforestdb at (null)

Hanging process...

2016-05-01T19:33:11.980-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 38670 Milliseconds Vbucket 716

2016-05-01T19:33:12.010-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 38700 Milliseconds Vbucket 716

2016-05-01T19:33:12.029-07:00 [Info] logWriterStat:: 817346399017306606 FlushedCount 210000 QueuedCount 13343

2016-05-01T19:33:12.040-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 38730 Milliseconds Vbucket 716

2016-05-01T19:33:12.070-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 38760 Milliseconds Vbucket 716

2016-05-01T19:33:12.100-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 38790 Milliseconds Vbucket 716

2016-05-01T19:33:12.130-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 38820 Milliseconds Vbucket 716

2016-05-01T19:33:12.160-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 38850 Milliseconds Vbucket 716

2016-05-01T19:33:12.190-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 38880 Milliseconds Vbucket 716

2016-05-01T19:33:12.220-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 38910 Milliseconds Vbucket 716

[root@hadesindex-s522 couchbase]# gdb core.24245 

GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)

Copyright (C) 2010 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-redhat-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>...

Reading symbols from /opt/couchbase/bin/indexer...Reading symbols from /usr/lib/debug/opt/couchbase/bin/indexer.debug...done.

done.

[New Thread 24279]

[New Thread 24248]

[New Thread 24272]

[New Thread 24260]

[New Thread 24271]

[New Thread 24251]

[New Thread 24280]

[New Thread 24249]

[New Thread 24263]

[New Thread 24257]

[New Thread 24262]

[New Thread 24253]

[New Thread 24264]

[New Thread 24268]

[New Thread 24254]

[New Thread 24269]

[New Thread 24284]

[New Thread 24265]

[New Thread 24245]

[New Thread 24252]

[New Thread 24250]

[New Thread 24259]

[New Thread 24275]

[New Thread 24267]

[New Thread 24276]

[New Thread 24261]

[New Thread 24277]

[New Thread 24273]

[New Thread 24266]

[New Thread 24258]

[New Thread 24256]

[New Thread 24278]

[New Thread 24270]

[New Thread 24255]

[New Thread 24274]

warning: Can't read pathname for load map: Input/output error.

Reading symbols from /opt/couchbase/lib/libforestdb.so...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libforestdb.so.debug...done.

done.

Loaded symbols for /opt/couchbase/lib/libforestdb.so

Reading symbols from /opt/couchbase/lib/libjemalloc.so.2...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libjemalloc.so.2.debug...done.

done.

Loaded symbols for /opt/couchbase/lib/libjemalloc.so.2

Reading symbols from /opt/couchbase/lib/libsnappy.so.1.2.0...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libsnappy.so.1.2.0.debug...done.

done.

Loaded symbols for /opt/couchbase/lib/libsnappy.so.1.2.0

Reading symbols from /opt/couchbase/lib/libplatform.so.0.1.0...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libplatform.so.0.1.0.debug...done.

done.

Loaded symbols for /opt/couchbase/lib/libplatform.so.0.1.0

warning: Can't read pathname for load map: Input/output error.

warning: Can't read pathname for load map: Input/output error.

Core was generated by `/opt/couchbase/bin/indexer -vbuckets=1024 -cluster=127.0.0.1:8091 -adminPort=91'.

Program terminated with signal 6, Aborted.

#0  runtime.raise () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:110

110	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s: No such file or directory.

	in /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s

warning: Missing auto-load scripts referenced in section .debug_gdb_scripts

of file /usr/lib/debug/opt/couchbase/bin/indexer.debug

Use `info auto-load python [REGEXP]' to list them.

Missing separate debuginfos, use: debuginfo-install couchbase-server-4.5.0-2428.x86_64

(gdb)  t a a bt

Thread 35 (Thread 24274):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc821209108, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc821209108) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x0000000000437788 in runtime.gcstopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1739

#5  0x00000000004379b5 in runtime.findrunnable (gp=0xffffffff00000000, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1804

#6  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#7  0x00000000004389db in runtime.park_m (gp=0xc8238a5e00) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#8  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#9  0x000000c8238a41c0 in ?? ()

#10 0x0000000000000000 in ?? ()

Thread 34 (Thread 24255):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e930 in runtime.futexsleep (addr=0x1984ed8, val=0, ns=8908744) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:57

#2  0x00000000004134db in runtime.notetsleep_internal (n=0x1984ed8, ns=8908744, ~r2=180) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:174

#3  0x0000000000413646 in runtime.notetsleepg (n=0x1984ed8, ns=8908744, ~r2=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:206

#4  0x000000000044f0ce in runtime.timerproc () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/time.go:209

#5  0x0000000000464201 in runtime.goexit () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:1998

#6  0x0000000000000000 in ?? ()

Thread 33 (Thread 24270):

#0  0x00000033d720b98e in ?? ()

#1  0x0000000000000000 in ?? ()

Thread 32 (Thread 24278):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc82604a108, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc82604a108) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x1, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x00000000004389db in runtime.park_m (gp=0xc828336480) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c82003aa00 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 31 (Thread 24256):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc821192508, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc821192508) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x1, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x00000000004389db in runtime.park_m (gp=0xc828336480) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c82001ea00 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 30 (Thread 24258):

#0  runtime.epollwait () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:440

#1  0x000000000042e6b4 in runtime.netpoll (block=true, ~r1=0x0) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/netpoll_epoll.go:67

#2  0x0000000000437fac in runtime.findrunnable (gp=0x0, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1955

#3  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#4  0x00000000004389db in runtime.park_m (gp=0xc8211d3980) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#5  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#6  0x000000c8211d4f40 in ?? ()

#7  0x0000000000000000 in ?? ()

Thread 29 (Thread 24266):

---Type <return> to continue, or q <return> to quit---

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc821208d08, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc821208d08) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x0, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x0000000000438e19 in runtime.goexit0 (gp=0xc828336900) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2207

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c82120ac40 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 28 (Thread 24273):

#0  0x00000033d720b98e in ?? ()

#1  0x0000000000000000 in ?? ()

Thread 27 (Thread 24277):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc82603a108, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc82603a108) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x0, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x0000000000438e19 in runtime.goexit0 (gp=0xc82610fe00) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2207

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c820039500 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 26 (Thread 24261):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc821218108, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc821218108) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x100000000, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x00000000004389db in runtime.park_m (gp=0xc820000300) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c820028000 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 25 (Thread 24276):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc826028108, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc826028108) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x1, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x00000000004389db in runtime.park_m (gp=0xc85bdd2180) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c826024f40 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 24 (Thread 24267):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc821219108, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc821219108) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x100000000, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x00000000004389db in runtime.park_m (gp=0xc828325380) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c82121adc0 in ?? ()

#9  0x0000000000000000 in ?? ()

---Type <return> to continue, or q <return> to quit---

Thread 23 (Thread 24275):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc8211f3108, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc8211f3108) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x0, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x00000000004389db in runtime.park_m (gp=0xc82497ef00) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c82009d240 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 22 (Thread 24259):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc8211f2108, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc8211f2108) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0xc800000000, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x00000000004389db in runtime.park_m (gp=0xc86cd42a80) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c820025500 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 21 (Thread 24250):

#0  syscall.Syscall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/syscall/asm_linux_amd64.s:27

#1  0x000000000062685f in syscall.read (fd=0, p=..., n=0, err=...) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/syscall/zsyscall_linux_amd64.go:783

#2  0x00000000006256dd in syscall.Read (fd=0, p=..., n=0, err=...) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/syscall/syscall_unix.go:161

#3  0x000000000046f493 in os.(*File).read (f=0xc820064008, b=..., n=0, err=...) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/os/file_unix.go:228

#4  0x000000000046c7ca in os.(*File).Read (f=0xc820064008, b=..., n=0, err=...) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/os/file.go:95

#5  0x0000000000496b67 in github.com/couchbase/indexing/secondary/common.ExitOnStdinClose ()

    at /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:318

#6  0x0000000000464201 in runtime.goexit () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:1998

#7  0x0000000000000000 in ?? ()

Thread 20 (Thread 24252):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc820067508, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc820067508) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x1, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x00000000004389db in runtime.park_m (gp=0xc828336480) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c820020000 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 19 (Thread 24245):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0x19885e8, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0x19885e8) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x11bf1afff, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x00000000004389db in runtime.park_m (gp=0xc85bdd2180) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x0000000001986500 in runtime.work ()

#9  0x00007fff7ce50970 in ?? ()

#10 0x0000000001986540 in runtime.work ()

#11 0x0000000000435c12 in runtime.mstart () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1065

#12 0x00000000004615b8 in runtime.rt0_go () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:149

---Type <return> to continue, or q <return> to quit---

#13 0x000000000000000d in ?? ()

#14 0x00007fff7ce50a88 in ?? ()

#15 0x000000000000000d in ?? ()

#16 0x00007fff7ce50a88 in ?? ()

#17 0x0000000000000000 in ?? ()

Thread 18 (Thread 24265):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc821208508, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc821208508) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x0000000000437788 in runtime.gcstopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1739

#5  0x00000000004379b5 in runtime.findrunnable (gp=0x100000000, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1804

#6  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#7  0x00000000004389db in runtime.park_m (gp=0xc82495c000) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#8  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#9  0x000000c82120a940 in ?? ()

#10 0x0000000000000000 in ?? ()

Thread 17 (Thread 24284):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc869f19d08, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc869f19d08) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x11bf1bdff, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x00000000004389db in runtime.park_m (gp=0xc828336480) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c820029500 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 16 (Thread 24269):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc821193508, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc821193508) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x1, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x00000000004389db in runtime.park_m (gp=0xc85bdd2180) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c82119cc40 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 15 (Thread 24254):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc821192108, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc821192108) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x1, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x00000000004389db in runtime.park_m (gp=0xc85bdd2180) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c82001ea00 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 14 (Thread 24268):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc82119e508, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc82119e508) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x0, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

---Type <return> to continue, or q <return> to quit---

#6  0x0000000000438e19 in runtime.goexit0 (gp=0xc8211b3c80) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2207

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c82120b240 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 13 (Thread 24264):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc821218d08, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc821218d08) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x0, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x00000000004389db in runtime.park_m (gp=0xc83a391b00) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c82121aac0 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 12 (Thread 24253):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc820067908, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc820067908) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x1, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x00000000004389db in runtime.park_m (gp=0xc85bdd2180) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c82001ea00 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 11 (Thread 24262):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc821218508, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc821218508) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x1, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x0000000000438e19 in runtime.goexit0 (gp=0xc85bdd2180) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2207

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c82121a4c0 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 10 (Thread 24257):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc821192908, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc821192908) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x7fe800000001, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x0000000000438e19 in runtime.goexit0 (gp=0xc82619f200) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2207

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c82001ea00 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 9 (Thread 24263):

#0  0x00007ffa13cf49b9 in crc32_8 (data=0x7ff6834b3000, len=<value optimized out>, prev_value=Unhandled dwarf expression opcode 0xf3

) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/utils/crc32.cc:344

#1  0x00007ffa13cad0b8 in _flush_dirty_blocks (fname_item=0x7fe9a4870f80, flush_all=true, immutables_only=false, sync=true) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/blockcache.cc:561

#2  0x00007ffa13cc52ca in filemgr_commit_bid (file=0x7fe9a482ba00, bid=30147058, bmp_revnum=0, sync=Unhandled dwarf expression opcode 0xf3

) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/filemgr.cc:2306

#3  0x00007ffa13cd1a96 in _fdb_commit (handle=0x7ff9e4a17e00, opt=Unhandled dwarf expression opcode 0xf3

) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/forestdb.cc:4304

#4  0x0000000000c7d7f7 in _cgo_ea7381d1a5e0_Cfunc_fdb_commit (v=0xc824993410) at /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/fdb/file.go:81

#5  0x0000000000463180 in runtime.asmcgocall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:582

#6  0x0000000000000008 in ?? ()

#7  0x000000c824993390 in ?? ()

---Type <return> to continue, or q <return> to quit---

#8  0x000000000040652a in runtime.cgocall (fn=0xc82121a7c0, arg=0x0, ~r2=555853696) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/cgocall.go:110

#9  0x0000000000435ba0 in runtime.startTheWorldWithSema () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:983

#10 0x000000c82121a7c0 in ?? ()

#11 0x0000000000000000 in ?? ()

Thread 8 (Thread 24249):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0x19a6940, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x000000000041342a in runtime.notetsleep_internal (n=0x19a6940, ns=-1, ~r2=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:161

#3  0x0000000000413646 in runtime.notetsleepg (n=0x19a6940, ns=-1, ~r2=true) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:206

#4  0x0000000000446682 in os/signal.signal_recv (~r0=0) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sigqueue.go:116

#5  0x0000000000c62238 in os/signal.loop () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/os/signal/signal_unix.go:22

#6  0x0000000000464201 in runtime.goexit () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:1998

#7  0x0000000000000000 in ?? ()

Thread 7 (Thread 24280):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc82604a508, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc82604a508) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x1, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x00000000004389db in runtime.park_m (gp=0xc8211b3e00) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c826202940 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 6 (Thread 24251):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc82009a108, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc82009a108) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x1, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x00000000004389db in runtime.park_m (gp=0xc828325380) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c82001ea00 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 5 (Thread 24271):

#0  0x00000033d720b98e in ?? ()

#1  0x0000000000000000 in ?? ()

Thread 4 (Thread 24260):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e893 in runtime.futexsleep (addr=0xc821208108, val=0, ns=-1) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:40

#2  0x0000000000413374 in runtime.notesleep (n=0xc821208108) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:145

#3  0x0000000000436d2b in runtime.stopm () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1535

#4  0x00000000004380b9 in runtime.findrunnable (gp=0x1, inheritTime=false) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1973

#5  0x000000000043873f in runtime.schedule () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2072

#6  0x00000000004389db in runtime.park_m (gp=0xc828325380) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:2137

#7  0x000000000046171b in runtime.mcall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:233

#8  0x000000c820026a00 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 3 (Thread 24272):

#0  0x00000033d720b98e in ?? ()

#1  0x0000000000000000 in ?? ()

Thread 2 (Thread 24248):

#0  runtime.futex () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:303

#1  0x000000000042e930 in runtime.futexsleep (addr=0x1985ad0, val=0, ns=60000000000) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/os1_linux.go:57

---Type <return> to continue, or q <return> to quit---

#2  0x00000000004134db in runtime.notetsleep_internal (n=0x1985ad0, ns=60000000000, ~r2=128) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:174

#3  0x00000000004135cb in runtime.notetsleep (n=0x1985ad0, ns=60000000000, ~r2=true) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/lock_futex.go:194

#4  0x000000000043ce27 in runtime.sysmon () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:3477

#5  0x0000000000435d0c in runtime.mstart1 () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1095

#6  0x0000000000435c12 in runtime.mstart () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:1065

#7  0x0000000000c7eb84 in crosscall_amd64 () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/cgo/gcc_amd64.S:35

#8  0x0000000000000003 in ?? ()

#9  0x0000000000000000 in ?? ()

Thread 1 (Thread 24279):

#0  runtime.raise () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sys_linux_amd64.s:110

#1  0x00000000004441c8 in runtime.dieFromSignal (sig=6) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/signal1_unix.go:192

#2  0x00000000004442ef in runtime.crash () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/signal1_unix.go:247

#3  0x0000000000431d4e in runtime.dopanic_m (gp=0xc8211d5980, pc=4396880, sp=140641287776672) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/panic.go:642

#4  0x0000000000458392 in runtime.dopanic.func1 () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/panic.go:517

#5  0x00000000004617eb in runtime.systemstack () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:307

#6  0x00000000004316b1 in runtime.dopanic (unused=0) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/panic.go:518

#7  0x0000000000431750 in runtime.throw (s=...) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/panic.go:530

#8  0x00000000004460ca in runtime.sigpanic () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/sigpanic_unix.go:12

#9  0x00000033d72093a0 in ?? ()

#10 0x00007ffa13a7e376 in je_malloc_mutex_lock (tsd=Unhandled dwarf expression opcode 0xf3

) at include/jemalloc/internal/mutex.h:85

#11 je_tcache_bin_flush_small (tsd=Unhandled dwarf expression opcode 0xf3

) at src/tcache.c:115

#12 0x00007ffa13a4ee71 in je_tcache_dalloc_small (ptr=0x7fe8cc153b00) at include/jemalloc/internal/tcache.h:376

#13 je_arena_dalloc (ptr=0x7fe8cc153b00) at include/jemalloc/internal/arena.h:1271

#14 je_idalloctm (ptr=0x7fe8cc153b00) at include/jemalloc/internal/jemalloc_internal.h:1009

#15 je_iqalloc (ptr=0x7fe8cc153b00) at include/jemalloc/internal/jemalloc_internal.h:1033

#16 ifree (ptr=0x7fe8cc153b00) at src/jemalloc.c:1745

#17 je_free (ptr=0x7fe8cc153b00) at src/jemalloc.c:1839

#18 0x00007ffa13cef97a in _wal_release_items (file=0x7fe92e299000, flush_items=Unhandled dwarf expression opcode 0xf3

) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/wal.cc:1480

#19 wal_release_flushed_items (file=0x7fe92e299000, flush_items=Unhandled dwarf expression opcode 0xf3

) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/wal.cc:1555

#20 0x00007ffa13cd568d in _fdb_compact_move_docs (handle=0x7fe999ffb330, new_file=Unhandled dwarf expression opcode 0xf3

) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/forestdb.cc:5415

#21 0x00007ffa13cd5ea1 in _fdb_compact_move_docs (rhandle=0x7ff9e4a17fc0, new_file=Unhandled dwarf expression opcode 0xf3

) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/forestdb.cc:5188

#22 _fdb_compact_move_docs_upto_marker (rhandle=0x7ff9e4a17fc0, new_file=Unhandled dwarf expression opcode 0xf3

) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/forestdb.cc:5618

#23 0x00007ffa13cd6eb0 in _fdb_compact_file (handle=0x7ff9e4a17fc0, new_file=0x7fe92e299000, new_bhandle=Unhandled dwarf expression opcode 0xf3

) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/forestdb.cc:6958

#24 0x00007ffa13cd81d0 in fdb_compact_file (fhandle=0x7ff9e4a14600, new_filename=Unhandled dwarf expression opcode 0xf3

) at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/forestdb.cc:6837

#25 0x00007ffa13cd8398 in _fdb_compact (fhandle=0x7ff9e4a14600, new_filename=<value optimized out>, marker=28032480, clone_docs=Unhandled dwarf expression opcode 0xf3

)

    at /home/couchbase/jenkins/workspace/watson-unix/forestdb/src/forestdb.cc:7199

#26 0x0000000000c7d8b7 in _cgo_ea7381d1a5e0_Cfunc_fdb_compact_upto (v=0xc870edfb38) at /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/fdb/file.go:112

#27 0x0000000000463180 in runtime.asmcgocall () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:582

#28 0x0000000000000008 in ?? ()

#29 0x000000c870edfab8 in ?? ()

#30 0x000000000040652a in runtime.cgocall (fn=0xc820030000, arg=0x0, ~r2=555571584) at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/cgocall.go:110

#31 0x0000000000435ba0 in runtime.startTheWorldWithSema () at /home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:983

#32 0x000000c820030000 in ?? ()

#33 0x0000000000000000 in ?? ()

(gdb) 

(gdb)