Basic crash dump analysis of /tmp//core.mc:writer_5.14369. Please send the file to support@couchbase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 couchbase couchbase 588888 Jul 27 13:31 /opt/couchbase/bin/memcached a407a6f1c29df3917ecc389552d5b093 /opt/couchbase/bin/memcached -rw------- 1 couchbase couchbase 4001189888 Aug 2 06:45 /tmp//core.mc:writer_5.14369 8b259e5b81c3e109a3fe0f69849d2840 /tmp//core.mc:writer_5.14369 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/couchbase/bin/memcached...(no debugging symbols found)...done. [New Thread 14602] [New Thread 14413] [New Thread 14603] [New Thread 14606] [New Thread 14379] [New Thread 14417] [New Thread 14390] [New Thread 14412] [New Thread 14381] [New Thread 14415] [New Thread 14369] [New Thread 14414] [New Thread 14601] [New Thread 14416] [New Thread 14600] [New Thread 14605] [New Thread 14596] [New Thread 14418] [New Thread 14599] [New Thread 14595] [New Thread 14419] [New Thread 14598] [New Thread 14593] [New Thread 14591] [New Thread 14597] [New Thread 14594] [New Thread 14604] Reading symbols from /opt/couchbase/bin/../lib/libauditd.so.0.1.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libauditd.so.0.1.0 Reading symbols from /opt/couchbase/bin/../lib/libmcd_time.so.1.0.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libmcd_time.so.1.0.0 Reading symbols from /opt/couchbase/bin/../lib/libdirutils.so.0.1.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libdirutils.so.0.1.0 Reading symbols from /opt/couchbase/bin/../lib/libmcd_util.so.1.0.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libmcd_util.so.1.0.0 Reading symbols from /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1 Reading symbols from /opt/couchbase/bin/../lib/libcJSON.so.1.1.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libcJSON.so.1.1.0 Reading symbols from /opt/couchbase/bin/../lib/libJSON_checker.so.1.0.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libJSON_checker.so.1.0.0 Reading symbols from /opt/couchbase/bin/../lib/libplatform.so.0.1.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libplatform.so.0.1.0 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1 Reading symbols from /opt/couchbase/bin/../lib/libjemalloc.so.2...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libjemalloc.so.2 Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libssl.so.10 Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libcrypto.so.10 Reading symbols from /opt/couchbase/bin/../lib/libnuma.so.1...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libnuma.so.1 Reading symbols from /opt/couchbase/bin/../lib/libevent_pthreads-2.0.so.5...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_pthreads-2.0.so.5 Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libgssapi_krb5.so.2 Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libk5crypto.so.3 Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so Reading symbols from /opt/couchbase/bin/../lib/../lib/nobucket.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/nobucket.so Reading symbols from /opt/couchbase/bin/../lib/../lib/default_engine.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/default_engine.so Reading symbols from /opt/couchbase/bin/../lib/../lib/ep.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/ep.so Reading symbols from /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so Reading symbols from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so Reading symbols from /opt/couchbase/bin/../lib/../lib/../lib/libv8.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/../lib/libv8.so Reading symbols from /opt/couchbase/bin/../lib/../lib/../lib/libicuuc.so.54...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/../lib/libicuuc.so.54 Reading symbols from /opt/couchbase/bin/../lib/../lib/../lib/libicudata.so.54...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/../lib/libicudata.so.54 Reading symbols from /opt/couchbase/bin/../lib/../lib/../lib/libicui18n.so.54...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/../lib/libicui18n.so.54 Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. Program terminated with signal 11, Segmentation fault. #0 0x00007f353b1c922e in ?? () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so Thread 27 (Thread 0x7f3466df8700 (LWP 14604)): #0 0x00007f3541dba98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f353b74c1db in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007f353b7502c0 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007f353b70c996 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007f353b70ca71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007f353b714628 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f346d5fe700 (LWP 14594)): #0 0x00007f3541dba98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f353b1d2939 in ?? () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #2 0x00007f353b1d2e49 in ?? () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #3 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f346b3ff700 (LWP 14597)): #0 0x00007f3541dba98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f353b74c1db in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007f353b7502c0 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007f353b70c996 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007f353b70ca71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007f353b714628 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f3530fff700 (LWP 14591)): #0 0x00007f3541dba98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f353b7260c0 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #3 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f346dfff700 (LWP 14593)): #0 0x00007f3541dba98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f353b1d2939 in ?? () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #2 0x00007f353b1d2e49 in ?? () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #3 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f346a9fe700 (LWP 14598)): #0 0x00007f3541dba98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f353b74c1db in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007f353b7502c0 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007f353b70c996 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007f353b70ca71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007f353b714628 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f35327f8700 (LWP 14419)): #0 0x00007f3541dba5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f354088a2fc in std::condition_variable::wait(std::unique_lock&) () from /usr/lib64/libstdc++.so.6 #2 0x0000000000437d40 in ?? () #3 0x00007f35421d9bab in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f346cbfd700 (LWP 14595)): #0 0x00007f3541dba98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f353b1d2939 in ?? () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #2 0x00007f353b1d2e49 in ?? () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #3 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f3469ffd700 (LWP 14599)): #0 0x00007f3541dba98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f353b74c1db in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007f353b7502c0 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007f353b70c996 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007f353b70ca71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007f353b714628 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f35331f9700 (LWP 14418)): #0 0x00007f3541dba5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f354088a2fc in std::condition_variable::wait(std::unique_lock&) () from /usr/lib64/libstdc++.so.6 #2 0x0000000000437d40 in ?? () #3 0x00007f35421d9bab in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f346c1fc700 (LWP 14596)): #0 0x00007f3541dba98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f353b1d2939 in ?? () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #2 0x00007f353b1d2e49 in ?? () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #3 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f34663f7700 (LWP 14605)): #0 0x00007f3541dba98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f353b74c1db in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007f353b7502c0 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007f353b70c996 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007f353b70ca71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007f353b714628 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f34695fc700 (LWP 14600)): #0 0x00007f3541dba98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f353b74c1db in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007f353b7502c0 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007f353b70c996 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007f353b70ca71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007f353b714628 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f35345fb700 (LWP 14416)): #0 0x00007f3541dba5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f354088a2fc in std::condition_variable::wait(std::unique_lock&) () from /usr/lib64/libstdc++.so.6 #2 0x0000000000437d40 in ?? () #3 0x00007f35421d9bab in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f3468bfb700 (LWP 14601)): #0 0x00007f3541dbd264 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f3541db8508 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00007f3541db83d7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007f353b1df4a0 in ?? () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #4 0x00007f353b20d1bc in ?? () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #5 0x00007f353b1ec254 in ?? () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #6 0x00007f353b1ed428 in fdb_open () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #7 0x00007f353b77d626 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #8 0x00007f353b6db4b0 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #9 0x00007f353b6db70e in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #10 0x00007f353b71447e in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #11 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #12 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f35359fd700 (LWP 14414)): #0 0x00007f354008eef3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f354154ce93 in ?? () from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 #2 0x00007f3541539f42 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 #3 0x0000000000423524 in ?? () #4 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f3543444820 (LWP 14369)): #0 0x00007f354008eef3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f354154ce93 in ?? () from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 #2 0x00007f3541539f42 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 #3 0x0000000000418984 in _start () Thread 10 (Thread 0x7f3534ffc700 (LWP 14415)): #0 0x00007f354008eef3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f354154ce93 in ?? () from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 #2 0x00007f3541539f42 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 #3 0x0000000000423524 in ?? () #4 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f353d1ff700 (LWP 14381)): #0 0x00007f3541dba98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f35421d6099 in cb_cond_timedwait () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #2 0x00007f353d7f99d5 in ?? () from /opt/couchbase/lib/memcached/file_logger.so #3 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f3536dff700 (LWP 14412)): #0 0x00007f354008eef3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f354154ce93 in ?? () from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 #2 0x00007f3541539f42 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 #3 0x0000000000423524 in ?? () #4 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f353c7fe700 (LWP 14390)): #0 0x00007f3541dba98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f35421d6099 in cb_cond_timedwait () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #2 0x00007f354302e317 in ?? () from /opt/couchbase/bin/../lib/libauditd.so.0.1.0 #3 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f3533bfa700 (LWP 14417)): #0 0x00007f3541dba5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f354088a2fc in std::condition_variable::wait(std::unique_lock&) () from /usr/lib64/libstdc++.so.6 #2 0x0000000000437d40 in ?? () #3 0x00007f35421d9bab in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f353e3fd700 (LWP 14379)): #0 0x00007f35400850d3 in poll () from /lib64/libc.so.6 #1 0x00007f353e3feb48 in ?? () from /opt/couchbase/lib/memcached/stdin_term_handler.so #2 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #3 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f34659f6700 (LWP 14606)): #0 0x00007f3541dba98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f353b74c1db in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007f353b7502c0 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007f353b70c996 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007f353b70ca71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007f353b714628 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f34677f9700 (LWP 14603)): #0 0x00007f3541dbd264 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f3541db8508 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00007f3541db83d7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007f353b1df4a0 in ?? () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #4 0x00007f353b1eccc8 in ?? () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #5 0x00007f353b1eda87 in ?? () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #6 0x00007f353b209f25 in fdb_kvs_open () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #7 0x00007f353b77c7e1 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #8 0x00007f353b783f30 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #9 0x00007f353b6da676 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #10 0x00007f353b748fbc in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #11 0x00007f353b71447e in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #12 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #13 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #14 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f35363fe700 (LWP 14413)): #0 0x00007f354008eef3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f354154ce93 in ?? () from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 #2 0x00007f3541539f42 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 #3 0x0000000000423524 in ?? () #4 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f354008e8fd in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f34681fa700 (LWP 14602)): #0 0x00007f353b1c922e in ?? () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #1 0x00007f353b1fb976 in ?? () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #2 0x00007f353b1fc4c7 in ?? () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #3 0x00007f353b208629 in ?? () from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so #4 0x00007f353b783fb4 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007f353b6da676 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007f353b748fbc in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #7 0x00007f353b71447e in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #8 0x00007f35421d589a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #9 0x00007f3541db69d1 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f354008e8fd in clone () from /lib64/libc.so.6 -------------------------------------------------------------------------------- Module information: /opt/couchbase/bin/../lib/libauditd.so.0.1.0: -rwxr-xr-x 1 couchbase couchbase 100288 Jul 27 13:31 /opt/couchbase/bin/../lib/libauditd.so.0.1.0 c6d611b8d5598537d80b41d561c24a0c /opt/couchbase/bin/../lib/libauditd.so.0.1.0 /opt/couchbase/bin/../lib/libmcd_time.so.1.0.0: -rwxr-xr-x 1 couchbase couchbase 7576 Jul 27 13:31 /opt/couchbase/bin/../lib/libmcd_time.so.1.0.0 14b384c0cc30b7b976ef31ebdc64d73d /opt/couchbase/bin/../lib/libmcd_time.so.1.0.0 /opt/couchbase/bin/../lib/libdirutils.so.0.1.0: -rwxr-xr-x 1 couchbase couchbase 15336 Jul 27 13:31 /opt/couchbase/bin/../lib/libdirutils.so.0.1.0 8dbbb3347b9bd4d0be2c2a2fe1e7d665 /opt/couchbase/bin/../lib/libdirutils.so.0.1.0 /opt/couchbase/bin/../lib/libmcd_util.so.1.0.0: -rwxr-xr-x 1 couchbase couchbase 31976 Jul 27 13:31 /opt/couchbase/bin/../lib/libmcd_util.so.1.0.0 f4b149b4fbd238984f266ac0d8db96d8 /opt/couchbase/bin/../lib/libmcd_util.so.1.0.0 /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1: -rwxr-xr-x 1 couchbase couchbase 159048 Jul 27 13:31 /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1 7a0f72793886b1a346c4c0a4b20a4386 /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1 /opt/couchbase/bin/../lib/libcJSON.so.1.1.0: -rwxr-xr-x 1 couchbase couchbase 23568 Jul 27 13:31 /opt/couchbase/bin/../lib/libcJSON.so.1.1.0 e1292e41a0927bfec7ae167a6aed7315 /opt/couchbase/bin/../lib/libcJSON.so.1.1.0 /opt/couchbase/bin/../lib/libJSON_checker.so.1.0.0: -rwxr-xr-x 1 couchbase couchbase 18008 Jul 27 13:31 /opt/couchbase/bin/../lib/libJSON_checker.so.1.0.0 ddb2c9f34859ba0e30b33a0ca17d0d7a /opt/couchbase/bin/../lib/libJSON_checker.so.1.0.0 /opt/couchbase/bin/../lib/libplatform.so.0.1.0: -rwxr-xr-x 1 couchbase couchbase 57576 Jul 27 13:31 /opt/couchbase/bin/../lib/libplatform.so.0.1.0 b561f77d8e050ea1ac7aeef09c2ba26b /opt/couchbase/bin/../lib/libplatform.so.0.1.0 /lib64/libdl.so.2: lrwxrwxrwx 1 root root 13 Apr 9 2015 /lib64/libdl.so.2 -> libdl-2.12.so cfd1aaf70fdb0ee7920978ff7c50041a /lib64/libdl.so.2 /lib64/libpthread.so.0: lrwxrwxrwx 1 root root 18 Apr 9 2015 /lib64/libpthread.so.0 -> libpthread-2.12.so 5a26076e27a50a397c984b815e468d12 /lib64/libpthread.so.0 /lib64/librt.so.1: lrwxrwxrwx 1 root root 13 Apr 9 2015 /lib64/librt.so.1 -> librt-2.12.so 45e47f58a540e504b0f368d899337747 /lib64/librt.so.1 /opt/couchbase/bin/../lib/libsnappy.so.1: lrwxrwxrwx 1 couchbase couchbase 18 Aug 2 02:36 /opt/couchbase/bin/../lib/libsnappy.so.1 -> libsnappy.so.1.2.0 2811d7c53df31bee5a33faf3b43c46bb /opt/couchbase/bin/../lib/libsnappy.so.1 /opt/couchbase/bin/../lib/libjemalloc.so.2: -rwxr-xr-x 1 couchbase couchbase 294224 Jul 27 13:31 /opt/couchbase/bin/../lib/libjemalloc.so.2 c3a03adcc357db7c7a6a016a6779486d /opt/couchbase/bin/../lib/libjemalloc.so.2 /opt/couchbase/bin/../lib/libevent_core-2.0.so.5: lrwxrwxrwx 1 couchbase couchbase 27 Aug 2 02:36 /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 -> libevent_core-2.0.so.5.1.10 26a7a84bd00c1b22b64468241b29fa36 /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 /usr/lib64/libssl.so.10: lrwxrwxrwx. 1 root root 16 Jun 26 2014 /usr/lib64/libssl.so.10 -> libssl.so.1.0.1e 3907083410e4dfed61210fcdf8c27a46 /usr/lib64/libssl.so.10 /usr/lib64/libcrypto.so.10: lrwxrwxrwx. 1 root root 19 Jun 26 2014 /usr/lib64/libcrypto.so.10 -> libcrypto.so.1.0.1e a6f00c5b250e2e547dbbb88e2e956129 /usr/lib64/libcrypto.so.10 /opt/couchbase/bin/../lib/libnuma.so.1: -rwxr-xr-x 1 couchbase couchbase 36240 Jul 27 13:31 /opt/couchbase/bin/../lib/libnuma.so.1 72a895738e8bebe59824fd81744eb3ad /opt/couchbase/bin/../lib/libnuma.so.1 /opt/couchbase/bin/../lib/libevent_pthreads-2.0.so.5: lrwxrwxrwx 1 couchbase couchbase 31 Aug 2 02:36 /opt/couchbase/bin/../lib/libevent_pthreads-2.0.so.5 -> libevent_pthreads-2.0.so.5.1.10 fb15fb0d0d000f17437d51cad511d194 /opt/couchbase/bin/../lib/libevent_pthreads-2.0.so.5 /usr/lib64/libstdc++.so.6: lrwxrwxrwx. 1 root root 19 Jun 26 2014 /usr/lib64/libstdc++.so.6 -> libstdc++.so.6.0.13 5fca8c3a8f76f6eca0c47ca31fa5fff9 /usr/lib64/libstdc++.so.6 /lib64/libm.so.6: lrwxrwxrwx 1 root root 12 Apr 9 2015 /lib64/libm.so.6 -> libm-2.12.so fce607ccbb1b96b006c26c152c0bbc7a /lib64/libm.so.6 /lib64/libgcc_s.so.1: lrwxrwxrwx. 1 root root 28 Jun 26 2014 /lib64/libgcc_s.so.1 -> libgcc_s-4.4.7-20120601.so.1 c38fffa3e98af77c14817139e1619684 /lib64/libgcc_s.so.1 /lib64/libc.so.6: lrwxrwxrwx 1 root root 12 Apr 9 2015 /lib64/libc.so.6 -> libc-2.12.so 445e3a92661760d421f81fd79d93d131 /lib64/libc.so.6 /lib64/ld-linux-x86-64.so.2: lrwxrwxrwx 1 root root 10 Apr 9 2015 /lib64/ld-linux-x86-64.so.2 -> ld-2.12.so 96cf94fba6995c3335f0b5e77e4090a4 /lib64/ld-linux-x86-64.so.2 /lib64/libgssapi_krb5.so.2: lrwxrwxrwx. 1 root root 21 Jun 26 2014 /lib64/libgssapi_krb5.so.2 -> libgssapi_krb5.so.2.2 953833a08c92ecd9650aa9f74772160d /lib64/libgssapi_krb5.so.2 /lib64/libkrb5.so.3: lrwxrwxrwx. 1 root root 14 Jun 26 2014 /lib64/libkrb5.so.3 -> libkrb5.so.3.3 982b29d819fd19092459777108bf5660 /lib64/libkrb5.so.3 /lib64/libcom_err.so.2: lrwxrwxrwx. 1 root root 17 Jun 26 2014 /lib64/libcom_err.so.2 -> libcom_err.so.2.1 b32f935ae5131dff1c7222bf398de5d1 /lib64/libcom_err.so.2 /lib64/libk5crypto.so.3: lrwxrwxrwx. 1 root root 18 Jun 26 2014 /lib64/libk5crypto.so.3 -> libk5crypto.so.3.1 300d67a9ca292c96eafafd796a10c6f4 /lib64/libk5crypto.so.3 /lib64/libz.so.1: lrwxrwxrwx. 1 root root 13 Jun 26 2014 /lib64/libz.so.1 -> libz.so.1.2.3 e8ac93fdb47f4beb4560abee1ae5f0da /lib64/libz.so.1 /lib64/libkrb5support.so.0: lrwxrwxrwx. 1 root root 21 Jun 26 2014 /lib64/libkrb5support.so.0 -> libkrb5support.so.0.1 f7d26e6c613b2f4998b298b43b509615 /lib64/libkrb5support.so.0 /lib64/libkeyutils.so.1: lrwxrwxrwx. 1 root root 18 Jun 26 2014 /lib64/libkeyutils.so.1 -> libkeyutils.so.1.3 8002611a776e9de0ccf7832721b86d95 /lib64/libkeyutils.so.1 /lib64/libresolv.so.2: lrwxrwxrwx 1 root root 17 Apr 9 2015 /lib64/libresolv.so.2 -> libresolv-2.12.so b8b911c1080f53b219d1e0edc8a0ecd8 /lib64/libresolv.so.2 /lib64/libselinux.so.1: -rwxr-xr-x. 1 root root 122040 May 10 2013 /lib64/libselinux.so.1 0a438a255ce11f51605efe623823f887 /lib64/libselinux.so.1 /opt/couchbase/lib/memcached/stdin_term_handler.so: -rwxr-xr-x 1 couchbase couchbase 6792 Jul 27 13:31 /opt/couchbase/lib/memcached/stdin_term_handler.so 0ef04d11dd0be10e3cf896396b3c4089 /opt/couchbase/lib/memcached/stdin_term_handler.so /opt/couchbase/lib/memcached/file_logger.so: -rwxr-xr-x 1 couchbase couchbase 23864 Jul 27 13:31 /opt/couchbase/lib/memcached/file_logger.so b63fbba26e3274ee7a318b67c698a7dc /opt/couchbase/lib/memcached/file_logger.so /opt/couchbase/bin/../lib/../lib/nobucket.so: -rwxr-xr-x 1 couchbase couchbase 6936 Jul 27 13:31 /opt/couchbase/bin/../lib/../lib/nobucket.so d8e518658a6258f730c052786d785882 /opt/couchbase/bin/../lib/../lib/nobucket.so /opt/couchbase/bin/../lib/../lib/default_engine.so: -rwxr-xr-x 1 couchbase couchbase 52488 Jul 27 13:31 /opt/couchbase/bin/../lib/../lib/default_engine.so 9ae24f1241a3219ddabfb582e0570fc6 /opt/couchbase/bin/../lib/../lib/default_engine.so /opt/couchbase/bin/../lib/../lib/ep.so: -rwxr-xr-x 1 couchbase couchbase 1530768 Jul 27 13:31 /opt/couchbase/bin/../lib/../lib/ep.so 16d7edefdec91378e8dbef1880701f5a /opt/couchbase/bin/../lib/../lib/ep.so /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so: -rwxr-xr-x 1 couchbase couchbase 200088 Jul 27 13:31 /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so 9a6845d11fb8a5f47ed4237cf9f3b79e /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so: -rwxr-xr-x 1 couchbase couchbase 601432 Jul 27 13:31 /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so 7a7d59e347041faf63a2710666b317ba /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so /opt/couchbase/bin/../lib/../lib/../lib/libv8.so: -rwxr-xr-x 1 couchbase couchbase 12605744 Jul 27 13:31 /opt/couchbase/bin/../lib/../lib/../lib/libv8.so 161d875708645183ab5175578ddba439 /opt/couchbase/bin/../lib/../lib/../lib/libv8.so /opt/couchbase/bin/../lib/../lib/../lib/libicuuc.so.54: lrwxrwxrwx 1 couchbase couchbase 16 Aug 2 02:36 /opt/couchbase/bin/../lib/../lib/../lib/libicuuc.so.54 -> libicuuc.so.54.1 1c55b75f5e597d1e772a720487560f32 /opt/couchbase/bin/../lib/../lib/../lib/libicuuc.so.54 /opt/couchbase/bin/../lib/../lib/../lib/libicudata.so.54: lrwxrwxrwx 1 couchbase couchbase 18 Aug 2 02:36 /opt/couchbase/bin/../lib/../lib/../lib/libicudata.so.54 -> libicudata.so.54.1 ae88c315fe48157aae6976c981e624ca /opt/couchbase/bin/../lib/../lib/../lib/libicudata.so.54 /opt/couchbase/bin/../lib/../lib/../lib/libicui18n.so.54: lrwxrwxrwx 1 couchbase couchbase 18 Aug 2 02:36 /opt/couchbase/bin/../lib/../lib/../lib/libicui18n.so.54 -> libicui18n.so.54.1 decf5cc88d23b661414408b17353326e /opt/couchbase/bin/../lib/../lib/../lib/libicui18n.so.54