Basic crash dump analysis of /tmp//core.mc:writer_5.10772. Please send the file to support@couchbase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 couchbase couchbase 629912 Jun 16 18:05 /opt/couchbase/bin/memcached 623cf7ef7f8118555e87058180d0db0a /opt/couchbase/bin/memcached -rw------- 1 couchbase couchbase 563941376 Jun 17 08:13 /tmp//core.mc:writer_5.10772 9f2cedc853f78c90c2a80845ceafd55f /tmp//core.mc:writer_5.10772 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/couchbase/bin/memcached...(no debugging symbols found)...done. [New Thread 11203] [New Thread 11206] [New Thread 11204] [New Thread 10772] [New Thread 10799] [New Thread 10798] [New Thread 11207] [New Thread 10791] [New Thread 11205] [New Thread 11456] [New Thread 10792] [New Thread 10793] [New Thread 10802] [New Thread 11202] [New Thread 10800] [New Thread 10805] [New Thread 10801] [New Thread 11201] [New Thread 11198] [New Thread 10803] [New Thread 11200] [New Thread 11199] [New Thread 11197] [New Thread 10804] [New Thread 10797] Reading symbols from /opt/couchbase/bin/../lib/libauditd.so.0.1.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libauditd.so.0.1.0 Reading symbols from /opt/couchbase/bin/../lib/libmcd_time.so.1.0.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libmcd_time.so.1.0.0 Reading symbols from /opt/couchbase/bin/../lib/libdirutils.so.0.1.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libdirutils.so.0.1.0 Reading symbols from /opt/couchbase/bin/../lib/libmcd_util.so.1.0.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libmcd_util.so.1.0.0 Reading symbols from /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1 Reading symbols from /opt/couchbase/bin/../lib/libcJSON.so.1.1.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libcJSON.so.1.1.0 Reading symbols from /opt/couchbase/bin/../lib/libJSON_checker.so.1.0.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libJSON_checker.so.1.0.0 Reading symbols from /opt/couchbase/bin/../lib/libplatform.so.0.1.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libplatform.so.0.1.0 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1 Reading symbols from /opt/couchbase/bin/../lib/libjemalloc.so.2...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libjemalloc.so.2 Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libssl.so.10 Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libcrypto.so.10 Reading symbols from /opt/couchbase/bin/../lib/libnuma.so.1...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libnuma.so.1 Reading symbols from /opt/couchbase/bin/../lib/libevent_pthreads-2.0.so.5...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_pthreads-2.0.so.5 Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libgssapi_krb5.so.2 Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libk5crypto.so.3 Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so Reading symbols from /opt/couchbase/bin/../lib/../lib/nobucket.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/nobucket.so Reading symbols from /opt/couchbase/bin/../lib/../lib/default_engine.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/default_engine.so Reading symbols from /opt/couchbase/bin/../lib/../lib/ep.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/ep.so Reading symbols from /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so Reading symbols from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so Reading symbols from /opt/couchbase/bin/../lib/../lib/../lib/libv8.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/../lib/libv8.so Reading symbols from /opt/couchbase/bin/../lib/../lib/../lib/libicuuc.so.54...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/../lib/libicuuc.so.54 Reading symbols from /opt/couchbase/bin/../lib/../lib/../lib/libicudata.so.54...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/../lib/libicudata.so.54 Reading symbols from /opt/couchbase/bin/../lib/../lib/../lib/libicui18n.so.54...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/../lib/libicui18n.so.54 Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. Program terminated with signal 11, Segmentation fault. #0 0x000000000042d89c in ?? () Thread 25 (Thread 0x7faa52fff700 (LWP 10797)): #0 0x00007faa5dfa2264 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007faa5df9d508 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00007faa5df9d3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007faa578bf8ac in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007faa578f3c8d in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007faa578f4f50 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x0000000000411fde in _start () Thread 24 (Thread 0x7faa4e9f8700 (LWP 10804)): #0 0x00007faa5df9f5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007faa5ca6f2fc in std::condition_variable::wait(std::unique_lock&) () from /usr/lib64/libstdc++.so.6 #2 0x0000000000442a20 in ?? () #3 0x00007faa5e3bebab in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7faa4bdfd700 (LWP 11197)): #0 0x00007faa5df9f98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007faa57928f50 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #3 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7faa4a9fb700 (LWP 11199)): #0 0x00007faa5df9f98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007faa5794fc6b in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007faa57951d10 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007faa5790ee96 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007faa5790ef71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007faa579177c8 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #8 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7faa49ffa700 (LWP 11200)): #0 0x00007faa5df9f98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007faa5794fc6b in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007faa57951d10 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007faa5790ee96 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007faa5790ef71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007faa579177c8 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #8 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7faa4f3f9700 (LWP 10803)): #0 0x00007faa5df9f5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007faa5ca6f2fc in std::condition_variable::wait(std::unique_lock&) () from /usr/lib64/libstdc++.so.6 #2 0x0000000000442a20 in ?? () #3 0x00007faa5e3bebab in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7faa4b3fc700 (LWP 11198)): #0 0x00007faa5df9f98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007faa5794fc6b in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007faa57951d10 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007faa5790ee96 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007faa5790ef71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007faa579177c8 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #8 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7faa495f9700 (LWP 11201)): #0 0x00007faa5df9f98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007faa5794fc6b in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007faa57951d10 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007faa5790ee96 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007faa5790ef71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007faa579177c8 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #8 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7faa507fb700 (LWP 10801)): #0 0x00007faa5c274163 in epoll_wait () from /lib64/libc.so.6 #1 0x00007faa5d731e93 in ?? () from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 #2 0x00007faa5d71ef42 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 #3 0x000000000042c834 in ?? () #4 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7faa4dff7700 (LWP 10805)): #0 0x00007faa5df9f5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007faa5ca6f2fc in std::condition_variable::wait(std::unique_lock&) () from /usr/lib64/libstdc++.so.6 #2 0x0000000000442a20 in ?? () #3 0x00007faa5e3bebab in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7faa511fc700 (LWP 10800)): #0 0x00007faa5c274163 in epoll_wait () from /lib64/libc.so.6 #1 0x00007faa5d731e93 in ?? () from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 #2 0x00007faa5d71ef42 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 #3 0x000000000042c834 in ?? () #4 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7faa48bf8700 (LWP 11202)): #0 0x00007faa5df9f98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007faa5794fc6b in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007faa57951d10 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007faa5790ee96 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007faa5790ef71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007faa579177c8 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #8 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7faa4fdfa700 (LWP 10802)): #0 0x00007faa5df9f5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007faa5ca6f2fc in std::condition_variable::wait(std::unique_lock&) () from /usr/lib64/libstdc++.so.6 #2 0x0000000000442a20 in ?? () #3 0x00007faa5e3bebab in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7faa589fe700 (LWP 10793)): #0 0x00007faa5df9f98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007faa5e3bb099 in cb_cond_timedwait () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #2 0x00007faa5f213327 in ?? () from /opt/couchbase/bin/../lib/libauditd.so.0.1.0 #3 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7faa593ff700 (LWP 10792)): #0 0x00007faa5df9f98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007faa5e3bb099 in cb_cond_timedwait () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #2 0x00007faa599f99f5 in ?? () from /opt/couchbase/lib/memcached/file_logger.so #3 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7faa4d1ff700 (LWP 11456)): #0 0x00007faa5dfa2264 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007faa5df9d508 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00007faa5df9d3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007faa5e3baafd in cb_mutex_enter () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x000000000042d7b9 in ?? () #5 0x00000000004143ad in _start () Thread 9 (Thread 0x7faa46df5700 (LWP 11205)): #0 0x00007faa5df9f98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007faa5794fc6b in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007faa57951d10 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007faa5790ee96 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007faa5790ef71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007faa579177c8 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #8 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7faa5a5fd700 (LWP 10791)): #0 0x00007faa5c26a343 in poll () from /lib64/libc.so.6 #1 0x00007faa5a5feb58 in ?? () from /opt/couchbase/lib/memcached/stdin_term_handler.so #2 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #3 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7faa459f3700 (LWP 11207)): #0 0x00007faa5df9f98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007faa5794fc6b in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007faa57951d10 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007faa5790ee96 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007faa5790ef71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007faa579177c8 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #8 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7faa525fe700 (LWP 10798)): #0 0x00007faa578a1984 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #1 0x00007faa578a1a30 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007faa578cfce2 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007faa578ceaab in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007faa578cf05e in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007faa578c357f in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007faa578bf66b in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #7 0x00007faa578bf8c0 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #8 0x00007faa578f3c8d in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #9 0x00007faa578f4f50 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #10 0x0000000000411fde in _start () Thread 5 (Thread 0x7faa51bfd700 (LWP 10799)): #0 0x00007faa5c274163 in epoll_wait () from /lib64/libc.so.6 #1 0x00007faa5d731e93 in ?? () from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 #2 0x00007faa5d71ef42 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 #3 0x000000000042c834 in ?? () #4 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7faa5f629820 (LWP 10772)): #0 0x00007faa5df9f5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007faa5ca6f2fc in std::condition_variable::wait(std::unique_lock&) () from /usr/lib64/libstdc++.so.6 #2 0x00007faa5e3beebb in Couchbase::Thread::waitForState(Couchbase::ThreadState const&) () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #3 0x0000000000414b90 in _start () Thread 3 (Thread 0x7faa477f6700 (LWP 11204)): #0 0x00007faa5dfa2264 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007faa5df9d508 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00007faa5df9d3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007faa5e3baafd in cb_mutex_enter () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x000000000042da32 in ?? () #5 0x00007faa578de3cd in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007faa5791761e in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #7 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #8 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #9 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7faa463f4700 (LWP 11206)): #0 0x00007faa5df9f98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007faa5794fc6b in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007faa57951d10 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007faa5790ee96 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007faa5790ef71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007faa579177c8 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #8 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7faa481f7700 (LWP 11203)): #0 0x000000000042d89c in ?? () #1 0x000000000042da55 in ?? () #2 0x00007faa578de3cd in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007faa5791761e in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007faa5e3ba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007faa5df9b9d1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007faa5c273b6d in clone () from /lib64/libc.so.6 -------------------------------------------------------------------------------- Module information: /opt/couchbase/bin/../lib/libauditd.so.0.1.0: -rwxr-xr-x 1 couchbase couchbase 100320 Jun 16 18:05 /opt/couchbase/bin/../lib/libauditd.so.0.1.0 1e6d627980d00c805a91a131c753e16a /opt/couchbase/bin/../lib/libauditd.so.0.1.0 /opt/couchbase/bin/../lib/libmcd_time.so.1.0.0: -rwxr-xr-x 1 couchbase couchbase 7592 Jun 16 18:05 /opt/couchbase/bin/../lib/libmcd_time.so.1.0.0 31fac57ea758664996f24423d4a6ba0e /opt/couchbase/bin/../lib/libmcd_time.so.1.0.0 /opt/couchbase/bin/../lib/libdirutils.so.0.1.0: -rwxr-xr-x 1 couchbase couchbase 15336 Jun 16 18:05 /opt/couchbase/bin/../lib/libdirutils.so.0.1.0 cbfe508e2ee49c37d9a27dc0cd99dc81 /opt/couchbase/bin/../lib/libdirutils.so.0.1.0 /opt/couchbase/bin/../lib/libmcd_util.so.1.0.0: -rwxr-xr-x 1 couchbase couchbase 31976 Jun 16 18:05 /opt/couchbase/bin/../lib/libmcd_util.so.1.0.0 c04560778cdcdd58506f204c61eae87b /opt/couchbase/bin/../lib/libmcd_util.so.1.0.0 /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1: -rwxr-xr-x 1 couchbase couchbase 159048 Jun 16 18:05 /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1 5e44db18d0049d6321bee0e27b95836b /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1 /opt/couchbase/bin/../lib/libcJSON.so.1.1.0: -rwxr-xr-x 1 couchbase couchbase 23568 Jun 16 18:05 /opt/couchbase/bin/../lib/libcJSON.so.1.1.0 1b9d2c33fb5fc804efcd4d6225248156 /opt/couchbase/bin/../lib/libcJSON.so.1.1.0 /opt/couchbase/bin/../lib/libJSON_checker.so.1.0.0: -rwxr-xr-x 1 couchbase couchbase 18008 Jun 16 18:05 /opt/couchbase/bin/../lib/libJSON_checker.so.1.0.0 7e91cf38da8765960b82e4459f040d3a /opt/couchbase/bin/../lib/libJSON_checker.so.1.0.0 /opt/couchbase/bin/../lib/libplatform.so.0.1.0: -rwxr-xr-x 1 couchbase couchbase 57576 Jun 16 18:05 /opt/couchbase/bin/../lib/libplatform.so.0.1.0 bc389857fef787f52203f9008c46a24a /opt/couchbase/bin/../lib/libplatform.so.0.1.0 /lib64/libdl.so.2: lrwxrwxrwx. 1 root root 13 Jun 26 2014 /lib64/libdl.so.2 -> libdl-2.12.so edb686710e606ba7f63ebf923fddc1de /lib64/libdl.so.2 /lib64/libpthread.so.0: lrwxrwxrwx. 1 root root 18 Jun 26 2014 /lib64/libpthread.so.0 -> libpthread-2.12.so b3e172fde149a8bbc0f8f8f219665cf7 /lib64/libpthread.so.0 /lib64/librt.so.1: lrwxrwxrwx. 1 root root 13 Jun 26 2014 /lib64/librt.so.1 -> librt-2.12.so 0fa1bf77fabba484c44107085db28216 /lib64/librt.so.1 /opt/couchbase/bin/../lib/libsnappy.so.1: lrwxrwxrwx 1 couchbase couchbase 18 Jun 17 06:51 /opt/couchbase/bin/../lib/libsnappy.so.1 -> libsnappy.so.1.2.0 2811d7c53df31bee5a33faf3b43c46bb /opt/couchbase/bin/../lib/libsnappy.so.1 /opt/couchbase/bin/../lib/libjemalloc.so.2: -rwxr-xr-x 1 couchbase couchbase 294224 Jun 16 18:05 /opt/couchbase/bin/../lib/libjemalloc.so.2 c3a03adcc357db7c7a6a016a6779486d /opt/couchbase/bin/../lib/libjemalloc.so.2 /opt/couchbase/bin/../lib/libevent_core-2.0.so.5: lrwxrwxrwx 1 couchbase couchbase 27 Jun 17 06:50 /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 -> libevent_core-2.0.so.5.1.10 26a7a84bd00c1b22b64468241b29fa36 /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 /usr/lib64/libssl.so.10: lrwxrwxrwx. 1 root root 16 Jun 26 2014 /usr/lib64/libssl.so.10 -> libssl.so.1.0.1e 3907083410e4dfed61210fcdf8c27a46 /usr/lib64/libssl.so.10 /usr/lib64/libcrypto.so.10: lrwxrwxrwx. 1 root root 19 Jun 26 2014 /usr/lib64/libcrypto.so.10 -> libcrypto.so.1.0.1e a6f00c5b250e2e547dbbb88e2e956129 /usr/lib64/libcrypto.so.10 /opt/couchbase/bin/../lib/libnuma.so.1: -rwxr-xr-x 1 couchbase couchbase 36240 Jun 16 18:05 /opt/couchbase/bin/../lib/libnuma.so.1 72a895738e8bebe59824fd81744eb3ad /opt/couchbase/bin/../lib/libnuma.so.1 /opt/couchbase/bin/../lib/libevent_pthreads-2.0.so.5: lrwxrwxrwx 1 couchbase couchbase 31 Jun 17 06:50 /opt/couchbase/bin/../lib/libevent_pthreads-2.0.so.5 -> libevent_pthreads-2.0.so.5.1.10 fb15fb0d0d000f17437d51cad511d194 /opt/couchbase/bin/../lib/libevent_pthreads-2.0.so.5 /usr/lib64/libstdc++.so.6: lrwxrwxrwx. 1 root root 19 Jun 26 2014 /usr/lib64/libstdc++.so.6 -> libstdc++.so.6.0.13 5fca8c3a8f76f6eca0c47ca31fa5fff9 /usr/lib64/libstdc++.so.6 /lib64/libm.so.6: lrwxrwxrwx. 1 root root 12 Jun 26 2014 /lib64/libm.so.6 -> libm-2.12.so de92a26a843b8f246860788d2d734479 /lib64/libm.so.6 /lib64/libgcc_s.so.1: lrwxrwxrwx. 1 root root 28 Jun 26 2014 /lib64/libgcc_s.so.1 -> libgcc_s-4.4.7-20120601.so.1 c38fffa3e98af77c14817139e1619684 /lib64/libgcc_s.so.1 /lib64/libc.so.6: lrwxrwxrwx. 1 root root 12 Jun 26 2014 /lib64/libc.so.6 -> libc-2.12.so c0090df6f1bb5511ac48b2e7e4ff3582 /lib64/libc.so.6 /lib64/ld-linux-x86-64.so.2: lrwxrwxrwx. 1 root root 10 Jun 26 2014 /lib64/ld-linux-x86-64.so.2 -> ld-2.12.so bebf225d79eebddf3f021fde519190c7 /lib64/ld-linux-x86-64.so.2 /lib64/libgssapi_krb5.so.2: lrwxrwxrwx. 1 root root 21 Jun 26 2014 /lib64/libgssapi_krb5.so.2 -> libgssapi_krb5.so.2.2 953833a08c92ecd9650aa9f74772160d /lib64/libgssapi_krb5.so.2 /lib64/libkrb5.so.3: lrwxrwxrwx. 1 root root 14 Jun 26 2014 /lib64/libkrb5.so.3 -> libkrb5.so.3.3 982b29d819fd19092459777108bf5660 /lib64/libkrb5.so.3 /lib64/libcom_err.so.2: lrwxrwxrwx. 1 root root 17 Jun 26 2014 /lib64/libcom_err.so.2 -> libcom_err.so.2.1 b32f935ae5131dff1c7222bf398de5d1 /lib64/libcom_err.so.2 /lib64/libk5crypto.so.3: lrwxrwxrwx. 1 root root 18 Jun 26 2014 /lib64/libk5crypto.so.3 -> libk5crypto.so.3.1 300d67a9ca292c96eafafd796a10c6f4 /lib64/libk5crypto.so.3 /lib64/libz.so.1: lrwxrwxrwx. 1 root root 13 Jun 26 2014 /lib64/libz.so.1 -> libz.so.1.2.3 e8ac93fdb47f4beb4560abee1ae5f0da /lib64/libz.so.1 /lib64/libkrb5support.so.0: lrwxrwxrwx. 1 root root 21 Jun 26 2014 /lib64/libkrb5support.so.0 -> libkrb5support.so.0.1 f7d26e6c613b2f4998b298b43b509615 /lib64/libkrb5support.so.0 /lib64/libkeyutils.so.1: lrwxrwxrwx. 1 root root 18 Jun 26 2014 /lib64/libkeyutils.so.1 -> libkeyutils.so.1.3 8002611a776e9de0ccf7832721b86d95 /lib64/libkeyutils.so.1 /lib64/libresolv.so.2: lrwxrwxrwx. 1 root root 17 Jun 26 2014 /lib64/libresolv.so.2 -> libresolv-2.12.so 48628ee5ee5efc24d872003a51ecef2a /lib64/libresolv.so.2 /lib64/libselinux.so.1: -rwxr-xr-x. 1 root root 122040 May 10 2013 /lib64/libselinux.so.1 0a438a255ce11f51605efe623823f887 /lib64/libselinux.so.1 /opt/couchbase/lib/memcached/stdin_term_handler.so: -rwxr-xr-x 1 couchbase couchbase 6792 Jun 16 18:05 /opt/couchbase/lib/memcached/stdin_term_handler.so 21deccdf1456a0982a716578fcc5454e /opt/couchbase/lib/memcached/stdin_term_handler.so /opt/couchbase/lib/memcached/file_logger.so: -rwxr-xr-x 1 couchbase couchbase 23864 Jun 16 18:05 /opt/couchbase/lib/memcached/file_logger.so a193c203591b106ab250877458032af3 /opt/couchbase/lib/memcached/file_logger.so /opt/couchbase/bin/../lib/../lib/nobucket.so: -rwxr-xr-x 1 couchbase couchbase 6960 Jun 16 18:05 /opt/couchbase/bin/../lib/../lib/nobucket.so 65ca7300c7715410453df45bd2831873 /opt/couchbase/bin/../lib/../lib/nobucket.so /opt/couchbase/bin/../lib/../lib/default_engine.so: -rwxr-xr-x 1 couchbase couchbase 52552 Jun 16 18:05 /opt/couchbase/bin/../lib/../lib/default_engine.so 78e8745ea49886fc5bfe057802f06c77 /opt/couchbase/bin/../lib/../lib/default_engine.so /opt/couchbase/bin/../lib/../lib/ep.so: -rwxr-xr-x 1 couchbase couchbase 1532624 Jun 16 18:05 /opt/couchbase/bin/../lib/../lib/ep.so 136ba8b11751b0b3df18bcbbec4d6f7c /opt/couchbase/bin/../lib/../lib/ep.so /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so: -rwxr-xr-x 1 couchbase couchbase 199816 Jun 16 18:05 /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so 90f402145e61875f85fa49d13bb9c6a4 /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so: -rwxr-xr-x 1 couchbase couchbase 523168 Jun 16 18:05 /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so 949ee4a732b839f28ab634bd536f007a /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so /opt/couchbase/bin/../lib/../lib/../lib/libv8.so: -rwxr-xr-x 1 couchbase couchbase 12605744 Jun 16 18:05 /opt/couchbase/bin/../lib/../lib/../lib/libv8.so 161d875708645183ab5175578ddba439 /opt/couchbase/bin/../lib/../lib/../lib/libv8.so /opt/couchbase/bin/../lib/../lib/../lib/libicuuc.so.54: lrwxrwxrwx 1 couchbase couchbase 16 Jun 17 06:51 /opt/couchbase/bin/../lib/../lib/../lib/libicuuc.so.54 -> libicuuc.so.54.1 1c55b75f5e597d1e772a720487560f32 /opt/couchbase/bin/../lib/../lib/../lib/libicuuc.so.54 /opt/couchbase/bin/../lib/../lib/../lib/libicudata.so.54: lrwxrwxrwx 1 couchbase couchbase 18 Jun 17 06:50 /opt/couchbase/bin/../lib/../lib/../lib/libicudata.so.54 -> libicudata.so.54.1 ae88c315fe48157aae6976c981e624ca /opt/couchbase/bin/../lib/../lib/../lib/libicudata.so.54 /opt/couchbase/bin/../lib/../lib/../lib/libicui18n.so.54: lrwxrwxrwx 1 couchbase couchbase 18 Jun 17 06:51 /opt/couchbase/bin/../lib/../lib/../lib/libicui18n.so.54 -> libicui18n.so.54.1 decf5cc88d23b661414408b17353326e /opt/couchbase/bin/../lib/../lib/../lib/libicui18n.so.54