Basic crash dump analysis of /tmp//core.mc:writer_4.14475. Please send the file to support@couchbase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 couchbase couchbase 629912 Jun 23 03:30 /opt/couchbase/bin/memcached b3e234755a07da018bf8b54356ae4078 /opt/couchbase/bin/memcached -rw------- 1 couchbase couchbase 689786880 Jun 23 11:52 /tmp//core.mc:writer_4.14475 e467d9a6f5ebe26f1a54504606d7e588 /tmp//core.mc:writer_4.14475 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) Red Hat Enterprise Linux (7.2-83.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/couchbase/bin/memcached...(no debugging symbols found)...done. [New Thread 21255] [New Thread 21256] [New Thread 14507] [New Thread 21258] [New Thread 21259] [New Thread 21257] [New Thread 21260] [New Thread 14475] [New Thread 14498] [New Thread 14496] [New Thread 14508] [New Thread 14499] [New Thread 14510] [New Thread 14506] [New Thread 14512] [New Thread 14509] [New Thread 14514] [New Thread 14511] [New Thread 14513] [New Thread 21251] [New Thread 14950] [New Thread 21252] [New Thread 21531] [New Thread 21254] [New Thread 21253] Reading symbols from /opt/couchbase/bin/../lib/libauditd.so.0.1.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libauditd.so.0.1.0 Reading symbols from /opt/couchbase/bin/../lib/libmcd_time.so.1.0.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libmcd_time.so.1.0.0 Reading symbols from /opt/couchbase/bin/../lib/libdirutils.so.0.1.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libdirutils.so.0.1.0 Reading symbols from /opt/couchbase/bin/../lib/libmcd_util.so.1.0.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libmcd_util.so.1.0.0 Reading symbols from /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1 Reading symbols from /opt/couchbase/bin/../lib/libcJSON.so.1.1.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libcJSON.so.1.1.0 Reading symbols from /opt/couchbase/bin/../lib/libJSON_checker.so.1.0.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libJSON_checker.so.1.0.0 Reading symbols from /opt/couchbase/bin/../lib/libplatform.so.0.1.0...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libplatform.so.0.1.0 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1 Reading symbols from /opt/couchbase/bin/../lib/libjemalloc.so.2...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libjemalloc.so.2 Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libssl.so.10 Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libcrypto.so.10 Reading symbols from /opt/couchbase/bin/../lib/libnuma.so.1...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libnuma.so.1 Reading symbols from /opt/couchbase/bin/../lib/libevent_pthreads-2.0.so.5...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_pthreads-2.0.so.5 Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libgssapi_krb5.so.2 Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libk5crypto.so.3 Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so Reading symbols from /opt/couchbase/bin/../lib/../lib/nobucket.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/nobucket.so Reading symbols from /opt/couchbase/bin/../lib/../lib/default_engine.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/default_engine.so Reading symbols from /opt/couchbase/bin/../lib/../lib/ep.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/ep.so Reading symbols from /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so Reading symbols from /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so Reading symbols from /opt/couchbase/bin/../lib/../lib/../lib/libv8.so...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/../lib/libv8.so Reading symbols from /opt/couchbase/bin/../lib/../lib/../lib/libicuuc.so.54...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/../lib/libicuuc.so.54 Reading symbols from /opt/couchbase/bin/../lib/../lib/../lib/libicudata.so.54...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/../lib/libicudata.so.54 Reading symbols from /opt/couchbase/bin/../lib/../lib/../lib/libicui18n.so.54...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/../lib/../lib/libicui18n.so.54 Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. Program terminated with signal 6, Aborted. #0 0x00007fc480cb8625 in raise () from /lib64/libc.so.6 Thread 25 (Thread 0x7fc46fffc700 (LWP 21253)): #0 0x00007fc482a9fa0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc47c54fc6b in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007fc47c551d10 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007fc47c50ee96 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007fc47c50ef71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007fc47c5177c8 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007fc482eba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007fc482a9ba51 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc480d6e93d in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fc46f5fb700 (LWP 21254)): #0 0x00007fc482a9fa0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc47c54fc6b in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007fc47c551d10 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007fc47c50ee96 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007fc47c50ef71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007fc47c5177c8 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007fc482eba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007fc482a9ba51 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc480d6e93d in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fc4713fe700 (LWP 21531)): #0 0x00007fc480cbb038 in getenv () from /lib64/libc.so.6 #1 0x00007fc480d238b6 in tzset_internal () from /lib64/libc.so.6 #2 0x00007fc480d23c10 in tzset () from /lib64/libc.so.6 #3 0x00007fc480d22789 in timelocal () from /lib64/libc.so.6 #4 0x00007fc483b04e87 in ISOTime::generatetimestamp(std::array&, long, unsigned int) () from /opt/couchbase/bin/../lib/libmcd_time.so.1.0.0 #5 0x00007fc47e5fa083 in ?? () from /opt/couchbase/lib/memcached/file_logger.so #6 0x00007fc47c5279cf in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #7 0x00007fc47c5278f3 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #8 0x00007fc47c4ceb30 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #9 0x00007fc47c4cf05e in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #10 0x00007fc47c4c6440 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #11 0x00007fc47c4bec88 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #12 0x00007fc47c4f4d66 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #13 0x0000000000411f59 in _start () Thread 22 (Thread 0x7fc471dff700 (LWP 21252)): #0 0x00007fc482a9fa0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc47c54fc6b in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007fc47c551d10 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007fc47c50ee96 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007fc47c50ef71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007fc47c5177c8 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007fc482eba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007fc482a9ba51 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc480d6e93d in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fc4709fd700 (LWP 14950)): #0 0x00007fc482a9fa0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc47c528f50 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007fc482eba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #3 0x00007fc482a9ba51 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc480d6e93d in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fc46c3f6700 (LWP 21251)): #0 0x00007fc482a9fa0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc47c54fc6b in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007fc47c551d10 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007fc47c50ee96 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007fc47c50ef71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007fc47c5177c8 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007fc482eba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007fc482a9ba51 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc480d6e93d in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fc4735f8700 (LWP 14513)): #0 0x00007fc482a9f63c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc48156a2fc in std::condition_variable::wait(std::unique_lock&) () from /usr/lib64/libstdc++.so.6 #2 0x0000000000442a20 in ?? () #3 0x00007fc482ebebab in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x00007fc482eba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007fc482a9ba51 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc480d6e93d in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fc4749fa700 (LWP 14511)): #0 0x00007fc482a9f63c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc48156a2fc in std::condition_variable::wait(std::unique_lock&) () from /usr/lib64/libstdc++.so.6 #2 0x0000000000442a20 in ?? () #3 0x00007fc482ebebab in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x00007fc482eba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007fc482a9ba51 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc480d6e93d in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fc472bf7700 (LWP 14514)): #0 0x00007fc482a9f63c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc48156a2fc in std::condition_variable::wait(std::unique_lock&) () from /usr/lib64/libstdc++.so.6 #2 0x0000000000442a20 in ?? () #3 0x00007fc482ebebab in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x00007fc482eba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007fc482a9ba51 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc480d6e93d in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fc475dfc700 (LWP 14509)): #0 0x00007fc482aa22e4 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fc482a9d588 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00007fc482a9d457 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fc47c4bf8ac in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007fc47c4f3c8d in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007fc47c4f4f50 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x0000000000411fde in _start () Thread 15 (Thread 0x7fc473ff9700 (LWP 14512)): #0 0x00007fc482a9f63c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc48156a2fc in std::condition_variable::wait(std::unique_lock&) () from /usr/lib64/libstdc++.so.6 #2 0x0000000000442a20 in ?? () #3 0x00007fc482ebebab in Couchbase::Thread::thread_entry() () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x00007fc482eba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007fc482a9ba51 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc480d6e93d in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fc477bff700 (LWP 14506)): #0 0x00007fc482aa22e4 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fc482a9d588 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00007fc482a9d457 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fc47c4bf8ac in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007fc47c4f3c8d in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007fc47c4f4f50 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x0000000000411fde in _start () Thread 13 (Thread 0x7fc4753fb700 (LWP 14510)): #0 0x00007fc480d6ef33 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fc482231e93 in ?? () from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 #2 0x00007fc48221ef42 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 #3 0x000000000042c834 in ?? () #4 0x00007fc482eba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #5 0x00007fc482a9ba51 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fc480d6e93d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fc47d5fe700 (LWP 14499)): #0 0x00007fc482a9fa0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc482ebb099 in cb_cond_timedwait () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #2 0x00007fc483d13327 in ?? () from /opt/couchbase/bin/../lib/libauditd.so.0.1.0 #3 0x00007fc482eba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x00007fc482a9ba51 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fc480d6e93d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fc4767fd700 (LWP 14508)): #0 0x00007fc482aa22e4 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fc482a9d588 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00007fc482a9d457 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fc47c4bf8ac in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007fc47c4f3c8d in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007fc47c4f4f50 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x0000000000411fde in _start () Thread 10 (Thread 0x7fc47f1fd700 (LWP 14496)): #0 0x00007fc480d65113 in poll () from /lib64/libc.so.6 #1 0x00007fc47f1feb58 in ?? () from /opt/couchbase/lib/memcached/stdin_term_handler.so #2 0x00007fc482eba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #3 0x00007fc482a9ba51 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fc480d6e93d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fc47dfff700 (LWP 14498)): #0 0x00007fc482a9fa0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc482ebb099 in cb_cond_timedwait () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #2 0x00007fc47e5f99f5 in ?? () from /opt/couchbase/lib/memcached/file_logger.so #3 0x00007fc482eba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #4 0x00007fc482a9ba51 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fc480d6e93d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fc484129820 (LWP 14475)): #0 0x00007fc482a9f63c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc48156a2fc in std::condition_variable::wait(std::unique_lock&) () from /usr/lib64/libstdc++.so.6 #2 0x00007fc482ebeebb in Couchbase::Thread::waitForState(Couchbase::ThreadState const&) () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #3 0x0000000000414b90 in _start () Thread 7 (Thread 0x7fc46aff4700 (LWP 21260)): #0 0x00007fc482a9fa0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc47c54fc6b in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007fc47c551d10 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007fc47c50ee96 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007fc47c50ef71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007fc47c5177c8 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007fc482eba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007fc482a9ba51 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc480d6e93d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fc46d7f8700 (LWP 21257)): #0 0x0000000000467589 in google_breakpad::ExceptionHandler::GenerateDump(google_breakpad::ExceptionHandler::CrashContext*) () #1 0x0000000000467a55 in ?? () #2 0x0000000000467b97 in google_breakpad::ExceptionHandler::SignalHandler(int, siginfo*, void*) () #3 #4 0x00007fc480cb8625 in raise () from /lib64/libc.so.6 #5 0x00007fc480cb9e05 in abort () from /lib64/libc.so.6 #6 0x00007fc481572991 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6 #7 0x0000000000433d3c in ?? () #8 0x00007fc481570bd6 in ?? () from /usr/lib64/libstdc++.so.6 #9 0x00007fc481570c03 in std::terminate() () from /usr/lib64/libstdc++.so.6 #10 0x00007fc481570d22 in __cxa_throw () from /usr/lib64/libstdc++.so.6 #11 0x000000000042db99 in ?? () #12 0x00007fc47c4de3cd in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #13 0x00007fc47c51761e in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #14 0x00007fc482eba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #15 0x00007fc482a9ba51 in start_thread () from /lib64/libpthread.so.0 #16 0x00007fc480d6e93d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fc46b9f5700 (LWP 21259)): #0 0x00007fc482a9fa0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc47c54fc6b in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007fc47c551d10 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007fc47c50ee96 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007fc47c50ef71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007fc47c5177c8 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007fc482eba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007fc482a9ba51 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc480d6e93d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fc46cdf7700 (LWP 21258)): #0 0x00007fc482a9fa0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc47c54fc6b in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007fc47c551d10 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007fc47c50ee96 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007fc47c50ef71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007fc47c5177c8 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007fc482eba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007fc482a9ba51 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc480d6e93d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fc4771fe700 (LWP 14507)): #0 0x00007fc482aa22e4 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fc482a9d588 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00007fc482a9d457 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fc47c4bf8ac in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007fc47c4f3c8d in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007fc47c4f4f50 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x0000000000411fde in _start () Thread 2 (Thread 0x7fc46e1f9700 (LWP 21256)): #0 0x00007fc482a9fa0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fc47c54fc6b in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #2 0x00007fc47c551d10 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #3 0x00007fc47c50ee96 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #4 0x00007fc47c50ef71 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #5 0x00007fc47c5177c8 in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #6 0x00007fc482eba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #7 0x00007fc482a9ba51 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fc480d6e93d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fc46ebfa700 (LWP 21255)): #0 0x00007fc480cb8625 in raise () from /lib64/libc.so.6 #1 0x00007fc480cb9d8d in abort () from /lib64/libc.so.6 #2 0x00007fc481572a7d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6 #3 0x0000000000433d3c in ?? () #4 0x00007fc481570bd6 in ?? () from /usr/lib64/libstdc++.so.6 #5 0x00007fc481570c03 in std::terminate() () from /usr/lib64/libstdc++.so.6 #6 0x00007fc481570d22 in __cxa_throw () from /usr/lib64/libstdc++.so.6 #7 0x000000000042db99 in ?? () #8 0x00007fc47c4de3cd in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #9 0x00007fc47c51761e in ?? () from /opt/couchbase/bin/../lib/../lib/ep.so #10 0x00007fc482eba89a in ?? () from /opt/couchbase/bin/../lib/libplatform.so.0.1.0 #11 0x00007fc482a9ba51 in start_thread () from /lib64/libpthread.so.0 #12 0x00007fc480d6e93d in clone () from /lib64/libc.so.6 -------------------------------------------------------------------------------- Module information: /opt/couchbase/bin/../lib/libauditd.so.0.1.0: -rwxr-xr-x 1 couchbase couchbase 100320 Jun 23 03:30 /opt/couchbase/bin/../lib/libauditd.so.0.1.0 3965ebd4406d5c1d9ee0493a62f341cd /opt/couchbase/bin/../lib/libauditd.so.0.1.0 /opt/couchbase/bin/../lib/libmcd_time.so.1.0.0: -rwxr-xr-x 1 couchbase couchbase 7592 Jun 23 03:30 /opt/couchbase/bin/../lib/libmcd_time.so.1.0.0 31fac57ea758664996f24423d4a6ba0e /opt/couchbase/bin/../lib/libmcd_time.so.1.0.0 /opt/couchbase/bin/../lib/libdirutils.so.0.1.0: -rwxr-xr-x 1 couchbase couchbase 15336 Jun 23 03:30 /opt/couchbase/bin/../lib/libdirutils.so.0.1.0 cbfe508e2ee49c37d9a27dc0cd99dc81 /opt/couchbase/bin/../lib/libdirutils.so.0.1.0 /opt/couchbase/bin/../lib/libmcd_util.so.1.0.0: -rwxr-xr-x 1 couchbase couchbase 31976 Jun 23 03:30 /opt/couchbase/bin/../lib/libmcd_util.so.1.0.0 c04560778cdcdd58506f204c61eae87b /opt/couchbase/bin/../lib/libmcd_util.so.1.0.0 /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1: -rwxr-xr-x 1 couchbase couchbase 159048 Jun 23 03:30 /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1 5e44db18d0049d6321bee0e27b95836b /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1 /opt/couchbase/bin/../lib/libcJSON.so.1.1.0: -rwxr-xr-x 1 couchbase couchbase 23568 Jun 23 03:30 /opt/couchbase/bin/../lib/libcJSON.so.1.1.0 1b9d2c33fb5fc804efcd4d6225248156 /opt/couchbase/bin/../lib/libcJSON.so.1.1.0 /opt/couchbase/bin/../lib/libJSON_checker.so.1.0.0: -rwxr-xr-x 1 couchbase couchbase 18008 Jun 23 03:30 /opt/couchbase/bin/../lib/libJSON_checker.so.1.0.0 7e91cf38da8765960b82e4459f040d3a /opt/couchbase/bin/../lib/libJSON_checker.so.1.0.0 /opt/couchbase/bin/../lib/libplatform.so.0.1.0: -rwxr-xr-x 1 couchbase couchbase 57576 Jun 23 03:30 /opt/couchbase/bin/../lib/libplatform.so.0.1.0 bc389857fef787f52203f9008c46a24a /opt/couchbase/bin/../lib/libplatform.so.0.1.0 /lib64/libdl.so.2: lrwxrwxrwx 1 root root 13 Nov 3 2015 /lib64/libdl.so.2 -> libdl-2.12.so d3f4f8999e18ab3473d1e8d8aef2e32d /lib64/libdl.so.2 /lib64/libpthread.so.0: lrwxrwxrwx 1 root root 18 Nov 3 2015 /lib64/libpthread.so.0 -> libpthread-2.12.so 6c10973a3f46caf6cfd96513f8271e9f /lib64/libpthread.so.0 /lib64/librt.so.1: lrwxrwxrwx 1 root root 13 Nov 3 2015 /lib64/librt.so.1 -> librt-2.12.so 0d8c84c52b679b15ffd902b3d067b660 /lib64/librt.so.1 /opt/couchbase/bin/../lib/libsnappy.so.1: lrwxrwxrwx 1 couchbase couchbase 18 Jun 23 10:38 /opt/couchbase/bin/../lib/libsnappy.so.1 -> libsnappy.so.1.2.0 2811d7c53df31bee5a33faf3b43c46bb /opt/couchbase/bin/../lib/libsnappy.so.1 /opt/couchbase/bin/../lib/libjemalloc.so.2: -rwxr-xr-x 1 couchbase couchbase 294224 Jun 23 03:30 /opt/couchbase/bin/../lib/libjemalloc.so.2 c3a03adcc357db7c7a6a016a6779486d /opt/couchbase/bin/../lib/libjemalloc.so.2 /opt/couchbase/bin/../lib/libevent_core-2.0.so.5: lrwxrwxrwx 1 couchbase couchbase 27 Jun 23 10:38 /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 -> libevent_core-2.0.so.5.1.10 26a7a84bd00c1b22b64468241b29fa36 /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 /usr/lib64/libssl.so.10: lrwxrwxrwx 1 root root 16 Nov 3 2015 /usr/lib64/libssl.so.10 -> libssl.so.1.0.1e 91b03058064463699751d141b5cc10bd /usr/lib64/libssl.so.10 /usr/lib64/libcrypto.so.10: lrwxrwxrwx 1 root root 19 Nov 3 2015 /usr/lib64/libcrypto.so.10 -> libcrypto.so.1.0.1e aec46cfdb1c7c5ff522aada9f0702930 /usr/lib64/libcrypto.so.10 /opt/couchbase/bin/../lib/libnuma.so.1: -rwxr-xr-x 1 couchbase couchbase 36240 Jun 23 03:30 /opt/couchbase/bin/../lib/libnuma.so.1 72a895738e8bebe59824fd81744eb3ad /opt/couchbase/bin/../lib/libnuma.so.1 /opt/couchbase/bin/../lib/libevent_pthreads-2.0.so.5: lrwxrwxrwx 1 couchbase couchbase 31 Jun 23 10:38 /opt/couchbase/bin/../lib/libevent_pthreads-2.0.so.5 -> libevent_pthreads-2.0.so.5.1.10 fb15fb0d0d000f17437d51cad511d194 /opt/couchbase/bin/../lib/libevent_pthreads-2.0.so.5 /usr/lib64/libstdc++.so.6: lrwxrwxrwx 1 root root 19 Nov 3 2015 /usr/lib64/libstdc++.so.6 -> libstdc++.so.6.0.13 a4cff5ad897cf82e976103a1f424a903 /usr/lib64/libstdc++.so.6 /lib64/libm.so.6: lrwxrwxrwx 1 root root 12 Nov 3 2015 /lib64/libm.so.6 -> libm-2.12.so d1623a913ba62a62c89a79fa5fd352db /lib64/libm.so.6 /lib64/libgcc_s.so.1: lrwxrwxrwx 1 root root 28 Nov 3 2015 /lib64/libgcc_s.so.1 -> libgcc_s-4.4.7-20120601.so.1 e782c2ecc54b2836a202d711dce6c9bb /lib64/libgcc_s.so.1 /lib64/libc.so.6: lrwxrwxrwx 1 root root 12 Nov 3 2015 /lib64/libc.so.6 -> libc-2.12.so dc5ce313310cfeb8e6abf8a4737fb5b3 /lib64/libc.so.6 /lib64/ld-linux-x86-64.so.2: lrwxrwxrwx 1 root root 10 Nov 3 2015 /lib64/ld-linux-x86-64.so.2 -> ld-2.12.so ee6986e1eef26e966c6b188d2ee024ff /lib64/ld-linux-x86-64.so.2 /lib64/libgssapi_krb5.so.2: lrwxrwxrwx 1 root root 21 Nov 3 2015 /lib64/libgssapi_krb5.so.2 -> libgssapi_krb5.so.2.2 d8cc02c82b07a3d0d92f502d23b28584 /lib64/libgssapi_krb5.so.2 /lib64/libkrb5.so.3: lrwxrwxrwx 1 root root 14 Nov 3 2015 /lib64/libkrb5.so.3 -> libkrb5.so.3.3 002441d2643cc8a455d38bd6fb50ca6a /lib64/libkrb5.so.3 /lib64/libcom_err.so.2: lrwxrwxrwx 1 root root 17 Nov 3 2015 /lib64/libcom_err.so.2 -> libcom_err.so.2.1 95b5dabf5e3f0e927435a9bfd8f59e85 /lib64/libcom_err.so.2 /lib64/libk5crypto.so.3: lrwxrwxrwx 1 root root 18 Nov 3 2015 /lib64/libk5crypto.so.3 -> libk5crypto.so.3.1 e086075b3ee5c9926fed4b6cfebe5750 /lib64/libk5crypto.so.3 /lib64/libz.so.1: lrwxrwxrwx. 1 root root 13 Jun 26 2014 /lib64/libz.so.1 -> libz.so.1.2.3 e8ac93fdb47f4beb4560abee1ae5f0da /lib64/libz.so.1 /lib64/libkrb5support.so.0: lrwxrwxrwx 1 root root 21 Nov 3 2015 /lib64/libkrb5support.so.0 -> libkrb5support.so.0.1 602da122691bf60cf393300086f92e91 /lib64/libkrb5support.so.0 /lib64/libkeyutils.so.1: lrwxrwxrwx 1 root root 18 Nov 3 2015 /lib64/libkeyutils.so.1 -> libkeyutils.so.1.3 e88a46c919b77562d26f8517f1daf1d6 /lib64/libkeyutils.so.1 /lib64/libresolv.so.2: lrwxrwxrwx 1 root root 17 Nov 3 2015 /lib64/libresolv.so.2 -> libresolv-2.12.so 395d39575d390786417990a9e3c1d607 /lib64/libresolv.so.2 /lib64/libselinux.so.1: -rwxr-xr-x 1 root root 122040 Oct 15 2014 /lib64/libselinux.so.1 382d521a9e3a27d6c96d30d354f7ee0e /lib64/libselinux.so.1 /opt/couchbase/lib/memcached/stdin_term_handler.so: -rwxr-xr-x 1 couchbase couchbase 6792 Jun 23 03:30 /opt/couchbase/lib/memcached/stdin_term_handler.so 21deccdf1456a0982a716578fcc5454e /opt/couchbase/lib/memcached/stdin_term_handler.so /opt/couchbase/lib/memcached/file_logger.so: -rwxr-xr-x 1 couchbase couchbase 23864 Jun 23 03:30 /opt/couchbase/lib/memcached/file_logger.so a193c203591b106ab250877458032af3 /opt/couchbase/lib/memcached/file_logger.so /opt/couchbase/bin/../lib/../lib/nobucket.so: -rwxr-xr-x 1 couchbase couchbase 6960 Jun 23 03:30 /opt/couchbase/bin/../lib/../lib/nobucket.so 65ca7300c7715410453df45bd2831873 /opt/couchbase/bin/../lib/../lib/nobucket.so /opt/couchbase/bin/../lib/../lib/default_engine.so: -rwxr-xr-x 1 couchbase couchbase 52552 Jun 23 03:30 /opt/couchbase/bin/../lib/../lib/default_engine.so 91a6d5ac218f687d733843f1cf9d2343 /opt/couchbase/bin/../lib/../lib/default_engine.so /opt/couchbase/bin/../lib/../lib/ep.so: -rwxr-xr-x 1 couchbase couchbase 1531440 Jun 23 03:30 /opt/couchbase/bin/../lib/../lib/ep.so 1e5c19a4cca6c49542b96fccc8a101dc /opt/couchbase/bin/../lib/../lib/ep.so /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so: -rwxr-xr-x 1 couchbase couchbase 199816 Jun 23 03:30 /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so 90f402145e61875f85fa49d13bb9c6a4 /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so: -rwxr-xr-x 1 couchbase couchbase 514192 Jun 23 03:30 /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so 0fa7d500374d064817d6a1273eea7ffd /opt/couchbase/bin/../lib/../lib/../lib/libforestdb.so /opt/couchbase/bin/../lib/../lib/../lib/libv8.so: -rwxr-xr-x 1 couchbase couchbase 12605744 Jun 23 03:30 /opt/couchbase/bin/../lib/../lib/../lib/libv8.so 161d875708645183ab5175578ddba439 /opt/couchbase/bin/../lib/../lib/../lib/libv8.so /opt/couchbase/bin/../lib/../lib/../lib/libicuuc.so.54: lrwxrwxrwx 1 couchbase couchbase 16 Jun 23 10:38 /opt/couchbase/bin/../lib/../lib/../lib/libicuuc.so.54 -> libicuuc.so.54.1 1c55b75f5e597d1e772a720487560f32 /opt/couchbase/bin/../lib/../lib/../lib/libicuuc.so.54 /opt/couchbase/bin/../lib/../lib/../lib/libicudata.so.54: lrwxrwxrwx 1 couchbase couchbase 18 Jun 23 10:38 /opt/couchbase/bin/../lib/../lib/../lib/libicudata.so.54 -> libicudata.so.54.1 ae88c315fe48157aae6976c981e624ca /opt/couchbase/bin/../lib/../lib/../lib/libicudata.so.54 /opt/couchbase/bin/../lib/../lib/../lib/libicui18n.so.54: lrwxrwxrwx 1 couchbase couchbase 18 Jun 23 10:38 /opt/couchbase/bin/../lib/../lib/../lib/libicui18n.so.54 -> libicui18n.so.54.1 decf5cc88d23b661414408b17353326e /opt/couchbase/bin/../lib/../lib/../lib/libicui18n.so.54