Basic crash dump analysis of /opt/couchbase/var/lib/couchbase/core. Please send the file to support@couchbase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 couchbase couchbase 396209 2011-12-29 08:18 /opt/couchbase/bin/memcached -rw------- 1 couchbase couchbase 44151832576 2012-02-02 14:32 /opt/couchbase/var/lib/couchbase/core 2a09593ea5d9e2bd02827468339cde89 /opt/couchbase/bin/memcached c72f895b67238f64595dfd28e0624a6a /opt/couchbase/var/lib/couchbase/core -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) 7.1-ubuntu Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/couchbase/bin/memcached...done. [New Thread 1712] [New Thread 1714] [New Thread 1762] [New Thread 1760] [New Thread 10224] [New Thread 10225] [New Thread 1710] [New Thread 1711] [New Thread 1709] [New Thread 10226] [New Thread 1761] [New Thread 1759] [New Thread 10223] [New Thread 1713] [New Thread 1699] [New Thread 10221] [New Thread 1708] warning: Can't read pathname for load map: Input/output error. Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/librt.so.1 Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.0...done. Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.0 Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libpthread.so.0 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libgcc_s.so.1 Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so Reading symbols from /opt/couchbase/lib/memcached/ep.so...done. Loaded symbols for /opt/couchbase/lib/memcached/ep.so Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'. Program terminated with signal 6, Aborted. #0 0x00007f2dafc7fa75 in raise () from /lib/libc.so.6 Thread 17 (Thread 1708): #0 0x00007f2dafd244bd in read () from /lib/libc.so.6 #1 0x00007f2dafcbf348 in _IO_file_underflow () from /lib/libc.so.6 #2 0x00007f2dafcc0eee in _IO_default_uflow () from /lib/libc.so.6 #3 0x00007f2dafcb7c7b in getc () from /lib/libc.so.6 #4 0x00007f2daf51f9c9 in check_stdin_thread (arg=0x4039f0) at extensions/daemon/stdin_check.c:19 #5 0x00007f2daffd59ca in start_thread () from /lib/libpthread.so.0 #6 0x00007f2dafd3270d in clone () from /lib/libc.so.6 #7 0x0000000000000000 in ?? () Thread 16 (Thread 10221): #0 0x00007f2daffdabc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00007f2dadf15d18 in engine_shutdown_thread (arg=0x10090c0) at bucket_engine.c:1432 #2 0x00007f2daffd59ca in start_thread () from /lib/libpthread.so.0 #3 0x00007f2dafd3270d in clone () from /lib/libc.so.6 #4 0x0000000000000000 in ?? () Thread 15 (Thread 1699): #0 0x00007f2dafd32d03 in epoll_wait () from /lib/libc.so.6 #1 0x00007f2db0aeb5a6 in epoll_dispatch (base=0xf3d000, tv=) at epoll.c:404 #2 0x00007f2db0ada171 in event_base_loop (base=0xf3d000, flags=) at event.c:1558 #3 0x000000000040906c in main (argc=, argv=) at daemon/memcached.c:7459 Thread 14 (Thread 1713): #0 0x00007f2dafd32d03 in epoll_wait () from /lib/libc.so.6 #1 0x00007f2db0aeb5a6 in epoll_dispatch (base=0xf3da00, tv=) at epoll.c:404 #2 0x00007f2db0ada171 in event_base_loop (base=0xf3da00, flags=) at event.c:1558 #3 0x0000000000414584 in worker_libevent (arg=0xf1f800) at daemon/thread.c:304 #4 0x00007f2daffd59ca in start_thread () from /lib/libpthread.so.0 #5 0x00007f2dafd3270d in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 13 (Thread 10223): #0 0x00007f2daffda85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00007f2daa9c53bf in SyncObject::wait (this=0x22c40fa80) at syncobject.hh:31 #2 Dispatcher::run (this=0x22c40fa80) at dispatcher.cc:85 #3 0x00007f2daa9c5dc3 in launch_dispatcher_thread (arg=0x22c40fad4) at dispatcher.cc:28 #4 0x00007f2daffd59ca in start_thread () from /lib/libpthread.so.0 #5 0x00007f2dafd3270d in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 12 (Thread 1759): #0 0x00007f2daffda85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00007f2daa9c53bf in SyncObject::wait (this=0x2110380) at syncobject.hh:31 #2 Dispatcher::run (this=0x2110380) at dispatcher.cc:85 #3 0x00007f2daa9c5dc3 in launch_dispatcher_thread (arg=0x21103d4) at dispatcher.cc:28 #4 0x00007f2daffd59ca in start_thread () from /lib/libpthread.so.0 #5 0x00007f2dafd3270d in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 11 (Thread 1761): #0 0x00007f2daffdabc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00007f2daa9c2dea in SyncObject::wait (this=0xf21b00, d=...) at syncobject.hh:42 #2 IdleTask::run (this=0xf21b00, d=...) at dispatcher.cc:244 #3 0x00007f2daa9c516f in Dispatcher::run (this=0x21101c0) at dispatcher.cc:119 #4 0x00007f2daa9c5dc3 in launch_dispatcher_thread (arg=0x2110214) at dispatcher.cc:28 #5 0x00007f2daffd59ca in start_thread () from /lib/libpthread.so.0 #6 0x00007f2dafd3270d in clone () from /lib/libc.so.6 #7 0x0000000000000000 in ?? () Thread 10 (Thread 10226): #0 0x00007f2daffdabc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00007f2daa9e1f62 in SyncObject::wait (this=0x695d1000) at syncobject.hh:42 #2 SyncObject::wait (this=0x695d1000) at syncobject.hh:58 #3 TapConnMap::wait (this=0x695d1000) at tapconnmap.hh:207 #4 EventuallyPersistentEngine::notifyPendingConnections (this=0x695d1000) at ep_engine.cc:3652 #5 0x00007f2daa9e2051 in EvpNotifyPendingConns (arg=0x695d1000) at ep_engine.cc:967 #6 0x00007f2daffd59ca in start_thread () from /lib/libpthread.so.0 #7 0x00007f2dafd3270d in clone () from /lib/libc.so.6 #8 0x0000000000000000 in ?? () Thread 9 (Thread 1709): #0 0x00007f2dafcf639d in nanosleep () from /lib/libc.so.6 #1 0x00007f2dafcf6210 in sleep () from /lib/libc.so.6 #2 0x0000000000416308 in check_isasl_db_thread (arg=) at daemon/isasl.c:233 #3 0x00007f2daffd59ca in start_thread () from /lib/libpthread.so.0 #4 0x00007f2dafd3270d in clone () from /lib/libc.so.6 #5 0x0000000000000000 in ?? () Thread 8 (Thread 1711): #0 0x00007f2dafd32d03 in epoll_wait () from /lib/libc.so.6 #1 0x00007f2db0aeb5a6 in epoll_dispatch (base=0xf3d280, tv=) at epoll.c:404 #2 0x00007f2db0ada171 in event_base_loop (base=0xf3d280, flags=) at event.c:1558 #3 0x0000000000414584 in worker_libevent (arg=0xf1f600) at daemon/thread.c:304 #4 0x00007f2daffd59ca in start_thread () from /lib/libpthread.so.0 #5 0x00007f2dafd3270d in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 7 (Thread 1710): #0 0x00007f2dafd32d03 in epoll_wait () from /lib/libc.so.6 #1 0x00007f2db0aeb5a6 in epoll_dispatch (base=0xf3d500, tv=) at epoll.c:404 #2 0x00007f2db0ada171 in event_base_loop (base=0xf3d500, flags=) at event.c:1558 #3 0x0000000000414584 in worker_libevent (arg=0xf1f500) at daemon/thread.c:304 #4 0x00007f2daffd59ca in start_thread () from /lib/libpthread.so.0 #5 0x00007f2dafd3270d in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 6 (Thread 10225): #0 0x00007f2daffdabc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00007f2daa9c2dea in SyncObject::wait (this=0x4365800, d=...) at syncobject.hh:42 #2 IdleTask::run (this=0x4365800, d=...) at dispatcher.cc:244 #3 0x00007f2daa9c516f in Dispatcher::run (this=0x22c40f8c0) at dispatcher.cc:119 #4 0x00007f2daa9c5dc3 in launch_dispatcher_thread (arg=0x22c40f914) at dispatcher.cc:28 #5 0x00007f2daffd59ca in start_thread () from /lib/libpthread.so.0 #6 0x00007f2dafd3270d in clone () from /lib/libc.so.6 #7 0x0000000000000000 in ?? () Thread 5 (Thread 10224): #0 0x00007f2daffdd93d in read () from /lib/libpthread.so.0 #1 0x00007f2daaa3c7f6 in read (id=0x226fa9e50, pBuf=0xef8d7550, amt=1024, offset=3590330368) at /usr/include/bits/unistd.h:45 #2 seekAndRead (id=0x226fa9e50, pBuf=0xef8d7550, amt=1024, offset=3590330368) at embedded/sqlite3.c:25247 #3 unixRead (id=0x226fa9e50, pBuf=0xef8d7550, amt=1024, offset=3590330368) at embedded/sqlite3.c:25281 #4 0x00007f2daaa56e7a in sqlite3OsRead (pPg=0xef8d7508) at embedded/sqlite3.c:13402 #5 readDbPage (pPg=0xef8d7508) at embedded/sqlite3.c:36959 #6 0x00007f2daaa5c0d4 in sqlite3PagerAcquire (pPager=0x226fa9d08, pgno=3506183, ppPage=0x7f2da8194050, noContent=0) at embedded/sqlite3.c:39100 #7 0x00007f2daaa6d3bb in btreeGetPage (pCur=) at embedded/sqlite3.c:46084 #8 getAndInitPage (pCur=) at embedded/sqlite3.c:46137 #9 balance_nonroot (pCur=) at embedded/sqlite3.c:50401 #10 balance (pCur=) at embedded/sqlite3.c:51077 #11 0x00007f2daaa79b3d in sqlite3BtreeDelete (pCur=0x2ad258548) at embedded/sqlite3.c:51370 #12 0x00007f2daaab3e7f in sqlite3VdbeExec (p=) at embedded/sqlite3.c:63444 #13 0x00007f2daaab9d60 in sqlite3Step (pStmt=0x16aeb8008) at embedded/sqlite3.c:57947 #14 sqlite3_step (pStmt=0x16aeb8008) at embedded/sqlite3.c:58011 #15 0x00007f2daaa2cc2a in PreparedStatement::execute (this=0x24a0ecd70) at sqlite-pst.cc:73 #16 0x00007f2daaa2a855 in StrategicSqlite3::del (this=0x217005510, key=, rowid=56683, vb=, vbver=, cb=...) at sqlite-kvstore.cc:157 #17 0x00007f2daa9cf94b in EventuallyPersistentStore::flushOneDelOrSet (this=0x24a499000, qi=, rejectQueue=) at ep.cc:1930 #18 0x00007f2daa9d002e in EventuallyPersistentStore::flushOne (this=0x24a499000, q=0x24a499130, rejectQueue=0x6f7b467d0) at ep.cc:1966 #19 0x00007f2daa9d00ce in EventuallyPersistentStore::flushSome (this=0x24a499000, q=0x24a499130, rejectQueue=0x6f7b467d0) at ep.cc:1539 #20 0x00007f2daa9fe70e in Flusher::doFlush (this=0x22bac6360) at flusher.cc:242 #21 0x00007f2daa9ff14d in Flusher::step (this=0x50, d=..., tid=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece. ) at flusher.cc:156 #22 0x00007f2daa9ff970 in FlusherStepper::callback (this=, d=..., t=) at flusher.cc:23 #23 0x00007f2daa9c6bef in Task::run (this=, d=..., t=) at dispatcher.hh:139 #24 0x00007f2daa9c516f in Dispatcher::run (this=0x24a0e9700) at dispatcher.cc:119 #25 0x00007f2daa9c5dc3 in launch_dispatcher_thread (arg=0x50) at dispatcher.cc:28 #26 0x00007f2daffd59ca in start_thread () from /lib/libpthread.so.0 #27 0x00007f2dafd3270d in clone () from /lib/libc.so.6 #28 0x0000000000000000 in ?? () Thread 4 (Thread 1760): #0 0x00007f2daffdabc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00007f2daa9c2dea in SyncObject::wait (this=0xf21480, d=...) at syncobject.hh:42 #2 IdleTask::run (this=0xf21480, d=...) at dispatcher.cc:244 #3 0x00007f2daa9c516f in Dispatcher::run (this=0x118a000) at dispatcher.cc:119 #4 0x00007f2daa9c5dc3 in launch_dispatcher_thread (arg=0x118a054) at dispatcher.cc:28 #5 0x00007f2daffd59ca in start_thread () from /lib/libpthread.so.0 #6 0x00007f2dafd3270d in clone () from /lib/libc.so.6 #7 0x0000000000000000 in ?? () Thread 3 (Thread 1762): #0 0x00007f2daffdabc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00007f2daa9e1f62 in SyncObject::wait (this=0x1018000) at syncobject.hh:42 #2 SyncObject::wait (this=0x1018000) at syncobject.hh:58 #3 TapConnMap::wait (this=0x1018000) at tapconnmap.hh:207 #4 EventuallyPersistentEngine::notifyPendingConnections (this=0x1018000) at ep_engine.cc:3652 #5 0x00007f2daa9e2051 in EvpNotifyPendingConns (arg=0x1018000) at ep_engine.cc:967 #6 0x00007f2daffd59ca in start_thread () from /lib/libpthread.so.0 #7 0x00007f2dafd3270d in clone () from /lib/libc.so.6 #8 0x0000000000000000 in ?? () Thread 2 (Thread 1714): #0 0x00007f2dafd32d03 in epoll_wait () from /lib/libc.so.6 #1 0x00007f2db0aeb5a6 in epoll_dispatch (base=0xf3d780, tv=) at epoll.c:404 #2 0x00007f2db0ada171 in event_base_loop (base=0xf3d780, flags=) at event.c:1558 #3 0x0000000000414584 in worker_libevent (arg=0xf1f900) at daemon/thread.c:304 #4 0x00007f2daffd59ca in start_thread () from /lib/libpthread.so.0 #5 0x00007f2dafd3270d in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 1 (Thread 1712): #0 0x00007f2dafc7fa75 in raise () from /lib/libc.so.6 #1 0x00007f2dafc835c0 in abort () from /lib/libc.so.6 #2 0x00007f2dafc78941 in __assert_fail () from /lib/libc.so.6 #3 0x00007f2dadf17b24 in get_engine_handle (h=0x7f2dae11c180, cookie=0x775cdc8) at bucket_engine.c:831 #4 0x00007f2dadf180e6 in bucket_item_delete (handle=0x6a3, cookie=0x6b0, key=0x6, nkey=18446744073709551615, cas=139834147538688, vbucket=19025) at bucket_engine.c:1554 #5 0x000000000040d319 in process_bin_delete (c=0x775cdc8) at daemon/memcached.c:3309 #6 0x000000000040e7f5 in complete_nread_binary (c=0x6a3) at daemon/memcached.c:3362 #7 0x000000000040f394 in complete_nread (c=0x775cdc8) at daemon/memcached.c:3471 #8 conn_nread (c=0x775cdc8) at daemon/memcached.c:5275 #9 0x0000000000406bb4 in event_handler (fd=, which=, arg=0x775cdc8) at daemon/memcached.c:5576 #10 0x00007f2db0ada26c in event_process_active_single_queue (base=0xf3dc80, flags=) at event.c:1308 #11 event_process_active (base=0xf3dc80, flags=) at event.c:1375 #12 event_base_loop (base=0xf3dc80, flags=) at event.c:1572 #13 0x0000000000414584 in worker_libevent (arg=0xf1f700) at daemon/thread.c:304 #14 0x00007f2daffd59ca in start_thread () from /lib/libpthread.so.0 #15 0x00007f2dafd3270d in clone () from /lib/libc.so.6 #16 0x0000000000000000 in ?? () -------------------------------------------------------------------------------- Module information: /opt/couchbase/lib/memcached/libmemcached_utilities.so.0: lrwxrwxrwx 1 couchbase couchbase 31 2012-02-02 16:54 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0 de2eff8fd52f7c4a1d2d91031d3b4577 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 /opt/couchbase/lib/libevent-2.0.so.5: lrwxrwxrwx 1 couchbase couchbase 21 2012-02-02 16:54 /opt/couchbase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0 40238d7626a1a46e3d9844ac3a2b1fd7 /opt/couchbase/lib/libevent-2.0.so.5 /lib/libdl.so.2: lrwxrwxrwx 1 root root 15 2012-01-01 23:40 /lib/libdl.so.2 -> libdl-2.11.1.so b000e5ace83cb232171a8c973c443299 /lib/libdl.so.2 /lib/libm.so.6: lrwxrwxrwx 1 root root 14 2012-01-01 23:40 /lib/libm.so.6 -> libm-2.11.1.so 77385c4ce7ee521e4c38fb9cc13f534e /lib/libm.so.6 /lib/librt.so.1: lrwxrwxrwx 1 root root 15 2012-01-01 23:40 /lib/librt.so.1 -> librt-2.11.1.so 28d8634d49bbe02e93af23a7adc632e8 /lib/librt.so.1 /opt/couchbase/lib/libtcmalloc_minimal.so.0: lrwxrwxrwx 1 couchbase couchbase 28 2012-02-02 16:54 /opt/couchbase/lib/libtcmalloc_minimal.so.0 -> libtcmalloc_minimal.so.0.1.0 4a5efc1231ff781a9cbff0a9c850ef3a /opt/couchbase/lib/libtcmalloc_minimal.so.0 /lib/libpthread.so.0: lrwxrwxrwx 1 root root 20 2012-01-01 23:40 /lib/libpthread.so.0 -> libpthread-2.11.1.so f636a70138b6fbae3d067d560725a238 /lib/libpthread.so.0 /lib/libc.so.6: lrwxrwxrwx 1 root root 14 2012-01-01 23:40 /lib/libc.so.6 -> libc-2.11.1.so fbb713d16c7979bf3d7bcd93ad9eb3de /lib/libc.so.6 /lib64/ld-linux-x86-64.so.2: lrwxrwxrwx 1 root root 12 2012-01-01 23:40 /lib64/ld-linux-x86-64.so.2 -> ld-2.11.1.so 834c71fd9e194a6bf9fc501f89575472 /lib64/ld-linux-x86-64.so.2 /usr/lib/libstdc++.so.6: lrwxrwxrwx 1 root root 19 2012-01-01 23:40 /usr/lib/libstdc++.so.6 -> libstdc++.so.6.0.13 145a441e95ff795ace078ba78840b91f /usr/lib/libstdc++.so.6 /lib/libgcc_s.so.1: -rw-r--r-- 1 root root 92552 2010-03-26 17:12 /lib/libgcc_s.so.1 2fb8dc6079174e77e399d0d5b57b11e5 /lib/libgcc_s.so.1 /opt/couchbase/lib/memcached/stdin_term_handler.so: -rw-r--r-- 1 couchbase couchbase 16160 2011-12-29 08:18 /opt/couchbase/lib/memcached/stdin_term_handler.so 8731cffa349c11c2ffa28e7b5371bf5a /opt/couchbase/lib/memcached/stdin_term_handler.so /opt/couchbase/lib/memcached/bucket_engine.so: lrwxrwxrwx 1 couchbase couchbase 22 2012-02-02 16:54 /opt/couchbase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0 9050bd7f1138001cf06c253bf7d0eebd /opt/couchbase/lib/memcached/bucket_engine.so /opt/couchbase/lib/memcached/ep.so: lrwxrwxrwx 1 couchbase couchbase 11 2012-02-02 16:54 /opt/couchbase/lib/memcached/ep.so -> ep.so.0.0.0 56b39d6e66cbebc603ce01677bdd9233 /opt/couchbase/lib/memcached/ep.so