Basic crash dump analysis of core.1203. Please send the file to support@couchbase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 bin bin 386843 Mar 19 17:33 /opt/couchbase/./bin/memcached f9bfe052a43451e6d545911efa383f59 /opt/couchbase/./bin/memcached memcached 1.4.4_504_g3cec9b1 memcached 1.4.4_504_g3cec9b1 VERSION 1.4.4_504_g3cec9b1 -rw------- 1 couchbase couchbase 1073389568 Apr 30 19:53 core.1203 715521b84c3873526f28e552736e6881 core.1203 version1.4.4_504_g3cec9b1 version1.4.4_504_g3cec9b1 version1.4.4_504_g3cec9b1 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) CentOS (7.0.1-42.el5.centos) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/couchbase/bin/memcached...done. [New Thread 1216] [New Thread 1222] [New Thread 1221] [New Thread 1220] [New Thread 1219] [New Thread 1218] [New Thread 1217] [New Thread 1215] [New Thread 1214] [New Thread 1213] [New Thread 1204] [New Thread 1203] Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.0...done. Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.0 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so Reading symbols from /opt/couchbase/lib/memcached/ep.so...done. Loaded symbols for /opt/couchbase/lib/memcached/ep.so Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff98df5000 Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'. Program terminated with signal 11, Segmentation fault. #0 do_calloc (n=98, elem_size=) at src/linked_list.h:58 58 src/linked_list.h: No such file or directory. in src/linked_list.h Thread 12 (Thread 0x2ae0166f9240 (LWP 1203)): #0 0x00002ae015f658a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002ae0150f2c28 in epoll_dispatch (base=0x16f00000, tv=) at epoll.c:404 #2 0x00002ae0150e1a4c in event_base_loop (base=0x16f00000, flags=0) at event.c:1558 #3 0x0000000000409956 in main (argc=15, argv=0x7fff98cdc5e8) at daemon/memcached.c:7781 Thread 11 (Thread 1204): #0 0x00002ae015f5778b in read () from /lib64/libc.so.6 #1 0x00002ae015efdd57 in _IO_new_file_underflow () from /lib64/libc.so.6 #2 0x00002ae015efe71e in _IO_default_uflow_internal () from /lib64/libc.so.6 #3 0x00002ae015ef9fdb in getc () from /lib64/libc.so.6 #4 0x00002ae0166fa875 in check_stdin_thread (arg=0x403450) at extensions/daemon/stdin_check.c:19 #5 0x00002ae015c7c73d in start_thread () from /lib64/libpthread.so.0 #6 0x00002ae015f654bd in clone () from /lib64/libc.so.6 Thread 10 (Thread 1213): #0 0x00002ae015f2b541 in nanosleep () from /lib64/libc.so.6 #1 0x00002ae015f2b364 in sleep () from /lib64/libc.so.6 #2 0x0000000000414dd8 in check_isasl_db_thread (arg=) at daemon/isasl.c:233 #3 0x00002ae015c7c73d in start_thread () from /lib64/libpthread.so.0 #4 0x00002ae015f654bd in clone () from /lib64/libc.so.6 Thread 9 (Thread 1214): #0 0x00002ae015f658a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002ae0150f2c28 in epoll_dispatch (base=0x16f00500, tv=) at epoll.c:404 #2 0x00002ae0150e1a4c in event_base_loop (base=0x16f00500, flags=0) at event.c:1558 #3 0x0000000000413694 in worker_libevent (arg=0x12aa2500) at daemon/thread.c:304 #4 0x00002ae015c7c73d in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae015f654bd in clone () from /lib64/libc.so.6 Thread 8 (Thread 1215): #0 0x00002ae015f658a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002ae0150f2c28 in epoll_dispatch (base=0x16f00280, tv=) at epoll.c:404 #2 0x00002ae0150e1a4c in event_base_loop (base=0x16f00280, flags=0) at event.c:1558 #3 0x0000000000413694 in worker_libevent (arg=0x12aa2600) at daemon/thread.c:304 #4 0x00002ae015c7c73d in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae015f654bd in clone () from /lib64/libc.so.6 Thread 7 (Thread 1217): #0 0x00002ae015f658a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002ae0150f2c28 in epoll_dispatch (base=0x16f00a00, tv=) at epoll.c:404 #2 0x00002ae0150e1a4c in event_base_loop (base=0x16f00a00, flags=0) at event.c:1558 #3 0x0000000000413694 in worker_libevent (arg=0x12aa2800) at daemon/thread.c:304 #4 0x00002ae015c7c73d in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae015f654bd in clone () from /lib64/libc.so.6 Thread 6 (Thread 1218): #0 0x00002ae015f658a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002ae0150f2c28 in epoll_dispatch (base=0x16f00780, tv=) at epoll.c:404 #2 0x00002ae0150e1a4c in event_base_loop (base=0x16f00780, flags=0) at event.c:1558 #3 0x0000000000413694 in worker_libevent (arg=0x12aa2900) at daemon/thread.c:304 #4 0x00002ae015c7c73d in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae015f654bd in clone () from /lib64/libc.so.6 Thread 5 (Thread 1219): #0 0x00002ae015f5c696 in poll () from /lib64/libc.so.6 #1 0x00002aaaaada69f3 in MemcachedEngine::waitForReadable (this=0x16fea800) at mc-kvstore/mc-engine.cc:1015 #2 0x00002aaaaada6db9 in MemcachedEngine::wait (this=0x46771aa0) at mc-kvstore/mc-engine.cc:1048 #3 0x00002aaaaada8325 in MemcachedEngine::get (this=0x16fea800, key="c5cb3362d5e07ce0", vb=, cb=...) at mc-kvstore/mc-engine.cc:1129 #4 0x00002aaaaadabc74 in MCKVStore::get (this=0x12aa0b80, key="c5cb3362d5e07ce0", vb=200, cb=...) at mc-kvstore/mc-kvstore.cc:57 #5 0x00002aaaaad2e11e in EventuallyPersistentStore::completeBGFetch (this=0x16fca480, key="c5cb3362d5e07ce0", vbucket=200, vbver=65535, rowid=1, cookie=0x16e80dc0, init=781421448404) at ep.cc:1254 #6 0x00002aaaaad4221c in BGFetchCallback::callback (this=) at ep.cc:161 #7 0x00002aaaaad1daaf in Task::run (this=, d=..., t=) at dispatcher.hh:139 #8 0x00002aaaaad1cc01 in Dispatcher::run (this=0x16fa81c0) at dispatcher.cc:119 #9 0x00002aaaaad1d453 in launch_dispatcher_thread (arg=0x46771aa0) at dispatcher.cc:28 #10 0x00002ae015c7c73d in start_thread () from /lib64/libpthread.so.0 #11 0x00002ae015f654bd in clone () from /lib64/libc.so.6 Thread 4 (Thread 1220): #0 0x00002ae015f0d83f in _wordcopy_fwd_dest_aligned () from /lib64/libc.so.6 #1 0x00002ae015f0bdcf in memmove () from /lib64/libc.so.6 #2 0x00002aaaaada6cea in MemcachedEngine::processInput (this=0x16fea800) at mc-kvstore/mc-engine.cc:970 #3 0x00002aaaaada6dc5 in MemcachedEngine::wait (this=0x16fea800) at mc-kvstore/mc-engine.cc:1051 #4 0x00002aaaaada7516 in MemcachedEngine::tap (this=0x16fea800, cb=std::tr1::shared_ptr (count -1425288096) 0x47172bb0) at mc-kvstore/mc-engine.cc:1284 #5 0x00002aaaaadacf65 in MCKVStore::dump (this=0x12aa0b80, cb=std::tr1::shared_ptr (count 0) 0x47172ca0) at mc-kvstore/mc-kvstore.cc:150 #6 0x00002aaaaad258d5 in EventuallyPersistentStore::warmup (this=0x16fca480, st=std::map with 103 elements = {...}, keysOnly=false) at ep.cc:2682 #7 0x00002aaaaad61cb1 in Flusher::step (this=0x16fc85a0, d=, tid=Traceback (most recent call last): File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 32, in to_string return '%s (count %d) %s' % (self.typename, TypeError: int argument required ) at flusher.cc:211 #8 0x00002aaaaad6236e in FlusherStepper::callback (this=0x302873c0, d=..., t=) at flusher.cc:23 #9 0x00002aaaaad1daaf in Task::run (this=, d=..., t=) at dispatcher.hh:139 #10 0x00002aaaaad1cc01 in Dispatcher::run (this=0x16fa8380) at dispatcher.cc:119 #11 0x00002aaaaad1d453 in launch_dispatcher_thread (arg=0x177ba988) at dispatcher.cc:28 #12 0x00002ae015c7c73d in start_thread () from /lib64/libpthread.so.0 #13 0x00002ae015f654bd in clone () from /lib64/libc.so.6 Thread 3 (Thread 1221): #0 0x00002ae015c81150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaaaad1adb8 in wait (this=0x16fe0080, d=...) at syncobject.hh:42 #2 IdleTask::run (this=0x16fe0080, d=...) at dispatcher.cc:244 #3 0x00002aaaaad1cc01 in Dispatcher::run (this=0x16fa88c0) at dispatcher.cc:119 #4 0x00002aaaaad1d453 in launch_dispatcher_thread (arg=0x16fa8914) at dispatcher.cc:28 #5 0x00002ae015c7c73d in start_thread () from /lib64/libpthread.so.0 #6 0x00002ae015f654bd in clone () from /lib64/libc.so.6 Thread 2 (Thread 1222): #0 0x00002ae015c81150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaaaad43700 in wait (this=0x16fce000) at syncobject.hh:42 #2 wait (this=0x16fce000) at syncobject.hh:58 #3 wait (this=0x16fce000) at tapconnmap.hh:209 #4 EventuallyPersistentEngine::notifyPendingConnections (this=0x16fce000) at ep_engine.cc:3197 #5 0x00002aaaaad437d1 in EvpNotifyPendingConns (arg=0x16fce000) at ep_engine.cc:971 #6 0x00002ae015c7c73d in start_thread () from /lib64/libpthread.so.0 #7 0x00002ae015f654bd in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x4496f940 (LWP 1216)): #0 do_calloc (n=98, elem_size=) at src/linked_list.h:58 #1 tc_calloc (n=98, elem_size=) at src/tcmalloc.cc:1437 #2 0x00002aaaaaab1d7a in topkey_item_init (tk=0x16f0c0f0, key=0x2aaab5534318, nkey=18, ct=760) at topkeys.c:10 #3 topkeys_item_get_or_create (tk=0x16f0c0f0, key=0x2aaab5534318, nkey=18, ct=760) at topkeys.c:110 #4 0x00002aaaaaaafce7 in bucket_store (handle=, cookie=, itm=0x170156c0, cas=0x16e81858, operation=OPERATION_SET, vbucket=) at bucket_engine.c:1848 #5 0x000000000041129d in complete_update_bin (c=0x16e81600) at daemon/memcached.c:1794 #6 complete_nread_binary (c=0x16e81600) at daemon/memcached.c:3623 #7 0x0000000000411bcd in complete_nread (c=0x16e81600) at daemon/memcached.c:3741 #8 conn_nread (c=0x16e81600) at daemon/memcached.c:5583 #9 0x0000000000407234 in event_handler (fd=, which=, arg=0x16e81600) at daemon/memcached.c:5884 #10 0x00002ae0150e1df9 in event_process_active_single_queue (base=0x16f00c80, flags=0) at event.c:1308 #11 event_process_active (base=0x16f00c80, flags=0) at event.c:1375 #12 event_base_loop (base=0x16f00c80, flags=0) at event.c:1572 #13 0x0000000000413694 in worker_libevent (arg=0x12aa2700) at daemon/thread.c:304 #14 0x00002ae015c7c73d in start_thread () from /lib64/libpthread.so.0 #15 0x00002ae015f654bd in clone () from /lib64/libc.so.6 -------------------------------------------------------------------------------- Module information: /opt/couchbase/lib/memcached/libmemcached_utilities.so.0: /opt/couchbase/lib/libevent-2.0.so.5: /lib64/libdl.so.2: /lib64/libm.so.6: /lib64/librt.so.1: /opt/couchbase/lib/libtcmalloc_minimal.so.0: /lib64/libpthread.so.0: /lib64/libc.so.6: /lib64/ld-linux-x86-64.so.2: /usr/lib64/libstdc++.so.6: /lib64/libgcc_s.so.1: /opt/couchbase/lib/memcached/stdin_term_handler.so: /opt/couchbase/lib/memcached/bucket_engine.so: /opt/couchbase/lib/memcached/ep.so: /lib64/libnss_files.so.2: