Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
3.0
-
Security Level: Public
-
Build 3.0.0-918
ubuntu 12.04, 64 bit
-
Untriaged
-
Unknown
-
June 30 - July 18
Description
http://qa.hq.northscale.net/job/ubuntu_x64--01_02--rebalanceXDCR-P0/19/consoleFull
[Core Logs]
Basic crash dump analysis of /tmp//core.memcached.21508.
Please send the file to support@couchbase.com
--------------------------------------------------------------------------------
File information:
-rwxr-xr-x 1 couchbase couchbase 4958595 2014-07-01 19:00 /opt/couchbase/bin/memcached
6cd323a6609b29186b45436c840e7580 /opt/couchbase/bin/memcached
rw------ 1 couchbase couchbase 332505088 2014-07-02 17:38 /tmp//core.memcached.21508
5b2434f7bc783b86c7d165249c783519 /tmp//core.memcached.21508
--------------------------------------------------------------------------------
Core file callstacks:
GNU gdb (GDB) 7.1-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 21511]
[New Thread 21512]
[New Thread 23321]
[New Thread 25659]
[New Thread 29947]
[New Thread 21508]
[New Thread 29948]
[New Thread 21509]
[New Thread 21510]
[New Thread 29934]
[New Thread 29950]
[New Thread 27817]
[New Thread 21514]
[New Thread 29949]
[New Thread 21515]
[New Thread 21513]
warning: Can't read pathname for load map: Input/output error.
Reading symbols from /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0
Reading symbols from /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1...done.
Loaded symbols for /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1
Reading symbols from /opt/couchbase/bin/../lib/libplatform.so.0.1.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libplatform.so.0.1.0
Reading symbols from /opt/couchbase/bin/../lib/libcJSON.so.1.0.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libcJSON.so.1.0.0
Reading symbols from /opt/couchbase/bin/../lib/libJSON_checker.so...done.
Loaded symbols for /opt/couchbase/bin/../lib/libJSON_checker.so
Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1
Reading symbols from /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4
Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5...done.
Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.0.so.5
Reading symbols from /lib/libssl.so.0.9.8...(no debugging symbols found)...done.
Loaded symbols for /lib/libssl.so.0.9.8
Reading symbols from /lib/libcrypto.so.0.9.8...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypto.so.0.9.8
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so
Reading symbols from /opt/couchbase/lib/libdirutils.so.0.1.0...done.
Loaded symbols for /opt/couchbase/lib/libdirutils.so.0.1.0
Reading symbols from /opt/couchbase/lib/libv8.so...done.
Loaded symbols for /opt/couchbase/lib/libv8.so
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
Program terminated with signal 6, Aborted.
#0 0x00007f4c6455ca75 in raise () from /lib/libc.so.6
Thread 16 (Thread 21513):
#0 0x00007f4c646102d3 in epoll_wait () from /lib/libc.so.6
#1 0x00007f4c65c875a6 in epoll_dispatch (base=0x6312780,
tv=<value optimized out>) at epoll.c:404
#2 0x00007f4c65c72a04 in event_base_loop (base=0x6312780,
flags=<value optimized out>) at event.c:1558
#3 0x00007f4c666f20df in platform_thread_wrap (arg=0x1a5e110)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f4c6546c9ca in start_thread () from /lib/libpthread.so.0
#5 0x00007f4c6460fcdd in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()
Thread 15 (Thread 21515):
#0 0x00007f4c646102d3 in epoll_wait () from /lib/libc.so.6
#1 0x00007f4c65c875a6 in epoll_dispatch (base=0x6312c80,
tv=<value optimized out>) at epoll.c:404
#2 0x00007f4c65c72a04 in event_base_loop (base=0x6312c80,
flags=<value optimized out>) at event.c:1558
#3 0x00007f4c666f20df in platform_thread_wrap (arg=0x1a5e130)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f4c6546c9ca in start_thread () from /lib/libpthread.so.0
#5 0x00007f4c6460fcdd in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()
Thread 14 (Thread 29949):
#0 0x00007f4c65471bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
#1 0x00007f4c666f21cb in cb_cond_timedwait (cond=0x64c8058, mutex=0x64c8020,
ms=<value optimized out>)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f4c5fa2c5ff in SyncObject::wait (this=0x64c8018,
tv=<value optimized out>)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 0x00007f4c5fa27cfc in ExecutorPool::trySleep (this=0x64c8000, t=...,
now=...)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:190
#4 0x00007f4c5fa28066 in ExecutorPool::_nextTask (this=0x64c8000, t=...,
tick=8 '\b')
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:155
#5 0x00007f4c5fa280bf in ExecutorPool::nextTask (this=0x64c8000, t=...,
tick=98 'b')
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:165
#6 0x00007f4c5fa39362 in ExecutorThread::run (this=0xb338760)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
#7 0x00007f4c5fa397ad in launch_executor_thread (arg=0x64c805c)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#8 0x00007f4c666f20df in platform_thread_wrap (arg=0x9045ef0)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#9 0x00007f4c6546c9ca in start_thread () from /lib/libpthread.so.0
#10 0x00007f4c6460fcdd in clone () from /lib/libc.so.6
#11 0x0000000000000000 in ?? ()
Thread 13 (Thread 21514):
#0 0x00007f4c646102d3 in epoll_wait () from /lib/libc.so.6
#1 0x00007f4c65c875a6 in epoll_dispatch (base=0x6312a00,
tv=<value optimized out>) at epoll.c:404
#2 0x00007f4c65c72a04 in event_base_loop (base=0x6312a00,
flags=<value optimized out>) at event.c:1558
#3 0x00007f4c666f20df in platform_thread_wrap (arg=0x1a5e120)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f4c6546c9ca in start_thread () from /lib/libpthread.so.0
#5 0x00007f4c6460fcdd in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()
Thread 12 (Thread 27817):
#0 0x00007f4c6547185c in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
#1 0x00007f4c62d00034 in engine_shutdown_thread (arg=0x6325180)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1610
#2 0x00007f4c666f20df in platform_thread_wrap (arg=0x1a5f700)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#3 0x00007f4c6546c9ca in start_thread () from /lib/libpthread.so.0
#4 0x00007f4c6460fcdd in clone () from /lib/libc.so.6
#5 0x0000000000000000 in ?? ()
Thread 11 (Thread 29950):
#0 0x00007f4c65471bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
#1 0x00007f4c666f21cb in cb_cond_timedwait (cond=0x64c8058, mutex=0x64c8020,
ms=<value optimized out>)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f4c5fa2c5ff in SyncObject::wait (this=0x64c8018,
tv=<value optimized out>)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 0x00007f4c5fa27cfc in ExecutorPool::trySleep (this=0x64c8000, t=...,
now=...)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:190
#4 0x00007f4c5fa28066 in ExecutorPool::_nextTask (this=0x64c8000, t=...,
tick=16 '\020')
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:155
#5 0x00007f4c5fa280bf in ExecutorPool::nextTask (this=0x64c8000, t=...,
tick=97 'a')
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:165
#6 0x00007f4c5fa39362 in ExecutorThread::run (this=0xb338260)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
#7 0x00007f4c5fa397ad in launch_executor_thread (arg=0x64c805c)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#8 0x00007f4c666f20df in platform_thread_wrap (arg=0x9045920)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#9 0x00007f4c6546c9ca in start_thread () from /lib/libpthread.so.0
#10 0x00007f4c6460fcdd in clone () from /lib/libc.so.6
#11 0x0000000000000000 in ?? ()
Thread 10 (Thread 29934):
#0 0x00007f4c6547185c in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
#1 0x00007f4c62d00034 in engine_shutdown_thread (arg=0xa0a3e20)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1610
#2 0x00007f4c666f20df in platform_thread_wrap (arg=0xafe2e90)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#3 0x00007f4c6546c9ca in start_thread () from /lib/libpthread.so.0
#4 0x00007f4c6460fcdd in clone () from /lib/libc.so.6
#5 0x0000000000000000 in ?? ()
Thread 9 (Thread 21510):
#0 0x00007f4c65471bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
#1 0x00007f4c666f21cb in cb_cond_timedwait (cond=0x7f4c6390e240,
mutex=0x7f4c6390e200, ms=<value optimized out>)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f4c6370d1e8 in logger_thead_main (arg=<value optimized out>)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/extensions/loggers/file_logger.c:372
#3 0x00007f4c666f20df in platform_thread_wrap (arg=0x1a5e070)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f4c6546c9ca in start_thread () from /lib/libpthread.so.0
#5 0x00007f4c6460fcdd in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()
Thread 8 (Thread 21509):
#0 0x00007f4c64601a6d in read () from /lib/libc.so.6
#1 0x00007f4c6459c598 in _IO_file_underflow () from /lib/libc.so.6
#2 0x00007f4c6459e13e in _IO_default_uflow () from /lib/libc.so.6
#3 0x00007f4c6459268e in _IO_getline_info () from /lib/libc.so.6
#4 0x00007f4c64591579 in fgets () from /lib/libc.so.6
#5 0x00007f4c64110a91 in fgets (arg=<value optimized out>)
at /usr/include/bits/stdio2.h:255
#6 check_stdin_thread (arg=<value optimized out>)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/extensions/daemon/stdin_check.c:38
#7 0x00007f4c666f20df in platform_thread_wrap (arg=0x1a5e060)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x00007f4c6546c9ca in start_thread () from /lib/libpthread.so.0
#9 0x00007f4c6460fcdd in clone () from /lib/libc.so.6
#10 0x0000000000000000 in ?? ()
Thread 7 (Thread 29948):
#0 0x00007f4c645af645 in mempcpy () from /lib/libc.so.6
#1 0x00007f4c6459ef8e in _IO_default_xsputn () from /lib/libc.so.6
#2 0x00007f4c6456ea10 in vfprintf () from /lib/libc.so.6
#3 0x00007f4c64626d30 in __vsnprintf_chk () from /lib/libc.so.6
#4 0x00007f4c64626c6a in __snprintf_chk () from /lib/libc.so.6
#5 0x00007f4c5f9e6bc5 in snprintf (this=0x6ec5698,
add_stat=0x4099e0 <append_stats>, cookie=0x62a5b00)
at /usr/include/bits/stdio2.h:66
#6 CheckpointManager::addStats (this=0x6ec5698,
add_stat=0x4099e0 <append_stats>, cookie=0x62a5b00)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/checkpoint.cc:1560
#7 0x00007f4c5fa24de3 in StatCheckpointVisitor::addCheckpointStat(void const*, void (char const*, unsigned short, char const*, unsigned int, void const*), EventuallyPersistentStore*, RCPtr<VBucket>&) ()
from /opt/couchbase/lib/memcached/ep.so
#8 0x00007f4c5fa24eb8 in StatCheckpointVisitor::visitBucket(RCPtr<VBucket>&)
() from /opt/couchbase/lib/memcached/ep.so
#9 0x00007f4c5f9f0ffc in EventuallyPersistentStore::visit (
this=<value optimized out>, visitor=...)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/ep.cc:2784
#10 0x00007f4c5fa26f68 in StatCheckpointTask::run() ()
from /opt/couchbase/lib/memcached/ep.so
#11 0x00007f4c5fa39401 in ExecutorThread::run (this=0xb338440)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:95
#12 0x00007f4c5fa397ad in launch_executor_thread (arg=0x7f4c5c476998)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#13 0x00007f4c666f20df in platform_thread_wrap (arg=0x9045e20)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#14 0x00007f4c6546c9ca in start_thread () from /lib/libpthread.so.0
#15 0x00007f4c6460fcdd in clone () from /lib/libc.so.6
#16 0x0000000000000000 in ?? ()
Thread 6 (Thread 21508):
#0 0x00007f4c646102d3 in epoll_wait () from /lib/libc.so.6
#1 0x00007f4c65c875a6 in epoll_dispatch (base=0x6312000,
tv=<value optimized out>) at epoll.c:404
#2 0x00007f4c65c72a04 in event_base_loop (base=0x6312000,
flags=<value optimized out>) at event.c:1558
#3 0x00000000004108d4 in main (argc=<value optimized out>,
argv=<value optimized out>)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/daemon/memcached.c:8768
Thread 5 (Thread 29947):
#0 0x00007f4c65471bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
#1 0x00007f4c666f21cb in cb_cond_timedwait (cond=0x64c8058, mutex=0x64c8020,
ms=<value optimized out>)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f4c5fa2c5ff in SyncObject::wait (this=0x64c8018,
tv=<value optimized out>)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 0x00007f4c5fa27cfc in ExecutorPool::trySleep (this=0x64c8000, t=...,
now=...)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:190
#4 0x00007f4c5fa28066 in ExecutorPool::_nextTask (this=0x64c8000, t=...,
tick=0 '\000')
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:155
#5 0x00007f4c5fa280bf in ExecutorPool::nextTask (this=0x64c8000, t=...,
tick=99 'c')
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:165
#6 0x00007f4c5fa39362 in ExecutorThread::run (this=0xb337d60)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
#7 0x00007f4c5fa397ad in launch_executor_thread (arg=0x64c805c)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#8 0x00007f4c666f20df in platform_thread_wrap (arg=0x9045c10)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#9 0x00007f4c6546c9ca in start_thread () from /lib/libpthread.so.0
#10 0x00007f4c6460fcdd in clone () from /lib/libc.so.6
#11 0x0000000000000000 in ?? ()
Thread 4 (Thread 25659):
#0 0x00007f4c6547185c in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
#1 0x00007f4c62d00034 in engine_shutdown_thread (arg=0x63242a0)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1610
#2 0x00007f4c666f20df in platform_thread_wrap (arg=0x1a5ee40)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#3 0x00007f4c6546c9ca in start_thread () from /lib/libpthread.so.0
#4 0x00007f4c6460fcdd in clone () from /lib/libc.so.6
#5 0x0000000000000000 in ?? ()
Thread 3 (Thread 23321):
#0 0x00007f4c645d369d in nanosleep () from /lib/libc.so.6
#1 0x00007f4c64608df4 in usleep () from /lib/libc.so.6
#2 0x00007f4c5fa37905 in updateStatsThread (arg=<value optimized out>)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/memory_tracker.cc:36
#3 0x00007f4c666f20df in platform_thread_wrap (arg=0x1a5e1f0)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f4c6546c9ca in start_thread () from /lib/libpthread.so.0
#5 0x00007f4c6460fcdd in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()
Thread 2 (Thread 21512):
#0 0x00007f4c646102d3 in epoll_wait () from /lib/libc.so.6
#1 0x00007f4c65c875a6 in epoll_dispatch (base=0x6312500,
tv=<value optimized out>) at epoll.c:404
#2 0x00007f4c65c72a04 in event_base_loop (base=0x6312500,
flags=<value optimized out>) at event.c:1558
#3 0x00007f4c666f20df in platform_thread_wrap (arg=0x1a5e100)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f4c6546c9ca in start_thread () from /lib/libpthread.so.0
#5 0x00007f4c6460fcdd in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()
Thread 1 (Thread 21511):
#0 0x00007f4c6455ca75 in raise () from /lib/libc.so.6
#1 0x00007f4c645605c0 in abort () from /lib/libc.so.6
#2 0x00007f4c64555941 in __assert_fail () from /lib/libc.so.6
#3 0x00000000004083ec in decrement_session_ctr ()
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/daemon/memcached.c:7731
#4 0x00007f4c5fa1f0be in EventuallyPersistentEngine::decrementSessionCtr (
this=0x6360800)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/ep_engine.h:510
#5 0x00007f4c5fa1df0b in processUnknownCommand (h=0x6360800,
cookie=0x617d800, request=0x6258000,
response=0x40cb20 <binary_response_handler>)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/ep_engine.cc:1193
#6 0x00007f4c5fa1ee0c in EvpUnknownCommand (handle=0x6360800,
cookie=0x617d800, request=0x6258000,
response=0x40cb20 <binary_response_handler>)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/ep_engine.cc:1260
#7 0x00007f4c62d02bb3 in bucket_unknown_command (handle=0x7f4c62f09220,
cookie=0x617d800, request=0x6258000,
response=0x40cb20 <binary_response_handler>)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:3215
#8 0x00000000004191ca in process_bin_unknown_packet (c=0x617d800)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/daemon/memcached.c:2663
#9 process_bin_packet (c=0x617d800)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/daemon/memcached.c:5392
#10 complete_nread (c=0x617d800)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/daemon/memcached.c:5796
#11 conn_nread (c=0x617d800)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/daemon/memcached.c:7003
#12 0x000000000040c73d in event_handler (fd=<value optimized out>,
which=<value optimized out>, arg=0x617d800)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/daemon/memcached.c:7276
#13 0x00007f4c65c72afc in event_process_active_single_queue (base=0x6312280,
flags=<value optimized out>) at event.c:1308
#14 event_process_active (base=0x6312280, flags=<value optimized out>)
at event.c:1375
#15 event_base_loop (base=0x6312280, flags=<value optimized out>)
at event.c:1572
#16 0x00007f4c666f20df in platform_thread_wrap (arg=0x1a5e0f0)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#17 0x00007f4c6546c9ca in start_thread () from /lib/libpthread.so.0
#18 0x00007f4c6460fcdd in clone () from /lib/libc.so.6
#19 0x0000000000000000 in ?? ()
--------------------------------------------------------------------------------
Module information:
/opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0:
rw-rr- 1 couchbase couchbase 415163 2014-07-01 19:01 /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0
b5f73dc290899e88fded946fde6a5244 /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0
/opt/couchbase/bin/../lib/libcbsasl.so.1.1.1:
rw-rr- 1 couchbase couchbase 412706 2014-07-01 19:01 /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1
bf540b44de8c56f1bcab31c97edf825d /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1
/opt/couchbase/bin/../lib/libplatform.so.0.1.0:
rw-rr- 1 couchbase couchbase 556937 2014-07-01 19:01 /opt/couchbase/bin/../lib/libplatform.so.0.1.0
852ed58c53d2c9e69cef21d27e08370b /opt/couchbase/bin/../lib/libplatform.so.0.1.0
/opt/couchbase/bin/../lib/libcJSON.so.1.0.0:
rw-rr- 1 couchbase couchbase 95332 2014-07-01 19:01 /opt/couchbase/bin/../lib/libcJSON.so.1.0.0
6e1bac6b7025efd048042f0da34841f5 /opt/couchbase/bin/../lib/libcJSON.so.1.0.0
/opt/couchbase/bin/../lib/libJSON_checker.so:
rw-rr- 1 couchbase couchbase 35648 2014-07-01 19:01 /opt/couchbase/bin/../lib/libJSON_checker.so
1fd90948b5c1feb47b290e8d1714f4ac /opt/couchbase/bin/../lib/libJSON_checker.so
/opt/couchbase/bin/../lib/libsnappy.so.1:
lrwxrwxrwx 1 couchbase couchbase 18 2014-07-01 22:35 /opt/couchbase/bin/../lib/libsnappy.so.1 -> libsnappy.so.1.1.2
7c50c2a147ab8247b5a2b61a38604ccc /opt/couchbase/bin/../lib/libsnappy.so.1
/opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4:
lrwxrwxrwx 1 couchbase couchbase 28 2014-07-01 22:35 /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4 -> libtcmalloc_minimal.so.4.2.1
a0669ae75ee5ae5352dd7c2704adf766 /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4
/opt/couchbase/bin/../lib/libevent_core-2.0.so.5:
lrwxrwxrwx 1 couchbase couchbase 26 2014-07-01 22:35 /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 -> libevent_core-2.0.so.5.1.0
7c23d736254e9dc999fcad4782e986e5 /opt/couchbase/bin/../lib/libevent_core-2.0.so.5
/lib/libssl.so.0.9.8:
rw-rr- 1 root root 333904 2012-04-24 08:29 /lib/libssl.so.0.9.8
d32b70676a28d4b191a859f5621d489d /lib/libssl.so.0.9.8
/lib/libcrypto.so.0.9.8:
rw-rr- 1 root root 1622304 2012-04-24 08:29 /lib/libcrypto.so.0.9.8
34885593167472209511acfacef5a962 /lib/libcrypto.so.0.9.8
/lib/libpthread.so.0:
lrwxrwxrwx 1 root root 20 2012-05-24 11:03 /lib/libpthread.so.0 -> libpthread-2.11.1.so
f636a70138b6fbae3d067d560725a238 /lib/libpthread.so.0
/lib/libdl.so.2:
lrwxrwxrwx 1 root root 15 2012-05-24 11:03 /lib/libdl.so.2 -> libdl-2.11.1.so
b000e5ace83cb232171a8c973c443299 /lib/libdl.so.2
/lib/librt.so.1:
lrwxrwxrwx 1 root root 15 2012-05-24 11:03 /lib/librt.so.1 -> librt-2.11.1.so
28d8634d49bbe02e93af23a7adc632e8 /lib/librt.so.1
/usr/lib/libstdc++.so.6:
lrwxrwxrwx 1 root root 19 2012-05-24 11:03 /usr/lib/libstdc+.so.6 -> libstdc+.so.6.0.13
778aaa89a71cdc1f55b5986a9a9e3499 /usr/lib/libstdc++.so.6
/lib/libm.so.6:
lrwxrwxrwx 1 root root 14 2012-05-24 11:03 /lib/libm.so.6 -> libm-2.11.1.so
77385c4ce7ee521e4c38fb9cc13f534e /lib/libm.so.6
/lib/libgcc_s.so.1:
rw-rr- 1 root root 92552 2012-03-08 20:17 /lib/libgcc_s.so.1
64bedbbb11f8452c9811c7d60c56b49a /lib/libgcc_s.so.1
/lib/libc.so.6:
lrwxrwxrwx 1 root root 14 2012-05-24 11:03 /lib/libc.so.6 -> libc-2.11.1.so
940dc8606a7975374e0be61e44c12fa8 /lib/libc.so.6
/lib/libz.so.1:
lrwxrwxrwx 1 root root 15 2012-05-24 10:41 /lib/libz.so.1 -> libz.so.1.2.3.3
6abd7af4f2752f371b0ecb7cc601c3ac /lib/libz.so.1
/lib64/ld-linux-x86-64.so.2:
lrwxrwxrwx 1 root root 12 2012-05-24 11:03 /lib64/ld-linux-x86-64.so.2 -> ld-2.11.1.so
c024251e1af3963fbb3fcef25d58410e /lib64/ld-linux-x86-64.so.2
/opt/couchbase/lib/memcached/stdin_term_handler.so:
rw-rr- 1 couchbase couchbase 119503 2014-07-01 19:01 /opt/couchbase/lib/memcached/stdin_term_handler.so
2ed71f63866e3ab453c975ecdd722b3b /opt/couchbase/lib/memcached/stdin_term_handler.so
/opt/couchbase/lib/memcached/file_logger.so:
rw-rr- 1 couchbase couchbase 143927 2014-07-01 19:01 /opt/couchbase/lib/memcached/file_logger.so
77dea259b81310f73ea9798dd8f60e5d /opt/couchbase/lib/memcached/file_logger.so
/opt/couchbase/lib/memcached/bucket_engine.so:
rw-rr- 1 couchbase couchbase 421096 2014-07-01 19:01 /opt/couchbase/lib/memcached/bucket_engine.so
f7d424c7f5a853fd5f84b6cffcfb1b38 /opt/couchbase/lib/memcached/bucket_engine.so
/opt/couchbase/lib/memcached/ep.so:
rw-rr- 1 couchbase couchbase 18673956 2014-07-01 19:01 /opt/couchbase/lib/memcached/ep.so
c30a26c281056fd381a09425633277fd /opt/couchbase/lib/memcached/ep.so
/opt/couchbase/lib/libcouchstore.so:
rw-rr- 1 couchbase couchbase 3933395 2014-07-01 19:01 /opt/couchbase/lib/libcouchstore.so
66dd2e0d770234cddd60446cfd95f867 /opt/couchbase/lib/libcouchstore.so
/opt/couchbase/lib/libdirutils.so.0.1.0:
rw-rr- 1 couchbase couchbase 181445 2014-07-01 19:01 /opt/couchbase/lib/libdirutils.so.0.1.0
de8c58cc7f1bc7a989e66becfefb2db2 /opt/couchbase/lib/libdirutils.so.0.1.0
/opt/couchbase/lib/libv8.so:
rw-rr- 1 couchbase couchbase 116909487 2014-07-01 19:01 /opt/couchbase/lib/libv8.so
02be0eea832a546d2493cf530b4535ad /opt/couchbase/lib/libv8.so
/opt/couchbase/lib/libicui18n.so.44:
lrwxrwxrwx 1 couchbase couchbase 18 2014-07-01 22:35 /opt/couchbase/lib/libicui18n.so.44 -> libicui18n.so.44.0
c7b26e773e42456459912c214e35c571 /opt/couchbase/lib/libicui18n.so.44
/opt/couchbase/lib/libicuuc.so.44:
lrwxrwxrwx 1 couchbase couchbase 16 2014-07-01 22:35 /opt/couchbase/lib/libicuuc.so.44 -> libicuuc.so.44.0
5183a8c242bef1c814ad4eb30308f779 /opt/couchbase/lib/libicuuc.so.44
/opt/couchbase/lib/libicudata.so.44:
lrwxrwxrwx 1 couchbase couchbase 18 2014-07-01 22:35 /opt/couchbase/lib/libicudata.so.44 -> libicudata.so.44.0
93c3bc2bf90a238425388d76543168a0 /opt/couchbase/lib/libicudata.so.44