Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-10038

ep-engine dies with segmentation fault on warmup

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • 3.0
    • 3.0
    • couchbase-bucket
    • Security Level: Public
    • None
    • Untriaged
    • 2
    • June 30 - July 18

    Description

      The bucket has few million of items and I'm pretty sure few previous warmups worked fine. But this time (perhaps just luck or maybe there's some disk state that's damaged). I'm getting a continuous crashes:

      warning: Could not load shared library symbols for linux-gate.so.1.
      Do you need "set solib-search-path" or "set sysroot"?
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
      Core was generated by `/root/src/altoros/moxi/repo30/install/bin/memcached.orig -u root -X /root/src/a'.
      Program terminated with signal 11, Segmentation fault.
      #0 0xf29ae56a in cJSON_GetObjectItem (object=0x0, string=0xf2b0187b "state") at /root/src/altoros/moxi/repo30/cmake/libvbucket/src/cJSON.c:452
      452 cJSON *cJSON_GetObjectItem(cJSON *object,const char *string)

      {cJSON *c=object->child; while (c && cJSON_strcasecmp(c->string,string)) c=c->next; return c;}

      (gdb) bt
      #0 0xf29ae56a in cJSON_GetObjectItem (object=0x0, string=0xf2b0187b "state") at /root/src/altoros/moxi/repo30/cmake/libvbucket/src/cJSON.c:452
      #1 0xf2acf29c in CouchKVStore::readVBState (db=0x8e72480, vbId=0, vbState=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/couch-kvstore/couch-kvstore.cc:1812
      #2 0xf2ac8cf3 in CouchKVStore::listPersistedVbuckets (this=0xbfae1a0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/couch-kvstore/couch-kvstore.cc:551
      #3 0xf2a3ac4e in EventuallyPersistentStore::loadVBucketState (this=0x92a42c0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/ep.cc:2636
      #4 0xf2ab5604 in Warmup::initialize (this=0xbf6c1b0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/warmup.cc:439
      #5 0xf2ab8230 in WarmupInitialize::run (this=0xbf4edc0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/warmup.h:220
      #6 0xf2a7bc6a in ExecutorThread::run (this=0xbf4e550) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:93
      #7 0xf2a7b71c in launch_executor_thread (arg=0xbf4e550) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:33
      #8 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e340) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #9 0xf76bccf1 in start_thread (arg=0xeeb4fb40) at pthread_create.c:311
      #10 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131
      (gdb) thread app all bt

      Thread 27 (Thread 0xe9344b40 (LWP 22038)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf76c33a2 in __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:144
      #2 0xf76bee24 in _L_lock_770 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
      #3 0xf76bec63 in _GI__pthread_mutex_lock (mutex=0xbf68a34) at pthread_mutex_lock.c:64
      #4 0xf7774194 in cb_mutex_enter (mutex=0xbf68a34) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:85
      #5 0xf2a7af04 in Mutex::acquire (this=0xbf68a30) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/mutex.cc:32
      #6 0xf2a1a659 in LockHolder::lock (this=0xe9344198) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/locks.h:66
      #7 0xf2a1a606 in LockHolder::LockHolder (this=0xe9344198, m=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/locks.h:44
      #8 0xf2a6ddec in ExecutorPool::trySleep (this=0xbf68a20, t=..., now=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:153
      #9 0xf2a6dd89 in ExecutorPool::nextTask (this=0xbf68a20, t=..., tick=1 '\001') at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:144
      #10 0xf2a7bb08 in ExecutorThread::run (this=0xbf4eaa0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:77
      #11 0xf2a7b71c in launch_executor_thread (arg=0xbf4eaa0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:33
      #12 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e3c0) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #13 0xf76bccf1 in start_thread (arg=0xe9344b40) at pthread_create.c:311
      #14 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 26 (Thread 0xec34ab40 (LWP 22032)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf76c33a2 in __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:144
      #2 0xf76c62f0 in _L_cond_lock_773 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
      #3 0xf76c6061 in __pthread_mutex_cond_lock (mutex=0xbf68a34) at ../nptl/pthread_mutex_lock.c:64
      #4 0xf76c0c7c in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:364
      #5 0xf7774442 in cb_cond_timedwait (cond=0xbf68a54, mutex=0xbf68a34, ms=2000) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:156
      #6 0xf2a6ff21 in SyncObject::wait (this=0xbf68a30, tv=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/syncobject.h:74
      #7 0xf2a6def7 in ExecutorPool::trySleep (this=0xbf68a20, t=..., now=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:168
      #8 0xf2a6dd89 in ExecutorPool::nextTask (this=0xbf68a20, t=..., tick=1 '\001') at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:144
      #9 0xf2a7bb08 in ExecutorThread::run (this=0xbf4e7d0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:77
      #10 0xf2a7b71c in launch_executor_thread (arg=0xbf4e7d0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:33
      #11 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e318) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #12 0xf76bccf1 in start_thread (arg=0xec34ab40) at pthread_create.c:311
      #13 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 25 (Thread 0xee34eb40 (LWP 22028)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf76c33a2 in __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:144
      #2 0xf76bee24 in _L_lock_770 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
      #3 0xf76bec63 in _GI__pthread_mutex_lock (mutex=0xbf68a34) at pthread_mutex_lock.c:64
      #4 0xf7774194 in cb_mutex_enter (mutex=0xbf68a34) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:85
      #5 0xf2a7af04 in Mutex::acquire (this=0xbf68a30) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/mutex.cc:32
      #6 0xf2a1a659 in LockHolder::lock (this=0xee34e198) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/locks.h:66
      #7 0xf2a1a606 in LockHolder::LockHolder (this=0xee34e198, m=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/locks.h:44
      #8 0xf2a6ddec in ExecutorPool::trySleep (this=0xbf68a20, t=..., now=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:153
      #9 0xf2a6dd89 in ExecutorPool::nextTask (this=0xbf68a20, t=..., tick=2 '\002') at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:144
      #10 0xf2a7bb08 in ExecutorThread::run (this=0xbf4e500) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:77
      #11 0xf2a7b71c in launch_executor_thread (arg=0xbf4e500) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:33
      #12 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e338) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #13 0xf76bccf1 in start_thread (arg=0xee34eb40) at pthread_create.c:311
      #14 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 24 (Thread 0xecb4bb40 (LWP 22031)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf76c33a2 in __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:144
      #2 0xf76c62f0 in _L_cond_lock_773 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
      #3 0xf76c6061 in __pthread_mutex_cond_lock (mutex=0xbf68a34) at ../nptl/pthread_mutex_lock.c:64
      #4 0xf76c0c7c in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:364
      #5 0xf7774442 in cb_cond_timedwait (cond=0xbf68a54, mutex=0xbf68a34, ms=2000) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:156
      #6 0xf2a6ff21 in SyncObject::wait (this=0xbf68a30, tv=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/syncobject.h:74
      #7 0xf2a6def7 in ExecutorPool::trySleep (this=0xbf68a20, t=..., now=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:168
      #8 0xf2a6dd89 in ExecutorPool::nextTask (this=0xbf68a20, t=..., tick=1 '\001') at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:144
      --Type <return> to continue, or q <return> to quit--
      #9 0xf2a7bb08 in ExecutorThread::run (this=0xbf4e820) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:77
      #10 0xf2a7b71c in launch_executor_thread (arg=0xbf4e820) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:33
      #11 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e320) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #12 0xf76bccf1 in start_thread (arg=0xecb4bb40) at pthread_create.c:311
      #13 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 23 (Thread 0xf4b71b40 (LWP 22019)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf75e8656 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
      #2 0xf76f20d0 in ?? () from /usr/lib/i386-linux-gnu/libevent_core-2.0.so.5
      #3 0xf76dbb93 in event_base_loop () from /usr/lib/i386-linux-gnu/libevent_core-2.0.so.5
      #4 0x0805ec59 in worker_libevent (arg=0xbf1e9a4) at /root/src/altoros/moxi/repo30/cmake/memcached/daemon/thread.c:285
      #5 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e0f0) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #6 0xf76bccf1 in start_thread (arg=0xf4b71b40) at pthread_create.c:311
      #7 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 22 (Thread 0xed34cb40 (LWP 22030)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf76c33a2 in __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:144
      #2 0xf76c62f0 in _L_cond_lock_773 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
      #3 0xf76c6061 in __pthread_mutex_cond_lock (mutex=0xbf68a34) at ../nptl/pthread_mutex_lock.c:64
      #4 0xf76c0c7c in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:364
      #5 0xf7774442 in cb_cond_timedwait (cond=0xbf68a54, mutex=0xbf68a34, ms=2000) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:156
      #6 0xf2a6ff21 in SyncObject::wait (this=0xbf68a30, tv=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/syncobject.h:74
      #7 0xf2a6def7 in ExecutorPool::trySleep (this=0xbf68a20, t=..., now=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:168
      #8 0xf2a6dd89 in ExecutorPool::nextTask (this=0xbf68a20, t=..., tick=1 '\001') at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:144
      #9 0xf2a7bb08 in ExecutorThread::run (this=0xbf4e870) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:77
      #10 0xf2a7b71c in launch_executor_thread (arg=0xbf4e870) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:33
      #11 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e328) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #12 0xf76bccf1 in start_thread (arg=0xed34cb40) at pthread_create.c:311
      #13 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 21 (Thread 0xebb49b40 (LWP 22033)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf76c33a2 in __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:144
      #2 0xf76bee24 in _L_lock_770 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
      #3 0xf76bec63 in _GI__pthread_mutex_lock (mutex=0xbf68ad0) at pthread_mutex_lock.c:64
      #4 0xf7774194 in cb_mutex_enter (mutex=0xbf68ad0) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:85
      #5 0xf2a7af04 in Mutex::acquire (this=0xbf68acc) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/mutex.cc:32
      #6 0xf2a1a659 in LockHolder::lock (this=0xebb49154) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/locks.h:66
      #7 0xf2a1a606 in LockHolder::LockHolder (this=0xebb49154, m=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/locks.h:44
      #8 0xf2a6e72a in ExecutorPool::snooze (this=0xbf68a20, taskId=8, tosleep=2) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:281
      #9 0xf2a1ccf4 in BgFetcher::run (this=0xbf84230, tid=8) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/bgfetcher.cc:157
      #10 0xf2a9e72a in BgFetcherTask::run (this=0xbf4e8c0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/tasks.cc:103
      #11 0xf2a7bc6a in ExecutorThread::run (this=0xbf4e780) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:93
      #12 0xf2a7b71c in launch_executor_thread (arg=0xbf4e780) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:33
      #13 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e310) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #14 0xf76bccf1 in start_thread (arg=0xebb49b40) at pthread_create.c:311
      #15 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 20 (Thread 0xf4370b40 (LWP 22020)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf75e8656 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
      #2 0xf76f20d0 in ?? () from /usr/lib/i386-linux-gnu/libevent_core-2.0.so.5
      #3 0xf76dbb93 in event_base_loop () from /usr/lib/i386-linux-gnu/libevent_core-2.0.so.5
      #4 0x0805ec59 in worker_libevent (arg=0xbf1ea30) at /root/src/altoros/moxi/repo30/cmake/memcached/daemon/thread.c:285
      #5 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e0e8) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #6 0xf76bccf1 in start_thread (arg=0xf4370b40) at pthread_create.c:311
      #7 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 19 (Thread 0xea346b40 (LWP 22036)):
      --Type <return> to continue, or q <return> to quit--
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf76c33a2 in __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:144
      #2 0xf76bee24 in _L_lock_770 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
      #3 0xf76bec63 in _GI__pthread_mutex_lock (mutex=0xbf68a34) at pthread_mutex_lock.c:64
      #4 0xf7774194 in cb_mutex_enter (mutex=0xbf68a34) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:85
      #5 0xf2a7af04 in Mutex::acquire (this=0xbf68a30) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/mutex.cc:32
      #6 0xf2a1a659 in LockHolder::lock (this=0xea346198) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/locks.h:66
      #7 0xf2a1a606 in LockHolder::LockHolder (this=0xea346198, m=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/locks.h:44
      #8 0xf2a6ddec in ExecutorPool::trySleep (this=0xbf68a20, t=..., now=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:153
      #9 0xf2a6dd89 in ExecutorPool::nextTask (this=0xbf68a20, t=..., tick=1 '\001') at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:144
      #10 0xf2a7bb08 in ExecutorThread::run (this=0xbf4e690) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:77
      #11 0xf2a7b71c in launch_executor_thread (arg=0xbf4e690) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:33
      #12 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e3d0) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #13 0xf76bccf1 in start_thread (arg=0xea346b40) at pthread_create.c:311
      #14 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 18 (Thread 0xe9b45b40 (LWP 22037)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf76c33a2 in __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:144
      #2 0xf76bee24 in _L_lock_770 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
      #3 0xf76bec63 in _GI__pthread_mutex_lock (mutex=0xbf700c4) at pthread_mutex_lock.c:64
      #4 0xf7774194 in cb_mutex_enter (mutex=0xbf700c4) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:85
      #5 0xf2a7af04 in Mutex::acquire (this=0xbf700c0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/mutex.cc:32
      #6 0xf2a1a659 in LockHolder::lock (this=0xe9b451e8) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/locks.h:66
      #7 0xf2a1a606 in LockHolder::LockHolder (this=0xe9b451e8, m=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/locks.h:44
      #8 0xf2a9fb8d in TaskQueue::reschedule (this=0xbf700c0, task=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/taskqueue.cc:139
      #9 0xf2a7bcf1 in ExecutorThread::run (this=0xbf4eaf0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:100
      #10 0xf2a7b71c in launch_executor_thread (arg=0xbf4eaf0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:33
      #11 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e3c8) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #12 0xf76bccf1 in start_thread (arg=0xe9b45b40) at pthread_create.c:311
      #13 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 17 (Thread 0xf3b6fb40 (LWP 22021)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf75e8656 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
      #2 0xf76f20d0 in ?? () from /usr/lib/i386-linux-gnu/libevent_core-2.0.so.5
      #3 0xf76dbb93 in event_base_loop () from /usr/lib/i386-linux-gnu/libevent_core-2.0.so.5
      #4 0x0805ec59 in worker_libevent (arg=0xbf1eabc) at /root/src/altoros/moxi/repo30/cmake/memcached/daemon/thread.c:285
      #5 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e0e0) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #6 0xf76bccf1 in start_thread (arg=0xf3b6fb40) at pthread_create.c:311
      #7 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 16 (Thread 0xf0b53b40 (LWP 22023)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf75b1896 in nanosleep () at ../sysdeps/unix/syscall-template.S:81
      #2 0xf75e080d in usleep (useconds=250000) at ../sysdeps/unix/sysv/linux/usleep.c:32
      #3 0xf2a79dc5 in updateStatsThread (arg=0x8e62500) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/memory_tracker.cc:36
      #4 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e188) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #5 0xf76bccf1 in start_thread (arg=0xf0b53b40) at pthread_create.c:311
      #6 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 15 (Thread 0xedb4db40 (LWP 22029)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf76c33a2 in __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:144
      #2 0xf76bee24 in _L_lock_770 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
      #3 0xf76bec63 in _GI__pthread_mutex_lock (mutex=0xbf68a34) at pthread_mutex_lock.c:64
      #4 0xf7774194 in cb_mutex_enter (mutex=0xbf68a34) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:85
      #5 0xf2a7af04 in Mutex::acquire (this=0xbf68a30) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/mutex.cc:32
      #6 0xf2a1a659 in LockHolder::lock (this=0xedb4d198) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/locks.h:66
      #7 0xf2a1a606 in LockHolder::LockHolder (this=0xedb4d198, m=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/locks.h:44
      #8 0xf2a6ddec in ExecutorPool::trySleep (this=0xbf68a20, t=..., now=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:153
      --Type <return> to continue, or q <return> to quit--
      #9 0xf2a6dd89 in ExecutorPool::nextTask (this=0xbf68a20, t=..., tick=2 '\002') at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:144
      #10 0xf2a7bb08 in ExecutorThread::run (this=0xbf4e4b0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:77
      #11 0xf2a7b71c in launch_executor_thread (arg=0xbf4e4b0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:33
      #12 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e330) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #13 0xf76bccf1 in start_thread (arg=0xedb4db40) at pthread_create.c:311
      #14 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 14 (Thread 0xf336eb40 (LWP 22022)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf75e8656 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
      #2 0xf76f20d0 in ?? () from /usr/lib/i386-linux-gnu/libevent_core-2.0.so.5
      #3 0xf76dbb93 in event_base_loop () from /usr/lib/i386-linux-gnu/libevent_core-2.0.so.5
      #4 0x0805ec59 in worker_libevent (arg=0xbf1eb48) at /root/src/altoros/moxi/repo30/cmake/memcached/daemon/thread.c:285
      #5 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e0d8) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #6 0xf76bccf1 in start_thread (arg=0xf336eb40) at pthread_create.c:311
      #7 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 13 (Thread 0xefb51b40 (LWP 22025)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf76c33a2 in __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:144
      #2 0xf76bee24 in _L_lock_770 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
      #3 0xf76bec63 in _GI__pthread_mutex_lock (mutex=0xbf700c4) at pthread_mutex_lock.c:64
      #4 0xf7774194 in cb_mutex_enter (mutex=0xbf700c4) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:85
      #5 0xf2a7af04 in Mutex::acquire (this=0xbf700c0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/mutex.cc:32
      #6 0xf2a1a659 in LockHolder::lock (this=0xefb511e8) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/locks.h:66
      #7 0xf2a1a606 in LockHolder::LockHolder (this=0xefb511e8, m=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/locks.h:44
      #8 0xf2a9fb8d in TaskQueue::reschedule (this=0xbf700c0, task=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/taskqueue.cc:139
      #9 0xf2a7bcf1 in ExecutorThread::run (this=0xbf4e5f0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:100
      #10 0xf2a7b71c in launch_executor_thread (arg=0xbf4e5f0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:33
      #11 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e358) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #12 0xf76bccf1 in start_thread (arg=0xefb51b40) at pthread_create.c:311
      #13 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 12 (Thread 0xe8b43b40 (LWP 22039)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf76c0ad3 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:124
      #2 0xf7774442 in cb_cond_timedwait (cond=0xbf68a54, mutex=0xbf68a34, ms=189) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:156
      #3 0xf2a6ff21 in SyncObject::wait (this=0xbf68a30, tv=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/syncobject.h:74
      #4 0xf2a6df11 in ExecutorPool::trySleep (this=0xbf68a20, t=..., now=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:170
      #5 0xf2a6dd89 in ExecutorPool::nextTask (this=0xbf68a20, t=..., tick=4 '\004') at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:144
      #6 0xf2a7bb08 in ExecutorThread::run (this=0xbf4ea50) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:77
      #7 0xf2a7b71c in launch_executor_thread (arg=0xbf4ea50) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:33
      #8 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e3b8) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #9 0xf76bccf1 in start_thread (arg=0xe8b43b40) at pthread_create.c:311
      #10 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 11 (Thread 0xf0352b40 (LWP 22024)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf76c33a2 in __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:144
      #2 0xf76c62f0 in _L_cond_lock_773 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
      #3 0xf76c6061 in __pthread_mutex_cond_lock (mutex=0xbf68a34) at ../nptl/pthread_mutex_lock.c:64
      #4 0xf76c0c7c in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:364
      #5 0xf7774442 in cb_cond_timedwait (cond=0xbf68a54, mutex=0xbf68a34, ms=2000) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:156
      #6 0xf2a6ff21 in SyncObject::wait (this=0xbf68a30, tv=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/syncobject.h:74
      #7 0xf2a6def7 in ExecutorPool::trySleep (this=0xbf68a20, t=..., now=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:168
      #8 0xf2a6dd89 in ExecutorPool::nextTask (this=0xbf68a20, t=..., tick=2 '\002') at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:144
      #9 0xf2a7bb08 in ExecutorThread::run (this=0xbf4e640) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:77
      #10 0xf2a7b71c in launch_executor_thread (arg=0xbf4e640) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:33
      #11 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e350) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #12 0xf76bccf1 in start_thread (arg=0xf0352b40) at pthread_create.c:311
      #13 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131
      --Type <return> to continue, or q <return> to quit--

      Thread 10 (Thread 0xeab47b40 (LWP 22035)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf76c3462 in __lll_unlock_wake () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:386
      #2 0xf76c0d20 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:510
      #3 0xf7774442 in cb_cond_timedwait (cond=0xbf68a54, mutex=0xbf68a34, ms=2000) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:156
      #4 0xf2a6ff21 in SyncObject::wait (this=0xbf68a30, tv=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/syncobject.h:74
      #5 0xf2a6def7 in ExecutorPool::trySleep (this=0xbf68a20, t=..., now=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:168
      #6 0xf2a6dd89 in ExecutorPool::nextTask (this=0xbf68a20, t=..., tick=1 '\001') at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:144
      #7 0xf2a7bb08 in ExecutorThread::run (this=0xbf4e6e0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:77
      #8 0xf2a7b71c in launch_executor_thread (arg=0xbf4e6e0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:33
      #9 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e300) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #10 0xf76bccf1 in start_thread (arg=0xeab47b40) at pthread_create.c:311
      #11 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 9 (Thread 0xef350b40 (LWP 22026)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf76c33a2 in __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:144
      #2 0xf76c62f0 in _L_cond_lock_773 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
      #3 0xf76c6061 in __pthread_mutex_cond_lock (mutex=0xbf68a34) at ../nptl/pthread_mutex_lock.c:64
      #4 0xf76c0c7c in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:364
      #5 0xf7774442 in cb_cond_timedwait (cond=0xbf68a54, mutex=0xbf68a34, ms=2000) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:156
      #6 0xf2a6ff21 in SyncObject::wait (this=0xbf68a30, tv=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/syncobject.h:74
      #7 0xf2a6def7 in ExecutorPool::trySleep (this=0xbf68a20, t=..., now=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:168
      #8 0xf2a6dd89 in ExecutorPool::nextTask (this=0xbf68a20, t=..., tick=4 '\004') at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:144
      #9 0xf2a7bb08 in ExecutorThread::run (this=0xbf4e5a0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:77
      #10 0xf2a7b71c in launch_executor_thread (arg=0xbf4e5a0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:33
      #11 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e348) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #12 0xf76bccf1 in start_thread (arg=0xef350b40) at pthread_create.c:311
      #13 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 8 (Thread 0xeb348b40 (LWP 22034)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf76c33a2 in __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:144
      #2 0xf76bee24 in _L_lock_770 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
      #3 0xf76bec63 in _GI__pthread_mutex_lock (mutex=0xbf68a34) at pthread_mutex_lock.c:64
      #4 0xf7774194 in cb_mutex_enter (mutex=0xbf68a34) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:85
      #5 0xf2a7af04 in Mutex::acquire (this=0xbf68a30) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/mutex.cc:32
      #6 0xf2a1a659 in LockHolder::lock (this=0xeb348198) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/locks.h:66
      #7 0xf2a1a606 in LockHolder::LockHolder (this=0xeb348198, m=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/locks.h:44
      #8 0xf2a6ddec in ExecutorPool::trySleep (this=0xbf68a20, t=..., now=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:153
      #9 0xf2a6dd89 in ExecutorPool::nextTask (this=0xbf68a20, t=..., tick=1 '\001') at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorpool.cc:144
      #10 0xf2a7bb08 in ExecutorThread::run (this=0xbf4e730) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:77
      #11 0xf2a7b71c in launch_executor_thread (arg=0xbf4e730) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:33
      #12 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e308) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #13 0xf76bccf1 in start_thread (arg=0xeb348b40) at pthread_create.c:311
      #14 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 7 (Thread 0xf5372b40 (LWP 22018)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf75e8656 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
      #2 0xf76f20d0 in ?? () from /usr/lib/i386-linux-gnu/libevent_core-2.0.so.5
      #3 0xf76dbb93 in event_base_loop () from /usr/lib/i386-linux-gnu/libevent_core-2.0.so.5
      #4 0x0805ec59 in worker_libevent (arg=0xbf1e918) at /root/src/altoros/moxi/repo30/cmake/memcached/daemon/thread.c:285
      #5 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e0f8) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #6 0xf76bccf1 in start_thread (arg=0xf5372b40) at pthread_create.c:311
      #7 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 6 (Thread 0xf6374b40 (LWP 22016)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf76c07ab in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_wait.S:187
      --Type <return> to continue, or q <return> to quit--
      #2 0xf77742de in cb_cond_wait (cond=0xf6373aac, mutex=0xf6373a8c) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:119
      #3 0xf2a3c864 in SyncObject::wait (this=0xf6373a88) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/syncobject.h:39
      #4 0xf2a40c10 in WarmupWaitListener::wait (this=0xf6373a7c) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/ep.cc:301
      #5 0xf2a30d66 in EventuallyPersistentStore::initialize (this=0x92a42c0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/ep.cc:331
      #6 0xf2a503e1 in EventuallyPersistentEngine::initialize (this=0xbf7c000,
      config=0xbf5603a "ht_size=3079;ht_locks=5;tap_noop_interval=20;max_txn_size=10000;max_size=943718400;tap_keepalive=300;dbname=/root/src/altoros/moxi/ns_server/data/n_1/data/default;allow_data_loss_during_shutdown=true;"...)
      at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/ep_engine.cc:1792
      #7 0xf2a4b534 in EvpInitialize (handle=0xbf7c000,
      config_str=0xbf5603a "ht_size=3079;ht_locks=5;tap_noop_interval=20;max_txn_size=10000;max_size=943718400;tap_keepalive=300;dbname=/root/src/altoros/moxi/ns_server/data/n_1/data/default;allow_data_loss_during_shutdown=true;"...)
      at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/ep_engine.cc:129
      #8 0xf7746e6a in create_bucket_UNLOCKED (e=0xf77517c0 <bucket_engine>, bucket_name=0x8e5e120 "default", path=0xbf56000 "/root/src/altoros/moxi/repo30/install/lib/memcached/ep.so",
      config=0xbf5603a "ht_size=3079;ht_locks=5;tap_noop_interval=20;max_txn_size=10000;max_size=943718400;tap_keepalive=300;dbname=/root/src/altoros/moxi/ns_server/data/n_1/data/default;allow_data_loss_during_shutdown=true;"...,
      e_out=0x0, msg=0xf6373c4c "", msglen=1024) at /root/src/altoros/moxi/repo30/cmake/memcached/engines/bucket_engine/bucket_engine.c:812
      #9 0xf774b156 in handle_create_bucket (handle=0xf77517c0 <bucket_engine>, cookie=0xbec8000, request=0xbf0e800, response=0x8050b2b <binary_response_handler>)
      at /root/src/altoros/moxi/repo30/cmake/memcached/engines/bucket_engine/bucket_engine.c:2676
      #10 0xf774c232 in bucket_unknown_command (handle=0xf77517c0 <bucket_engine>, cookie=0xbec8000, request=0xbf0e800, response=0x8050b2b <binary_response_handler>)
      at /root/src/altoros/moxi/repo30/cmake/memcached/engines/bucket_engine/bucket_engine.c:3016
      #11 0x08051ab9 in default_unknown_command (descriptor=0x0, handle=0xf77517c0 <bucket_engine>, cookie=0xbec8000, request=0xbf0e800, response=0x8050b2b <binary_response_handler>)
      at /root/src/altoros/moxi/repo30/cmake/memcached/daemon/memcached.c:2704
      #12 0x08051b7a in process_bin_unknown_packet (c=0xbec8000) at /root/src/altoros/moxi/repo30/cmake/memcached/daemon/memcached.c:2737
      #13 0x08055923 in process_bin_packet (c=0xbec8000) at /root/src/altoros/moxi/repo30/cmake/memcached/daemon/memcached.c:4397
      #14 0x08056e35 in complete_nread (c=0xbec8000) at /root/src/altoros/moxi/repo30/cmake/memcached/daemon/memcached.c:4969
      #15 0x08059896 in conn_nread (c=0xbec8000) at /root/src/altoros/moxi/repo30/cmake/memcached/daemon/memcached.c:5842
      #16 0x0805a315 in event_handler (fd=47, which=2, arg=0xbec8000) at /root/src/altoros/moxi/repo30/cmake/memcached/daemon/memcached.c:6098
      #17 0xf76dbe52 in event_base_loop () from /usr/lib/i386-linux-gnu/libevent_core-2.0.so.5
      #18 0x0805ec59 in worker_libevent (arg=0xbf1e800) at /root/src/altoros/moxi/repo30/cmake/memcached/daemon/thread.c:285
      #19 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e108) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #20 0xf76bccf1 in start_thread (arg=0xf6374b40) at pthread_create.c:311
      #21 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 5 (Thread 0xf73abb40 (LWP 22014)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf75d74bb in read () at ../sysdeps/unix/syscall-template.S:81
      #2 0xf756a52b in IO_new_file_underflow (fp=0xf76a4ac0 <_IO_2_1_stdin>) at fileops.c:613
      #3 0xf756b459 in _GIIO_default_uflow (fp=0xf76a4ac0 <_IO_2_1_stdin>) at genops.c:436
      #4 0xf756b28b in _GI_uflow (fp=fp@entry=0xf76a4ac0 <_IO_2_1_stdin>) at genops.c:390
      #5 0xf755ee8a in _GIIO_getline_info (fp=fp@entry=0xf76a4ac0 <_IO_2_1_stdin>, buf=buf@entry=0xf73ab2a0 "", n=n@entry=79, delim=delim@entry=10, extract_delim=extract_delim@entry=1, eof=eof@entry=0x0) at iogetline.c:69
      #6 0xf755efc7 in _GIIO_getline (fp=fp@entry=0xf76a4ac0 <_IO_2_1_stdin>, buf=buf@entry=0xf73ab2a0 "", n=n@entry=79, delim=delim@entry=10, extract_delim=extract_delim@entry=1) at iogetline.c:38
      #7 0xf755decf in IO_fgets (buf=0xf73ab2a0 "", n=80, fp=0xf76a4ac0 <_IO_2_1_stdin>) at iofgets.c:56
      #8 0xf77708b8 in check_stdin_thread (arg=0x805c552 <shutdown_server>) at /root/src/altoros/moxi/repo30/cmake/memcached/extensions/daemon/stdin_check.c:38
      #9 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e018) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #10 0xf76bccf1 in start_thread (arg=0xf73abb40) at pthread_create.c:311
      #11 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 4 (Thread 0xf6baab40 (LWP 22015)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf76c3462 in __lll_unlock_wake () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/lowlevellock.S:386
      #2 0xf76bfc57 in _L_unlock_670 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
      #3 0xf76bfb9a in __pthread_mutex_unlock_usercnt (mutex=0xf776f83c <mutex>, decr=0) at pthread_mutex_unlock.c:52
      #4 0xf76c0a83 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:89
      #5 0xf7774442 in cb_cond_timedwait (cond=0xf776f860 <cond>, mutex=0xf776f83c <mutex>, ms=19000) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:156
      #6 0xf776d74a in logger_thead_main (arg=0x8e62120) at /root/src/altoros/moxi/repo30/cmake/memcached/extensions/loggers/file_logger.c:363
      #7 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e010) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #8 0xf76bccf1 in start_thread (arg=0xf6baab40) at pthread_create.c:311
      #9 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 3 (Thread 0xf73ad700 (LWP 22012)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf75e8656 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
      #2 0xf76f20d0 in ?? () from /usr/lib/i386-linux-gnu/libevent_core-2.0.so.5
      #3 0xf76dbb93 in event_base_loop () from /usr/lib/i386-linux-gnu/libevent_core-2.0.so.5
      --Type <return> to continue, or q <return> to quit--
      #4 0x0805dfe4 in main (argc=21, argv=0xffe5b974) at /root/src/altoros/moxi/repo30/cmake/memcached/daemon/memcached.c:8050

      Thread 2 (Thread 0xf5b73b40 (LWP 22017)):
      #0 0xf7785430 in __kernel_vsyscall ()
      #1 0xf75e8656 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
      #2 0xf76f20d0 in ?? () from /usr/lib/i386-linux-gnu/libevent_core-2.0.so.5
      #3 0xf76dbb93 in event_base_loop () from /usr/lib/i386-linux-gnu/libevent_core-2.0.so.5
      #4 0x0805ec59 in worker_libevent (arg=0xbf1e88c) at /root/src/altoros/moxi/repo30/cmake/memcached/daemon/thread.c:285
      #5 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e100) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #6 0xf76bccf1 in start_thread (arg=0xf5b73b40) at pthread_create.c:311
      #7 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131

      Thread 1 (Thread 0xeeb4fb40 (LWP 22027)):
      #0 0xf29ae56a in cJSON_GetObjectItem (object=0x0, string=0xf2b0187b "state") at /root/src/altoros/moxi/repo30/cmake/libvbucket/src/cJSON.c:452
      #1 0xf2acf29c in CouchKVStore::readVBState (db=0x8e72480, vbId=0, vbState=...) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/couch-kvstore/couch-kvstore.cc:1812
      #2 0xf2ac8cf3 in CouchKVStore::listPersistedVbuckets (this=0xbfae1a0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/couch-kvstore/couch-kvstore.cc:551
      #3 0xf2a3ac4e in EventuallyPersistentStore::loadVBucketState (this=0x92a42c0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/ep.cc:2636
      #4 0xf2ab5604 in Warmup::initialize (this=0xbf6c1b0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/warmup.cc:439
      #5 0xf2ab8230 in WarmupInitialize::run (this=0xbf4edc0) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/warmup.h:220
      #6 0xf2a7bc6a in ExecutorThread::run (this=0xbf4e550) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:93
      #7 0xf2a7b71c in launch_executor_thread (arg=0xbf4e550) at /root/src/altoros/moxi/repo30/cmake/ep-engine/src/executorthread.cc:33
      #8 0xf7773fc3 in platform_thread_wrap (arg=0x8e5e340) at /root/src/altoros/moxi/repo30/cmake/platform/src/cb_pthreads.c:19
      #9 0xf76bccf1 in start_thread (arg=0xeeb4fb40) at pthread_create.c:311
      #10 0xf75e7c3e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:131
      (gdb)

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            abhinav Abhi Dangeti
            alkondratenko Aleksey Kondratenko (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty