Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-9237

create bucket with 3 replica cause couchbase server hang

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 3.0
    • 3.0
    • couchbase-bucket
    • Security Level: Public
    • ubuntu 12.04 64bit
    • Ubuntu 64-bit

    Description

      Install couchbase server 3.0.0-25 on one ubuntu 12.04 64-bit
      Create buckets with 1 replica ==> ok
      Create buckets with 2 replica ==> ok
      Create a bucket with 3 replica ==> failed
      Couchbase server hang made all other buckets hang

      Server is in hang status with IP 10.3.2.199 (Sundar has credential to access this server)
      Link to manifest file of this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_3.0.0-25-rel.deb.manifest.xml

      Below is backtrace from memcached of this node

      GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
      Copyright (C) 2012 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law. Type "show copying"
      and "show warranty" for details.
      This GDB was configured as "x86_64-linux-gnu".
      For bug reporting instructions, please see:
      <http://bugs.launchpad.net/gdb-linaro/>.
      Attaching to process 3782
      Reading symbols from /opt/couchbase/bin/memcached...done.
      Reading symbols from /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0...done.
      Loaded symbols for /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0
      Reading symbols from /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1...done.
      Loaded symbols for /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1
      Reading symbols from /opt/couchbase/bin/../lib/libplatform.so.0.1.0...done.
      Loaded symbols for /opt/couchbase/bin/../lib/libplatform.so.0.1.0
      Reading symbols from /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4...done.
      Loaded symbols for /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4
      Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5...done.
      Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.0.so.5
      Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...(no debugging symbols found)...done.
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
      [New Thread 0x7f45c12f0700 (LWP 3925)]
      [New Thread 0x7f45c1af1700 (LWP 3924)]
      [New Thread 0x7f45c22f2700 (LWP 3923)]
      [New Thread 0x7f45c2af3700 (LWP 3922)]
      [New Thread 0x7f45c32f4700 (LWP 3921)]
      [New Thread 0x7f45c3af5700 (LWP 3920)]
      [New Thread 0x7f45c42f6700 (LWP 3919)]
      [New Thread 0x7f45c4af7700 (LWP 3918)]
      [New Thread 0x7f45c52f8700 (LWP 3917)]
      [New Thread 0x7f45c5af9700 (LWP 3916)]
      [New Thread 0x7f45c62fa700 (LWP 3915)]
      [New Thread 0x7f45c6afb700 (LWP 3895)]
      [New Thread 0x7f45c72fc700 (LWP 3894)]
      [New Thread 0x7f45c7afd700 (LWP 3893)]
      [New Thread 0x7f45c82fe700 (LWP 3892)]
      [New Thread 0x7f45c8aff700 (LWP 3891)]
      [New Thread 0x7f45c9300700 (LWP 3890)]
      [New Thread 0x7f45c9b01700 (LWP 3889)]
      [New Thread 0x7f45ca302700 (LWP 3888)]
      [New Thread 0x7f45cab03700 (LWP 3887)]
      [New Thread 0x7f45cb304700 (LWP 3886)]
      [New Thread 0x7f45cbb05700 (LWP 3885)]
      [New Thread 0x7f45cc306700 (LWP 3864)]
      [New Thread 0x7f45ccb07700 (LWP 3863)]
      [New Thread 0x7f45cd308700 (LWP 3862)]
      [New Thread 0x7f45cdb09700 (LWP 3861)]
      [New Thread 0x7f45ce30a700 (LWP 3860)]
      [New Thread 0x7f45ceb0b700 (LWP 3859)]
      [New Thread 0x7f45cf30c700 (LWP 3858)]
      [New Thread 0x7f45cfb0d700 (LWP 3857)]
      [New Thread 0x7f45d030e700 (LWP 3856)]
      [New Thread 0x7f45d0b0f700 (LWP 3855)]
      [New Thread 0x7f45d1310700 (LWP 3854)]
      [New Thread 0x7f45d1b11700 (LWP 3833)]
      [New Thread 0x7f45d2312700 (LWP 3832)]
      [New Thread 0x7f45d2b13700 (LWP 3831)]
      [New Thread 0x7f45d3314700 (LWP 3830)]
      [New Thread 0x7f45d3b15700 (LWP 3829)]
      [New Thread 0x7f45d4316700 (LWP 3828)]
      [New Thread 0x7f45d4b17700 (LWP 3827)]
      [New Thread 0x7f45d5318700 (LWP 3826)]
      [New Thread 0x7f45d5b19700 (LWP 3825)]
      [New Thread 0x7f45d631a700 (LWP 3824)]
      [New Thread 0x7f45d6b1b700 (LWP 3823)]
      [New Thread 0x7f45d9bd5700 (LWP 3822)]
      [New Thread 0x7f45da3d6700 (LWP 3797)]
      [New Thread 0x7f45dabd7700 (LWP 3796)]
      [New Thread 0x7f45db3d8700 (LWP 3795)]
      [New Thread 0x7f45dbbd9700 (LWP 3794)]
      [New Thread 0x7f45dc3da700 (LWP 3793)]
      [New Thread 0x7f45dcde8700 (LWP 3791)]
      [New Thread 0x7f45dda04700 (LWP 3790)]
      Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
      Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
      Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib/x86_64-linux-gnu/librt.so.1
      Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done.
      Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
      Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
      Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
      Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done.
      Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
      Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
      Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib64/ld-linux-x86-64.so.2
      Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
      Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
      Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
      Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
      Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/ep.so
      Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
      Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
      Reading symbols from /opt/couchbase/lib/libv8.so...done.
      Loaded symbols for /opt/couchbase/lib/libv8.so
      Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
      Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
      Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
      Loaded symbols for /opt/couchbase/lib/libicudata.so.44
      Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
      Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
      Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
      Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
      Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
      Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
      0x00007f45de50d363 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
      (gdb) thread apply all bt

      Thread 53 (Thread 0x7f45dda04700 (LWP 3790)):
      #0 0x00007f45de4ff8cd in read () from /lib/x86_64-linux-gnu/libc.so.6
      #1 0x00007f45de493ff8 in _IO_file_underflow () from /lib/x86_64-linux-gnu/libc.so.6
      #2 0x00007f45de49503e in _IO_default_uflow () from /lib/x86_64-linux-gnu/libc.so.6
      #3 0x00007f45de48918a in _IO_getline_info () from /lib/x86_64-linux-gnu/libc.so.6
      #4 0x00007f45de48806b in fgets () from /lib/x86_64-linux-gnu/libc.so.6
      #5 0x00007f45dda05aa1 in fgets (__stream=<optimized out>, __n=<optimized out>, __s=<optimized out>) at /usr/include/bits/stdio2.h:255
      #6 check_stdin_thread (arg=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/extensions/daemon/stdin_check.c:38
      #7 0x00007f45df2679bf in platform_thread_wrap (arg=0xec0020) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:18
      #8 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #9 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #10 0x0000000000000000 in ?? ()

      Thread 52 (Thread 0x7f45dcde8700 (LWP 3791)):
      #0 0x00007f45debf00fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45df267a63 in cb_cond_timedwait (cond=0x7f45dd203200, mutex=0x7f45dd2031c0, ms=<optimized out>)
      at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:133
      #2 0x00007f45dd00204d in logger_thead_main (arg=<optimized out>)
      at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/extensions/loggers/file_logger.c:363
      #3 0x00007f45df2679bf in platform_thread_wrap (arg=0xec0010) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:18
      #4 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #5 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #6 0x0000000000000000 in ?? ()

      Thread 51 (Thread 0x7f45dc3da700 (LWP 3793)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45df267819 in cb_mutex_enter (mutex=0x7f45dc5e73c0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:79
      #4 0x00007f45dc3dd8a0 in lock_engines () at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/engines/bucket_engine/bucket_engine.c:301
      #5 find_bucket (name=0xec1ad0 "bucket-2") at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/engines/bucket_engine/bucket_engine.c:639
      #6 0x00007f45dc3e1e3a in handle_select_bucket (response=<optimized out>, request=<optimized out>, cookie=<optimized out>, handle=<optimized out>)
      at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/engines/bucket_engine/bucket_engine.c:2778
      #7 bucket_unknown_command (handle=0x7f45dc5e7220, cookie=0x569fb80, request=0x56a3800, response=0x4078e0 <binary_response_handler>)
      at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/engines/bucket_engine/bucket_engine.c:2918
      #8 0x00000000004142dc in process_bin_unknown_packet (c=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:2611
      #9 process_bin_packet (c=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:4055
      #10 complete_nread (c=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:4620
      #11 conn_nread (c=0x569fb80) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:5493
      #12 0x00000000004076cd in event_handler (fd=<optimized out>, which=<optimized out>, arg=0x569fb80)
      at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:5749
      #13 0x00007f45dee0eafc in event_process_active_single_queue (activeq=<optimized out>, base=<optimized out>) at event.c:1308
      #14 event_process_active (base=<optimized out>) at event.c:1375
      #15 event_base_loop (base=0x569e500, flags=<optimized out>) at event.c:1572
      #16 0x00007f45df2679bf in platform_thread_wrap (arg=0xec0140) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:18
      #17 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #18 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #19 0x0000000000000000 in ?? ()

      Thread 50 (Thread 0x7f45dbbd9700 (LWP 3794)):
      #0 0x00007f45de50d363 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
      #1 0x00007f45dee235a6 in epoll_dispatch (base=0x569e280, tv=<optimized out>) at epoll.c:404
      #2 0x00007f45dee0ea04 in event_base_loop (base=0x569e280, flags=<optimized out>) at event.c:1558
      --Type <return> to continue, or q <return> to quit--
      #3 0x00007f45df2679bf in platform_thread_wrap (arg=0xec0130) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:18
      #4 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #5 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #6 0x0000000000000000 in ?? ()

      Thread 49 (Thread 0x7f45db3d8700 (LWP 3795)):
      #0 ExecutorPool::startWorkers (this=<optimized out>, workload=...) at src/scheduler.cc:550
      #1 0x00007f45d913c9c1 in ExecutorPool::registerBucket (this=0x578ba00, engine=0x572b200) at src/scheduler.cc:528
      #2 0x00007f45d910e43c in EventuallyPersistentStore::EventuallyPersistentStore (this=0xbef9200, theEngine=...) at src/ep.cc:153
      #3 0x00007f45d9118f6f in EventuallyPersistentEngine::initialize (this=0x572b200, config=<optimized out>) at src/ep_engine.cc:1355
      #4 0x00007f45d9119256 in EvpInitialize (handle=0x572b200,
      config_str=0x56f1823 "ht_size=3079;ht_locks=5;tap_noop_interval=20;max_txn_size=10000;max_size=524288000;tap_keepalive=300;dbname=/opt/couchbase/var/lib/couchbase/data/bucket-3;allow_data_loss_during_shutdown=true;backend="...) at src/ep_engine.cc:127
      #5 0x00007f45dc3e183a in create_bucket_UNLOCKED (e=<optimized out>, bucket_name=0xec05d0 "bucket-3", path=0x56f1800 "/opt/couchbase/lib/memcached/ep.so",
      config=<optimized out>, e_out=<optimized out>, msg=0x7f45db3d77f0 "", msglen=1024)
      at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/engines/bucket_engine/bucket_engine.c:781
      #6 0x00007f45dc3e1aae in handle_create_bucket (handle=0x7f45dc5e7220, cookie=0x564a500, request=<optimized out>, response=0x4078e0 <binary_response_handler>)
      at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/engines/bucket_engine/bucket_engine.c:2567
      #7 0x00007f45dc3e1f69 in bucket_unknown_command (handle=0x7f45dc5e7220, cookie=0x564a500, request=0x5670800, response=0x4078e0 <binary_response_handler>)
      at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/engines/bucket_engine/bucket_engine.c:2906
      #8 0x00000000004142dc in process_bin_unknown_packet (c=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:2611
      #9 process_bin_packet (c=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:4055
      #10 complete_nread (c=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:4620
      #11 conn_nread (c=0x564a500) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:5493
      #12 0x00000000004076cd in event_handler (fd=<optimized out>, which=<optimized out>, arg=0x564a500)
      at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:5749
      #13 0x00007f45dee0eafc in event_process_active_single_queue (activeq=<optimized out>, base=<optimized out>) at event.c:1308
      #14 event_process_active (base=<optimized out>) at event.c:1375
      #15 event_base_loop (base=0x569e000, flags=<optimized out>) at event.c:1572
      #16 0x00007f45df2679bf in platform_thread_wrap (arg=0xec0120) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:18
      #17 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #18 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #19 0x0000000000000000 in ?? ()

      Thread 48 (Thread 0x7f45dabd7700 (LWP 3796)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45df267819 in cb_mutex_enter (mutex=0x7f45dc5e73c0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:79
      #4 0x00007f45dc3dd8a0 in lock_engines () at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/engines/bucket_engine/bucket_engine.c:301
      #5 find_bucket (name=0xebe040 "default") at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/engines/bucket_engine/bucket_engine.c:639
      #6 0x00007f45dc3e2a72 in handle_connect (cookie=0x564a780, type=128, event_data=0x0, cb_data=0xffffffffffffffff)
      at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/engines/bucket_engine/bucket_engine.c:1201
      #7 0x0000000000408713 in perform_callbacks (c=<optimized out>, data=<optimized out>, type=<optimized out>)
      at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:242
      #8 conn_new (sfd=68, parent_port=11209, init_state=<optimized out>, event_flags=18, read_buffer_size=<optimized out>, base=0x56f3b80, timeout=0x0)
      at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:920
      #9 0x00000000004153d9 in thread_libevent_process (fd=<optimized out>, which=<optimized out>, arg=<optimized out>)
      at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/thread.c:320
      #10 0x00007f45dee0eafc in event_process_active_single_queue (activeq=<optimized out>, base=<optimized out>) at event.c:1308
      #11 event_process_active (base=<optimized out>) at event.c:1375
      #12 event_base_loop (base=0x56f3b80, flags=<optimized out>) at event.c:1572
      #13 0x00007f45df2679bf in platform_thread_wrap (arg=0xec0110) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:18
      #14 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      --Type <return> to continue, or q <return> to quit--
      #15 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #16 0x0000000000000000 in ?? ()

      Thread 47 (Thread 0x7f45da3d6700 (LWP 3797)):
      #0 0x00007f45de50d363 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
      #1 0x00007f45dee235a6 in epoll_dispatch (base=0x56f3900, tv=<optimized out>) at epoll.c:404
      #2 0x00007f45dee0ea04 in event_base_loop (base=0x56f3900, flags=<optimized out>) at event.c:1558
      #3 0x00007f45df2679bf in platform_thread_wrap (arg=0xec0100) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:18
      #4 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #5 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #6 0x0000000000000000 in ?? ()

      Thread 46 (Thread 0x7f45d9bd5700 (LWP 3822)):
      #0 0x00007f45de4d884d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
      #1 0x00007f45de506784 in usleep () from /lib/x86_64-linux-gnu/libc.so.6
      #2 0x00007f45d9137bd5 in updateStatsThread (arg=<optimized out>) at src/memory_tracker.cc:36
      #3 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #4 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #5 0x0000000000000000 in ?? ()

      Thread 45 (Thread 0x7f45d6b1b700 (LWP 3823)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=1) at src/scheduler.cc:437
      #7 0x00007f45d90f0729 in BgFetcher::run (this=0x5710480, tid=<optimized out>) at src/bgfetcher.cc:163
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x56b7540) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 44 (Thread 0x7f45d631a700 (LWP 3824)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=1) at src/scheduler.cc:437
      #7 0x00007f45d90f0729 in BgFetcher::run (this=0x5710120, tid=<optimized out>) at src/bgfetcher.cc:163
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x56b74a0) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 43 (Thread 0x7f45d5b19700 (LWP 3825)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      --Type <return> to continue, or q <return> to quit--
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=1) at src/scheduler.cc:437
      #7 0x00007f45d90f0729 in BgFetcher::run (this=0x57105a0, tid=<optimized out>) at src/bgfetcher.cc:163
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x56b7720) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 42 (Thread 0x7f45d5318700 (LWP 3826)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=60) at src/scheduler.cc:437
      #7 0x00007f45d915f114 in StatSnap::run (this=<optimized out>) at src/tasks.cc:96
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x56b7680) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 41 (Thread 0x7f45d4b17700 (LWP 3827)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=1) at src/scheduler.cc:437
      #7 0x00007f45d90f0729 in BgFetcher::run (this=0x5711440, tid=<optimized out>) at src/bgfetcher.cc:163
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x56b75e0) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 40 (Thread 0x7f45d4316700 (LWP 3828)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=1) at src/scheduler.cc:437
      #7 0x00007f45d90f0729 in BgFetcher::run (this=0x5710c60, tid=<optimized out>) at src/bgfetcher.cc:163
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x56b79a0) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      --Type <return> to continue, or q <return> to quit--
      #12 0x0000000000000000 in ?? ()

      Thread 39 (Thread 0x7f45d3b15700 (LWP 3829)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=1) at src/scheduler.cc:437
      #7 0x00007f45d90f0729 in BgFetcher::run (this=0x5711320, tid=<optimized out>) at src/bgfetcher.cc:163
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x56b7900) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 38 (Thread 0x7f45d3314700 (LWP 3830)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=1) at src/scheduler.cc:437
      #7 0x00007f45d90f0729 in BgFetcher::run (this=0x5710360, tid=<optimized out>) at src/bgfetcher.cc:163
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x56b7860) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 37 (Thread 0x7f45d2b13700 (LWP 3831)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=1) at src/scheduler.cc:437
      #7 0x00007f45d90f0729 in BgFetcher::run (this=0x57110e0, tid=<optimized out>) at src/bgfetcher.cc:163
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x56b77c0) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 36 (Thread 0x7f45d2312700 (LWP 3832)):
      #0 0x00007f45debf00fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45d91186e2 in wait (tv=..., this=<optimized out>) at src/syncobject.h:77
      #2 wait (secs=<optimized out>, this=<optimized out>) at src/syncobject.h:93
      #3 wait (previousCounter=<optimized out>, howlong=<optimized out>, this=<optimized out>) at src/tapconnmap.h:225
      #4 EventuallyPersistentEngine::notifyPendingConnections (this=0x5728000) at src/ep_engine.cc:3383
      #5 0x00007f45d91187d3 in EvpNotifyPendingConns (arg=0x5728000) at src/ep_engine.cc:1139
      --Type <return> to continue, or q <return> to quit--
      #6 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #7 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 35 (Thread 0x7f45d1b11700 (LWP 3833)):
      #0 0x00007f45debf00fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45d90fbfb8 in wait (tv=..., this=<optimized out>) at src/syncobject.h:77
      #2 IdleTask::run (this=0xed03f0, d=...) at src/dispatcher.cc:344
      #3 0x00007f45d90feb6a in Dispatcher::run (this=0x5708c40) at src/dispatcher.cc:186
      #4 0x00007f45d90ff33d in launch_dispatcher_thread (arg=0x5708c94) at src/dispatcher.cc:30
      #5 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #6 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #7 0x0000000000000000 in ?? ()

      Thread 34 (Thread 0x7f45d1310700 (LWP 3854)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=0.20000000000000001) at src/scheduler.cc:437
      #7 0x00007f45d913186e in Flusher::step (this=0x57bde00, tid=2047) at src/flusher.cc:165
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x6370b40) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 33 (Thread 0x7f45d0b0f700 (LWP 3855)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=1) at src/scheduler.cc:437
      #7 0x00007f45d90f0729 in BgFetcher::run (this=0x5711680, tid=<optimized out>) at src/bgfetcher.cc:163
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x6370aa0) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 32 (Thread 0x7f45d030e700 (LWP 3856)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=1) at src/scheduler.cc:437
      #7 0x00007f45d90f0729 in BgFetcher::run (this=0x57117a0, tid=<optimized out>) at src/bgfetcher.cc:163
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x6370a00) at src/scheduler.cc:97
      --Type <return> to continue, or q <return> to quit--
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 31 (Thread 0x7f45cfb0d700 (LWP 3857)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=0.20000000000000001) at src/scheduler.cc:437
      #7 0x00007f45d913186e in Flusher::step (this=0x57bcdc0, tid=1070) at src/flusher.cc:165
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x6370960) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 30 (Thread 0x7f45cf30c700 (LWP 3858)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=0.20000000000000001) at src/scheduler.cc:437
      #7 0x00007f45d913186e in Flusher::step (this=0xecaf00, tid=10) at src/flusher.cc:165
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x63708c0) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 29 (Thread 0x7f45ceb0b700 (LWP 3859)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=0.20000000000000001) at src/scheduler.cc:437
      #7 0x00007f45d913186e in Flusher::step (this=0x9c93e00, tid=3011) at src/flusher.cc:165
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x6370820) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 28 (Thread 0x7f45ce30a700 (LWP 3860)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      --Type <return> to continue, or q <return> to quit--
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=0.20000000000000001) at src/scheduler.cc:437
      #7 0x00007f45d913186e in Flusher::step (this=0x9c93cc0, tid=3010) at src/flusher.cc:165
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x6370780) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 27 (Thread 0x7f45cdb09700 (LWP 3861)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=0.20000000000000001) at src/scheduler.cc:437
      #7 0x00007f45d913186e in Flusher::step (this=0xecb040, tid=11) at src/flusher.cc:165
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x63706e0) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 26 (Thread 0x7f45cd308700 (LWP 3862)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=0.20000000000000001) at src/scheduler.cc:437
      #7 0x00007f45d913186e in Flusher::step (this=0x57bc000, tid=2051) at src/flusher.cc:165
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x6370640) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 25 (Thread 0x7f45ccb07700 (LWP 3863)):
      #0 0x00007f45debf00fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45d91186e2 in wait (tv=..., this=<optimized out>) at src/syncobject.h:77
      #2 wait (secs=<optimized out>, this=<optimized out>) at src/syncobject.h:93
      #3 wait (previousCounter=<optimized out>, howlong=<optimized out>, this=<optimized out>) at src/tapconnmap.h:225
      #4 EventuallyPersistentEngine::notifyPendingConnections (this=0x5728a00) at src/ep_engine.cc:3383
      #5 0x00007f45d91187d3 in EvpNotifyPendingConns (arg=0x5728a00) at src/ep_engine.cc:1139
      #6 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #7 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 24 (Thread 0x7f45cc306700 (LWP 3864)):
      #0 0x00007f45debf00fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
      --Type <return> to continue, or q <return> to quit--
      #1 0x00007f45d90fbfb8 in wait (tv=..., this=<optimized out>) at src/syncobject.h:77
      #2 IdleTask::run (this=0x5806480, d=...) at src/dispatcher.cc:344
      #3 0x00007f45d90feb6a in Dispatcher::run (this=0x5709a40) at src/dispatcher.cc:186
      #4 0x00007f45d90ff33d in launch_dispatcher_thread (arg=0x5709a94) at src/dispatcher.cc:30
      #5 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #6 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #7 0x0000000000000000 in ?? ()

      Thread 23 (Thread 0x7f45cbb05700 (LWP 3885)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=0.20000000000000001) at src/scheduler.cc:437
      #7 0x00007f45d913186e in Flusher::step (this=0x57bdb80, tid=2050) at src/flusher.cc:165
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x791c3c0) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 22 (Thread 0x7f45cb304700 (LWP 3886)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=60) at src/scheduler.cc:437
      #7 0x00007f45d915f114 in StatSnap::run (this=<optimized out>) at src/tasks.cc:96
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x791c320) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 21 (Thread 0x7f45cab03700 (LWP 3887)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=0.20000000000000001) at src/scheduler.cc:437
      #7 0x00007f45d913186e in Flusher::step (this=0x57bca00, tid=1067) at src/flusher.cc:165
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x791c280) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 20 (Thread 0x7f45ca302700 (LWP 3888)):
      --Type <return> to continue, or q <return> to quit--
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=1) at src/scheduler.cc:437
      #7 0x00007f45d90f0729 in BgFetcher::run (this=0x5710ea0, tid=<optimized out>) at src/bgfetcher.cc:163
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x791c1e0) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 19 (Thread 0x7f45c9b01700 (LWP 3889)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=1) at src/scheduler.cc:437
      #7 0x00007f45d90f0729 in BgFetcher::run (this=0x5710b40, tid=<optimized out>) at src/bgfetcher.cc:163
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x791c140) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 18 (Thread 0x7f45c9300700 (LWP 3890)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=0.20000000000000001) at src/scheduler.cc:437
      #7 0x00007f45d913186e in Flusher::step (this=0x9c93b80, tid=3008) at src/flusher.cc:165
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x791c0a0) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 17 (Thread 0x7f45c8aff700 (LWP 3891)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=1) at src/scheduler.cc:437
      #7 0x00007f45d90f0729 in BgFetcher::run (this=0x5710d80, tid=<optimized out>) at src/bgfetcher.cc:163
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x791c000) at src/scheduler.cc:97
      --Type <return> to continue, or q <return> to quit--
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 16 (Thread 0x7f45c82fe700 (LWP 3892)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=0.20000000000000001) at src/scheduler.cc:437
      #7 0x00007f45d913186e in Flusher::step (this=0xecb400, tid=9) at src/flusher.cc:165
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x9871680) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 15 (Thread 0x7f45c7afd700 (LWP 3893)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=1) at src/scheduler.cc:437
      #7 0x00007f45d90f0729 in BgFetcher::run (this=0x57106c0, tid=<optimized out>) at src/bgfetcher.cc:163
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0x98715e0) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 14 (Thread 0x7f45c72fc700 (LWP 3894)):
      #0 0x00007f45debf00fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45d91186e2 in wait (tv=..., this=<optimized out>) at src/syncobject.h:77
      #2 wait (secs=<optimized out>, this=<optimized out>) at src/syncobject.h:93
      #3 wait (previousCounter=<optimized out>, howlong=<optimized out>, this=<optimized out>) at src/tapconnmap.h:225
      #4 EventuallyPersistentEngine::notifyPendingConnections (this=0x5729e00) at src/ep_engine.cc:3383
      #5 0x00007f45d91187d3 in EvpNotifyPendingConns (arg=0x5729e00) at src/ep_engine.cc:1139
      #6 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #7 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 13 (Thread 0x7f45c6afb700 (LWP 3895)):
      #0 0x00007f45debf00fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45d90fbfb8 in wait (tv=..., this=<optimized out>) at src/syncobject.h:77
      #2 IdleTask::run (this=0x5804d80, d=...) at src/dispatcher.cc:344
      #3 0x00007f45d90feb6a in Dispatcher::run (this=0x7a6ca80) at src/dispatcher.cc:186
      #4 0x00007f45d90ff33d in launch_dispatcher_thread (arg=0x7a6cad4) at src/dispatcher.cc:30
      #5 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #6 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      --Type <return> to continue, or q <return> to quit--
      #7 0x0000000000000000 in ?? ()

      Thread 12 (Thread 0x7f45c62fa700 (LWP 3915)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=0.20000000000000001) at src/scheduler.cc:437
      #7 0x00007f45d913186e in Flusher::step (this=0x57bc8c0, tid=1071) at src/flusher.cc:165
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0xbd065a0) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 11 (Thread 0x7f45c5af9700 (LWP 3916)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=60) at src/scheduler.cc:437
      #7 0x00007f45d915f114 in StatSnap::run (this=<optimized out>) at src/tasks.cc:96
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0xbd06500) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 10 (Thread 0x7f45c52f8700 (LWP 3917)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=0.20000000000000001) at src/scheduler.cc:437
      #7 0x00007f45d913186e in Flusher::step (this=0x57bdcc0, tid=2048) at src/flusher.cc:165
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0xbd06460) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 9 (Thread 0x7f45c4af7700 (LWP 3918)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      --Type <return> to continue, or q <return> to quit--
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=60) at src/scheduler.cc:437
      #7 0x00007f45d915f114 in StatSnap::run (this=<optimized out>) at src/tasks.cc:96
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0xbd063c0) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 8 (Thread 0x7f45c42f6700 (LWP 3919)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=0.20000000000000001) at src/scheduler.cc:437
      #7 0x00007f45d913186e in Flusher::step (this=0x9c93a40, tid=3012) at src/flusher.cc:165
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0xbd06320) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 7 (Thread 0x7f45c3af5700 (LWP 3920)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=1) at src/scheduler.cc:437
      #7 0x00007f45d90f0729 in BgFetcher::run (this=0x5711200, tid=<optimized out>) at src/bgfetcher.cc:163
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0xbd06280) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 6 (Thread 0x7f45c32f4700 (LWP 3921)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=1) at src/scheduler.cc:437
      #7 0x00007f45d90f0729 in BgFetcher::run (this=0x5711560, tid=<optimized out>) at src/bgfetcher.cc:163
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0xbd061e0) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 5 (Thread 0x7f45c2af3700 (LWP 3922)):
      --Type <return> to continue, or q <return> to quit--
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=0.20000000000000001) at src/scheduler.cc:437
      #7 0x00007f45d913186e in Flusher::step (this=0xecb2c0, tid=13) at src/flusher.cc:165
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0xbd06140) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 4 (Thread 0x7f45c22f2700 (LWP 3923)):
      #0 0x00007f45debf289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45debee065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #2 0x00007f45debedeba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
      #3 0x00007f45d9138aea in Mutex::acquire (this=0x578bb10) at src/mutex.cc:80
      #4 0x00007f45d913a458 in lock (this=<optimized out>) at src/locks.h:66
      #5 LockHolder (m=..., this=<optimized out>) at src/locks.h:44
      #6 ExecutorPool::snooze (this=0x578ba00, taskId=128, tosleep=0.20000000000000001) at src/scheduler.cc:437
      #7 0x00007f45d913186e in Flusher::step (this=0x57bcc80, tid=1069) at src/flusher.cc:165
      #8 0x00007f45d913e31b in ExecutorThread::run (this=0xbd060a0) at src/scheduler.cc:97
      #9 0x00007f45d913e81d in launch_executor_thread (arg=0x578bb18) at src/scheduler.cc:38
      #10 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #11 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #12 0x0000000000000000 in ?? ()

      Thread 3 (Thread 0x7f45c1af1700 (LWP 3924)):
      #0 0x00007f45debf00fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45d91186e2 in wait (tv=..., this=<optimized out>) at src/syncobject.h:77
      #2 wait (secs=<optimized out>, this=<optimized out>) at src/syncobject.h:93
      #3 wait (previousCounter=<optimized out>, howlong=<optimized out>, this=<optimized out>) at src/tapconnmap.h:225
      #4 EventuallyPersistentEngine::notifyPendingConnections (this=0x572a800) at src/ep_engine.cc:3383
      #5 0x00007f45d91187d3 in EvpNotifyPendingConns (arg=0x572a800) at src/ep_engine.cc:1139
      #6 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #7 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 2 (Thread 0x7f45c12f0700 (LWP 3925)):
      #0 0x00007f45debf00fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
      #1 0x00007f45d90fbfb8 in wait (tv=..., this=<optimized out>) at src/syncobject.h:77
      #2 IdleTask::run (this=0x9bdcab0, d=...) at src/dispatcher.cc:344
      #3 0x00007f45d90feb6a in Dispatcher::run (this=0xbf82a80) at src/dispatcher.cc:186
      #4 0x00007f45d90ff33d in launch_dispatcher_thread (arg=0xbf82ad4) at src/dispatcher.cc:30
      #5 0x00007f45debebe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      #6 0x00007f45de50cccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
      #7 0x0000000000000000 in ?? ()

      Thread 1 (Thread 0x7f45dfa82740 (LWP 3782)):
      #0 0x00007f45de50d363 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
      #1 0x00007f45dee235a6 in epoll_dispatch (base=0x569e780, tv=<optimized out>) at epoll.c:404
      #2 0x00007f45dee0ea04 in event_base_loop (base=0x569e780, flags=<optimized out>) at event.c:1558
      --Type <return> to continue, or q <return> to quit--
      #3 0x000000000040af4d in main (argc=<optimized out>, argv=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:7667
      (gdb)

      Attachments

        For Gerrit Dashboard: MB-9237
        # Subject Branch Project Status CR V

        Activity

          People

            thuan Thuan Nguyen
            thuan Thuan Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty