memcached crashed with signal 11, Segmentation fault in HashTable::visit()

build 2.0.2-769
on run
http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/593/consoleFull
I see several identical crashes

andrey@baranouski:~/repository/testrunner$ python scripts/ssh.py -i centos-64-2.0-failover-tests.ini "ls -la /tmp"
10.1.3.114
total 2105288
drwxrwxrwt 5 root root 4096 Apr 22 09:11 .
drwxr-xr-x 25 root root 4096 Jan 29 10:56 ..
rw------ 1 couchbase couchbase 2229407744 Apr 22 05:17 core.memcached.26483
rw-rr- 1 root root 127465162 Apr 20 00:25 couchbase-server-enterprise_x86_64_2.0.2-769-rel.rpm
drwxrwxrwt 2 root root 4096 Jan 29 10:56 .font-unix
srw-rw-rw- 1 root root 0 Jan 29 10:56 .gdm_socket
drwxrwxrwt 2 root root 4096 Jan 29 10:56 .ICE-unix
rrr- 1 root root 11 Jan 29 10:56 .X0-lock
drwxrwxrwt 2 root root 4096 Jan 29 10:56 .X11-unix

10.1.3.117
total 125044
drwxrwxrwt 5 root root 413696 Apr 22 09:11 .
drwxr-xr-x 25 root root 4096 Feb 11 13:26 ..
rw-rr- 1 root root 127465162 Apr 20 00:25 couchbase-server-enterprise_x86_64_2.0.2-769-rel.rpm
drwxrwxrwt 2 root root 4096 Feb 11 13:26 .font-unix
srw-rw-rw- 1 root root 0 Feb 11 13:26 .gdm_socket
drwxrwxrwt 2 root root 4096 Feb 11 13:26 .ICE-unix
rrr- 1 root root 11 Feb 11 13:26 .X0-lock
drwxrwxrwt 2 root root 4096 Feb 11 13:26 .X11-unix

10.1.3.115
total 2083176
drwxrwxrwt 5 root root 401408 Apr 22 09:11 .
drwxr-xr-x 25 root root 4096 Oct 25 16:14 ..
rw------ 1 couchbase couchbase 2157531136 Apr 22 05:24 core.memcached.1265
rw-rr- 1 root root 127465162 Apr 20 00:25 couchbase-server-enterprise_x86_64_2.0.2-769-rel.rpm
drwxrwxrwt 2 root root 4096 Oct 25 16:19 .font-unix
srw-rw-rw- 1 root root 0 Oct 25 16:19 .gdm_socket
drwxrwxrwt 2 root root 4096 Oct 25 16:14 .ICE-unix
rrr- 1 root root 11 Oct 25 16:19 .X0-lock
drwxrwxrwt 2 root root 4096 Oct 25 16:19 .X11-unix

10.1.3.118
total 3928388
drwxrwxrwt 6 root root 413696 Apr 22 09:11 .
drwxr-xr-x 25 root root 4096 Oct 25 16:15 ..
rw------ 1 couchbase couchbase 2261405696 Apr 22 04:01 core.memcached.23290
rw------ 1 couchbase couchbase 2153013248 Apr 22 04:32 core.memcached.30599
rw-rr- 1 root root 127465162 Apr 20 00:25 couchbase-server-enterprise_x86_64_2.0.2-769-rel.rpm
drwxrwxrwt 2 root root 4096 Oct 25 16:20 .font-unix
srw-rw-rw- 1 root root 0 Oct 25 16:20 .gdm_socket
drwxrwxrwt 2 root root 4096 Oct 25 16:15 .ICE-unix
drwx------ 2 root root 4096 May 15 2012 keyring-cqfvn9
rrr- 1 root root 11 Oct 25 16:20 .X0-lock
drwxrwxrwt 2 root root 4096 Oct 25 16:20 .X11-unix

10.1.3.116
total 125016
drwxrwxrwt 5 root root 385024 Apr 22 09:11 .
drwxr-xr-x 25 root root 4096 Oct 25 16:15 ..
rw-rr- 1 root root 127465162 Apr 20 00:25 couchbase-server-enterprise_x86_64_2.0.2-769-rel.rpm
drwxrwxrwt 2 root root 4096 Oct 25 16:19 .font-unix
srw-rw-rw- 1 root root 0 Oct 25 16:20 .gdm_socket
drwxrwxrwt 2 root root 4096 Oct 25 16:15 .ICE-unix
rrr- 1 root root 11 Oct 25 16:20 .X0-lock
drwxrwxrwt 2 root root 4096 Oct 25 16:20 .X11-unix

[root@localhost tmp]# gdb /opt/couchbase/bin/memcached core.memcached.30599
GNU gdb (GDB) CentOS (7.0.1-42.el5.centos)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 30684]
[New Thread 30685]
[New Thread 30683]
[New Thread 30682]
[New Thread 30681]
[New Thread 30634]
[New Thread 30633]
[New Thread 30632]
[New Thread 30631]
[New Thread 30630]
[New Thread 30621]
[New Thread 30620]
[New Thread 30619]
[New Thread 30618]
[New Thread 30617]
[New Thread 30616]
[New Thread 30615]
[New Thread 30614]
[New Thread 30613]
[New Thread 30612]
[New Thread 30611]
[New Thread 30610]
[New Thread 30609]
[New Thread 30599]
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff6d350000
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 11, Segmentation fault.
#0 0x00002aaaaaf32d11 in HashTable::visit (this=0x2aaac5cce408, visitor=...) at src/stored-value.cc:395
395 src/stored-value.cc: No such file or directory.
in src/stored-value.cc
(gdb) t a a bt

Thread 24 (Thread 0x2adfa3764220 (LWP 30599)):
#0 0x00000038286d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002adfa32e4576 in epoll_dispatch (base=0x8b4c000, tv=<value optimized out>) at epoll.c:404
#2 0x00002adfa32cfe44 in event_base_loop (base=0x8b4c000, flags=<value optimized out>) at event.c:1558
#3 0x0000000000409742 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7919

Thread 23 (Thread 30609):
#0 0x00000038286c678b in read () from /lib64/libc.so.6
#1 0x000000382866cd57 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x000000382866d71e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x0000003828662804 in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x00000038286616a9 in fgets () from /lib64/libc.so.6
#5 0x00002adfa3765939 in check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
#6 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#7 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 22 (Thread 30610):
#0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaaae4d6 in logger_thead_main (arg=0x42e8040) at extensions/loggers/file_logger.c:368
#2 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#3 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 21 (Thread 30611):
#0 0x00000038286d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002adfa32e4576 in epoll_dispatch (base=0x8b4c500, tv=<value optimized out>) at epoll.c:404
#2 0x00002adfa32cfe44 in event_base_loop (base=0x8b4c500, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414574 in worker_libevent (arg=0x42eb900) at daemon/thread.c:301
#4 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 20 (Thread 30612):
#0 0x00000038286d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002adfa32e4576 in epoll_dispatch (base=0x8b4c280, tv=<value optimized out>) at epoll.c:404
#2 0x00002adfa32cfe44 in event_base_loop (base=0x8b4c280, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414574 in worker_libevent (arg=0x42eb9f8) at daemon/thread.c:301
#4 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 19 (Thread 30613):
#0 0x00000038286d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002adfa32e4576 in epoll_dispatch (base=0x8b4cc80, tv=<value optimized out>) at epoll.c:404
#2 0x00002adfa32cfe44 in event_base_loop (base=0x8b4cc80, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414574 in worker_libevent (arg=0x42ebaf0) at daemon/thread.c:301
#4 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 18 (Thread 30614):
#0 0x00000038286d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002adfa32e4576 in epoll_dispatch (base=0x8b4ca00, tv=<value optimized out>) at epoll.c:404
#2 0x00002adfa32cfe44 in event_base_loop (base=0x8b4ca00, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414574 in worker_libevent (arg=0x42ebbe8) at daemon/thread.c:301
#4 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000038286d44bd in clone () from /lib64/libc.so.6

--Type <return> to continue, or q <return> to quit--
Thread 17 (Thread 30615):
#0 0x00000038286d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002adfa32e4576 in epoll_dispatch (base=0x8b4c780, tv=<value optimized out>) at epoll.c:404
#2 0x00002adfa32cfe44 in event_base_loop (base=0x8b4c780, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414574 in worker_libevent (arg=0x42ebce0) at daemon/thread.c:301
#4 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 16 (Thread 30616):
#0 0x000000382869a541 in nanosleep () from /lib64/libc.so.6
#1 0x00000038286cded4 in usleep () from /lib64/libc.so.6
#2 0x00002aaaaaf2f255 in updateStatsThread (arg=0x42e84c0) at src/memory_tracker.cc:31
#3 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#4 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 15 (Thread 30617):
#0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef29f8 in wait (this=0x8b98090, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x8b98090, d=...) at src/dispatcher.cc:328
#3 0x00002aaaaaef522a in Dispatcher::run (this=0x8b96c40) at src/dispatcher.cc:171
#4 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x8b96c40) at src/dispatcher.cc:28
#5 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#6 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 14 (Thread 30618):
#0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef29f8 in wait (this=0x8b982d0, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x8b982d0, d=...) at src/dispatcher.cc:328
#3 0x00002aaaaaef522a in Dispatcher::run (this=0x8b96a80) at src/dispatcher.cc:171
#4 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x8b96a80) at src/dispatcher.cc:28
#5 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#6 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 13 (Thread 30619):
#0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef29f8 in wait (this=0x8b98240, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x8b98240, d=...) at src/dispatcher.cc:328
#3 0x00002aaaaaef522a in Dispatcher::run (this=0x8b97880) at src/dispatcher.cc:171
#4 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x8b97880) at src/dispatcher.cc:28
#5 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#6 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 12 (Thread 30620):
#0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef29f8 in wait (this=0x8b981b0, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x8b981b0, d=...) at src/dispatcher.cc:328
#3 0x00002aaaaaef522a in Dispatcher::run (this=0x8b976c0) at src/dispatcher.cc:171
#4 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x8b976c0) at src/dispatcher.cc:28
#5 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#6 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 11 (Thread 30621):

#0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf0fd7f in wait (this=0x8ba4000) at src/syncobject.hh:57
#2 wait (this=0x8ba4000) at src/syncobject.hh:73
--Type <return> to continue, or q <return> to quit--
#3 wait (this=0x8ba4000) at src/tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x8ba4000) at src/ep_engine.cc:3369
#5 0x00002aaaaaf0fe63 in EvpNotifyPendingConns (arg=0x8ba4000) at src/ep_engine.cc:1121
#6 0x000000382920673d in start_thread () from /lib64/libpthread.so.0

#7 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 10 (Thread 30630):
#0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef29f8 in wait (this=0x8b985a0, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x8b985a0, d=...) at src/dispatcher.cc:328
#3 0x00002aaaaaef522a in Dispatcher::run (this=0x8cfa700) at src/dispatcher.cc:171
#4 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x8cfa700) at src/dispatcher.cc:28
#5 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#6 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 9 (Thread 30631):
#0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef29f8 in wait (this=0x8b986c0, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x8b986c0, d=...) at src/dispatcher.cc:328
#3 0x00002aaaaaef522a in Dispatcher::run (this=0x8cfa540) at src/dispatcher.cc:171
#4 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x8cfa540) at src/dispatcher.cc:28
#5 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#6 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 8 (Thread 30632):
#0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef29f8 in wait (this=0x8b98630, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x8b98630, d=...) at src/dispatcher.cc:328
#3 0x00002aaaaaef522a in Dispatcher::run (this=0x8cfb340) at src/dispatcher.cc:171
#4 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x8cfb340) at src/dispatcher.cc:28
#5 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#6 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 7 (Thread 30633):
#0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef29f8 in wait (this=0x8b98870, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x8b98870, d=...) at src/dispatcher.cc:328
#3 0x00002aaaaaef522a in Dispatcher::run (this=0x8cfb180) at src/dispatcher.cc:171
#4 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x8cfb180) at src/dispatcher.cc:28
#5 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#6 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 6 (Thread 30634):
#0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf0fd7f in wait (this=0x8ba5b00) at src/syncobject.hh:57
#2 wait (this=0x8ba5b00) at src/syncobject.hh:73
#3 wait (this=0x8ba5b00) at src/tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x8ba5b00) at src/ep_engine.cc:3369
#5 0x00002aaaaaf0fe63 in EvpNotifyPendingConns (arg=0x8ba5b00) at src/ep_engine.cc:1121
#6 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#7 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 5 (Thread 30681):
#0 0x000000382920e498 in pread64 () from /lib64/libpthread.so.0
#1 0x00002aaaab1cb738 in couch_pread (handle=<value optimized out>, buf=0x39510c31, nbyte=8192, offset=1982464) at src/os.c:38
--Type <return> to continue, or q <return> to quit--
#2 0x00002aaaaaf6e13d in cfs_pread (h=0x40456020, buf=0x39510c31, sz=8192, off=1982464) at src/couch-kvstore/couch-fs-stats.cc:70
#3 0x00002aaaab1c8a2b in load_buffer_from (handle=0x2aaac0f09bc0, buf=<value optimized out>, nbyte=2, offset=<value optimized out>) at src/iobuffer.c:153
#4 buffered_pread (handle=0x2aaac0f09bc0, buf=<value optimized out>, nbyte=2, offset=<value optimized out>) at src/iobuffer.c:292
#5 0x00002aaaab1c5323 in find_header_at_pos (filename=<value optimized out>, flags=<value optimized out>, ops=<value optimized out>, pDb=0x4ceced90) at src/couch_db.c:42
#6 find_header (filename=<value optimized out>, flags=<value optimized out>, ops=<value optimized out>, pDb=0x4ceced90) at src/couch_db.c:88
#7 couchstore_open_db_ex (filename=<value optimized out>, flags=<value optimized out>, ops=<value optimized out>, pDb=0x4ceced90) at src/couch_db.c:250
#8 0x00002aaaaaf64974 in CouchKVStore::openDB (this=0x8bb9b00, vbucketId=374, fileRev=2, db=0x4ceced90, options=1, newFileRev=0x4ceced88) at src/couch-kvstore/couch-kvstore.cc:1129
#9 0x00002aaaaaf67c4e in CouchKVStore::setVBucketState (this=0x8bb9b00, vbucketId=374, vbstate=..., vb_change_type=1, newfile=128, notify=true) at src/couch-kvstore/couch-kvstore.cc:706
#10 0x00002aaaaaf68a39 in CouchKVStore::snapshotVBuckets (this=0x8bb9b00, vbstates=std::map with 833 elements =

) at src/couch-kvstore/couch-kvstore.cc:596
#11 0x00002aaaaaefb0fe in EventuallyPersistentStore::snapshotVBuckets (this=0x8ba3600, priority=...) at src/ep.cc:845
#12 0x00002aaaaaf09d41 in SnapshotVBucketsCallback::callback(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /opt/couchbase/lib/memcached/ep.so
#13 0x00002aaaaaef522a in Dispatcher::run (this=0x9368000) at src/dispatcher.cc:171
#14 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x9368000) at src/dispatcher.cc:28
#15 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#16 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 4 (Thread 30682):
#0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef29f8 in wait (this=0x8b98c60, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x8b98c60, d=...) at src/dispatcher.cc:328
#3 0x00002aaaaaef522a in Dispatcher::run (this=0x9368e00) at src/dispatcher.cc:171
#4 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x9368e00) at src/dispatcher.cc:28
#5 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#6 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 3 (Thread 30683):
#0 0x000000382920e498 in pread64 () from /lib64/libpthread.so.0
#1 0x00002aaaab1cb738 in couch_pread (handle=<value optimized out>, buf=0x2aaad9b7c031, nbyte=8192, offset=1376256) at src/os.c:38
#2 0x00002aaaaaf6e13d in cfs_pread (h=0x9d75680, buf=0x2aaad9b7c031, sz=8192, off=1376256) at src/couch-kvstore/couch-fs-stats.cc:70
#3 0x00002aaaab1c8a2b in load_buffer_from (handle=0xcf07d70, buf=<value optimized out>, nbyte=1996, offset=<value optimized out>) at src/iobuffer.c:153
#4 buffered_pread (handle=0xcf07d70, buf=<value optimized out>, nbyte=1996, offset=<value optimized out>) at src/iobuffer.c:292
#5 0x00002aaaab1c7512 in read_skipping_prefixes (file=0x2aaac5ac12d0, pos=1376257, ret_ptr=0x4e2d02e8, header=<value optimized out>) at src/couch_file_read.c:68
#6 pread_bin_internal (file=0x2aaac5ac12d0, pos=1376257, ret_ptr=0x4e2d02e8, header=<value optimized out>) at src/couch_file_read.c:111
#7 0x00002aaaab1c7627 in pread_compressed (file=<value optimized out>, pos=<value optimized out>, ret_ptr=0x4e2d0328) at src/couch_file_read.c:133
#8 0x00002aaaab1c495a in bp_to_doc (db=<value optimized out>, docinfo=0x2aaae1ae4090, pDoc=0x4e2d03d8, options=<value optimized out>) at src/couch_db.c:403
#9 couchstore_open_doc_with_docinfo (db=<value optimized out>, docinfo=0x2aaae1ae4090, pDoc=0x4e2d03d8, options=<value optimized out>) at src/couch_db.c:549
#10 0x00002aaaaaf611a5 in CouchKVStore::recordDbDump (db=0x2aaac5ac12d0, docinfo=0x2aaae1ae4090, ctx=0x4e2d0d10) at src/couch-kvstore/couch-kvstore.cc:1342
#11 0x00002aaaab1c62bd in lookup_callback (rq=<value optimized out>, k=0x4e2d0510, v=0x4e2d0500) at src/couch_db.c:638
#12 0x00002aaaab1c3064 in btree_lookup_inner (rq=0x4e2d0750, diskpos=<value optimized out>, current=0, end=1) at src/btree_read.c:78
#13 0x00002aaaab1c2f58 in btree_lookup_inner (rq=0x4e2d0750, diskpos=<value optimized out>, current=0, end=1) at src/btree_read.c:52
#14 0x00002aaaab1c2f58 in btree_lookup_inner (rq=0x4e2d0750, diskpos=<value optimized out>, current=0, end=1) at src/btree_read.c:52
#15 0x00002aaaab1c478c in couchstore_changes_since (db=<value optimized out>, since=<value optimized out>, options=<value optimized out>, callback=<value optimized out>, ctx=<value optimized out>)
at src/couch_db.c:682
#16 0x00002aaaaaf65e2c in CouchKVStore::loadDB (this=0x9cf0000, cb=std::tr1::shared_ptr (count 5) 0xb697860, keysOnly=false, vbids=<value optimized out>, options=0)
at src/couch-kvstore/couch-kvstore.cc:962
#17 0x00002aaaaaf66913 in CouchKVStore::dump (this=0x9cf0000, vb=414, cb=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:780
#18 0x00002aaaaaee458a in BackfillDiskLoad::callback (this=0x2aaac5f12b40, d=<value optimized out>, t=<value optimized out>) at src/backfill.cc:60
#19 0x00002aaaaaef522a in Dispatcher::run (this=0x9368c40) at src/dispatcher.cc:171
#20 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x9368c40) at src/dispatcher.cc:28
#21 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#22 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 2 (Thread 30685):
#0 0x000000382920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf0fd7f in wait (this=0x8ba7600) at src/syncobject.hh:57
--Type <return> to continue, or q <return> to quit--
#2 wait (this=0x8ba7600) at src/syncobject.hh:73
#3 wait (this=0x8ba7600) at src/tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x8ba7600) at src/ep_engine.cc:3369
#5 0x00002aaaaaf0fe63 in EvpNotifyPendingConns (arg=0x8ba7600) at src/ep_engine.cc:1121
#6 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#7 0x00000038286d44bd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x4ecd2940 (LWP 30684)):
#0 0x00002aaaaaf32d11 in HashTable::visit (this=0x2aaac5cce408, visitor=...) at src/stored-value.cc:395
#1 0x00002aaaaaef8e62 in VBCBAdaptor::callback (this=0x3907f7a0, d=..., t=...) at src/ep.cc:2522
#2 0x00002aaaaaef522a in Dispatcher::run (this=0x9368a80) at src/dispatcher.cc:171
#3 0x00002aaaaaef5add in launch_dispatcher_thread (arg=0x9368a80) at src/dispatcher.cc:28
#4 0x000000382920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000038286d44bd in clone () from /lib64/libc.so.6