Memcached core caused rebalance in (crash on node that was being rebalanced in) to fail on cluster with bidirectional and unidirectional replication set up

• Set up procedure: Set up 10:10 bidirectional + unidirectional replication with 2 buckets
• Failover a node on C1, rebalance out.
• Rebalance in a new node on C1, crash observed at this point.

Crash on node ec2-204-236-136-169.us-west-1.compute.amazonaws.com on C1.

core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'

Results of ..
gdb /opt/couchbase/bin/memcached core
t a a bt
:-

GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New LWP 1461]
[New LWP 1463]
[New LWP 1464]
[New LWP 1210]
[New LWP 1214]
[New LWP 1462]
[New LWP 1221]
[New LWP 1220]
[New LWP 1219]
[New LWP 1208]
[New LWP 1211]
[New LWP 1217]
[New LWP 1199]
[New LWP 1222]
[New LWP 1213]
[New LWP 1209]
[New LWP 1212]
[New LWP 1460]
[New LWP 1218]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0 0x00007fa3312e6445 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) t a a bt

Thread 19 (Thread 0x7fa32bc49700 (LWP 1218)):
#0 0x00007fa33139ba5d in fdatasync () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fa32c868daf in couch_sync (handle=<optimized out>) at src/os.c:117
#2 0x00007fa32cb19bbf in cfs_sync (h=0x3e64e5a0) at src/couch-kvstore/couch-fs-stats.cc:86
#3 0x00007fa32c865153 in couchstore_commit (db=0x2c6f6310) at src/couch_db.c:190
#4 0x00007fa32cb13496 in CouchKVStore::saveDocs (this=0x5ff8000, vbid=240, rev=<optimized out>, docs=<optimized out>, docinfos=0x3e64e5c0, docCount=3) at src/couch-kvstore/couch-kvstore.cc:1563
#5 0x00007fa32cb13b41 in CouchKVStore::commit2couchstore (this=0x5ff8000) at src/couch-kvstore/couch-kvstore.cc:1485
#6 0x00007fa32cb13e79 in queueItem (req=0x3da123c0, this=<optimized out>) at src/couch-kvstore/couch-kvstore.cc:1617
#7 CouchKVStore::set (this=0x5ff8000, itm=..., cb=...) at src/couch-kvstore/couch-kvstore.cc:342
#8 0x00007fa32ca9f867 in EventuallyPersistentStore::flushOneDelOrSet (this=0x5fde480, qi=..., rejectQueue=..., vb=...) at src/ep.cc:2399
#9 0x00007fa32caa3d53 in EventuallyPersistentStore::flushOne (this=0x5fde480, queue=..., rejectQueue=..., vb=...) at src/ep.cc:2447
#10 0x00007fa32caa401b in EventuallyPersistentStore::flushVBQueue (this=0x5fde480, vb=..., vb_queue=..., vbid=<optimized out>, data_age=0) at src/ep.cc:2009
#11 0x00007fa32caa4c7e in EventuallyPersistentStore::flushOutgoingQueue (this=0x5fde480, flushQueue=0x5fde748, flushPhase=<optimized out>, nextVbid=<optimized out>) at src/ep.cc:1954
#12 0x00007fa32cacf14c in Flusher::doFlush (this=0x5ff65a0) at src/flusher.cc:245
#13 0x00007fa32cacfae5 in Flusher::step (this=0x7b, d=..., tid=...) at src/flusher.cc:158
#14 0x00007fa32ca9629a in Dispatcher::run (this=0x5fd8a80) at src/dispatcher.cc:173
#15 0x00007fa32ca96b9b in launch_dispatcher_thread (arg=0x7b) at src/dispatcher.cc:28
#16 0x00007fa331674e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#17 0x00007fa3313a24bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#18 0x0000000000000000 in ?? ()

Thread 18 (Thread 0x7fa329444700 (LWP 1460)):
#0 0x00007fa33167c6c3 in pwrite64 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fa32c868e08 in couch_pwrite (handle=<optimized out>, buf=0x3c160031, nbyte=755, offset=4435678) at src/os.c:56
#2 0x00007fa32cb19df0 in cfs_pwrite (h=0x3e6cd540, buf=0x3c160031, sz=755, off=4435678) at src/couch-kvstore/couch-fs-stats.cc:75
#3 0x00007fa32c867848 in flush_buffer (buf=<optimized out>) at src/iobuffer.c:106
#4 buffered_pread (handle=0x3dde3fb0, buf=<optimized out>, nbyte=8, offset=3727192) at src/iobuffer.c:271
#5 0x00007fa32c865efc in read_skipping_prefixes (dst=0x7fa329442f00, len=<optimized out>, pos=<optimized out>, db=<optimized out>) at src/couch_file_read.c:27
#6 pread_bin_internal (db=0x2c6f6fc0, pos=3727192, ret_ptr=<optimized out>, header=0) at src/couch_file_read.c:53
#7 0x00007fa32c866107 in pread_compressed (db=0x80, pos=1008074801, ret_ptr=0x7fa329442ff8) at src/couch_file_read.c:92
#8 0x00007fa32c861cda in modify_node (rq=0x7fa329443290, nptr=<optimized out>, start=781, end=-1, dst=0x1e5fec38) at src/btree_modify.c:321
#9 0x00007fa32c861f65 in modify_node (rq=0x7fa329443290, nptr=<optimized out>, start=781, end=922, dst=<optimized out>) at src/btree_modify.c:468
#10 0x00007fa32c861f65 in modify_node (rq=0x7fa329443290, nptr=<optimized out>, start=768, end=2980, dst=<optimized out>) at src/btree_modify.c:468
#11 0x00007fa32c8628b7 in modify_btree (rq=0x7fa329443290, root=0x2f7d47c0, errcode=0x7fa32944333c) at src/btree_modify.c:620
#12 0x00007fa32c86567f in update_indexes (db=<optimized out>, seqs=0x3e6e0010, seqvals=0x3e6eba50, ids=<optimized out>, idvals=<optimized out>, numdocs=1490) at src/couch_save.c:222
#13 0x00007fa32c865c41 in couchstore_save_documents (db=<optimized out>, docs=0x3e634000, infos=0x3e6dd000, numdocs=<optimized out>, options=<optimized out>) at src/couch_save.c:354
#14 0x00007fa32cb133d7 in CouchKVStore::saveDocs (this=0x5ff9600, vbid=886, rev=<optimized out>, docs=<optimized out>, docinfos=0x3e6dd000, docCount=1490) at src/couch-kvstore/couch-kvstore.cc:1550
#15 0x00007fa32cb13b41 in CouchKVStore::commit2couchstore (this=0x5ff9600) at src/couch-kvstore/couch-kvstore.cc:1485
#16 0x00007fa32cb13e79 in queueItem (req=0x3e6d9e00, this=<optimized out>) at src/couch-kvstore/couch-kvstore.cc:1617
#17 CouchKVStore::set (this=0x5ff9600, itm=..., cb=...) at src/couch-kvstore/couch-kvstore.cc:342
#18 0x00007fa32ca9f867 in EventuallyPersistentStore::flushOneDelOrSet (this=0x5fe0d00, qi=..., rejectQueue=..., vb=...) at src/ep.cc:2399
#19 0x00007fa32caa3d53 in EventuallyPersistentStore::flushOne (this=0x5fe0d00, queue=..., rejectQueue=..., vb=...) at src/ep.cc:2447
#20 0x00007fa32caa401b in EventuallyPersistentStore::flushVBQueue (this=0x5fe0d00, vb=..., vb_queue=..., vbid=<optimized out>, data_age=0) at src/ep.cc:2009
#21 0x00007fa32caa4c7e in EventuallyPersistentStore::flushOutgoingQueue (this=0x5fe0d00, flushQueue=0x5fe0fc8, flushPhase=<optimized out>, nextVbid=<optimized out>) at src/ep.cc:1954
#22 0x00007fa32cacf14c in Flusher::doFlush (this=0x5ff6c60) at src/flusher.cc:245
#23 0x00007fa32cacfae5 in Flusher::step (this=0x80, d=..., tid=...) at src/flusher.cc:158
#24 0x00007fa32ca9629a in Dispatcher::run (this=0x1e8a2a80) at src/dispatcher.cc:173
#25 0x00007fa32ca96b9b in launch_dispatcher_thread (arg=0x80) at src/dispatcher.cc:28
#26 0x00007fa331674e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#27 0x00007fa3313a24bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#28 0x0000000000000000 in ?? ()

Thread 17 (Thread 0x7fa32e56a700 (LWP 1212)):
#0 0x00007fa3313a2b53 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fa332200f36 in epoll_dispatch (base=0x5f8cc80, tv=<optimized out>) at epoll.c:404
#2 0x00007fa3321ec394 in event_base_loop (base=0x5f8cc80, flags=<optimized out>) at event.c:1558
#3 0x0000000000414b94 in worker_libevent (arg=0x172b5f0) at daemon/thread.c:301
--Type <return> to continue, or q <return> to quit--
#4 0x00007fa331674e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fa3313a24bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 16 (Thread 0x7fa32ff7a700 (LWP 1209)):
#0 0x00007fa3316790fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fa330194176 in logger_thead_main (arg=<optimized out>) at extensions/loggers/file_logger.c:368
#2 0x00007fa331674e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fa3313a24bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000000000000000 in ?? ()

Thread 15 (Thread 0x7fa32dd69700 (LWP 1213)):
#0 0x00007fa3313a2b53 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fa332200f36 in epoll_dispatch (base=0x5f8ca00, tv=<optimized out>) at epoll.c:404
#2 0x00007fa3321ec394 in event_base_loop (base=0x5f8ca00, flags=<optimized out>) at event.c:1558
#3 0x0000000000414b94 in worker_libevent (arg=0x172b6e8) at daemon/thread.c:301
#4 0x00007fa331674e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fa3313a24bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 14 (Thread 0x7fa329c45700 (LWP 1222)):
#0 0x00007fa3316790fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fa32cab3faf in wait (tv=..., this=<optimized out>) at src/syncobject.hh:58
#2 wait (secs=<optimized out>, this=<optimized out>) at src/syncobject.hh:74
#3 wait (previousCounter=<optimized out>, howlong=<optimized out>, this=<optimized out>) at src/tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x5fe2000) at src/ep_engine.cc:3382
#5 0x00007fa32cab4093 in EvpNotifyPendingConns (arg=0x5fe2000) at src/ep_engine.cc:1128
#6 0x00007fa331674e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7 0x00007fa3313a24bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 13 (Thread 0x7fa33283e740 (LWP 1199)):
#0 0x00007fa3313a2b53 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fa332200f36 in epoll_dispatch (base=0x5f8c000, tv=<optimized out>) at epoll.c:404
#2 0x00007fa3321ec394 in event_base_loop (base=0x5f8c000, flags=<optimized out>) at event.c:1558
#3 0x000000000040c2e1 in main (argc=<optimized out>, argv=<optimized out>) at daemon/memcached.c:7918

Thread 12 (Thread 0x7fa32c657700 (LWP 1217)):
#0 0x00007fa33136e03d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fa33139bf74 in usleep () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fa32cad4fb5 in updateStatsThread (arg=<optimized out>) at src/memory_tracker.cc:31
#3 0x00007fa331674e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fa3313a24bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7fa32ed6b700 (LWP 1211)):
#0 0x00007fa3313a2b53 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fa332200f36 in epoll_dispatch (base=0x5f8c280, tv=<optimized out>) at epoll.c:404
#2 0x00007fa3321ec394 in event_base_loop (base=0x5f8c280, flags=<optimized out>) at event.c:1558
#3 0x0000000000414b94 in worker_libevent (arg=0x172b4f8) at daemon/thread.c:301
#4 0x00007fa331674e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fa3313a24bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7fa330b96700 (LWP 1208)):
#0 0x00007fa3313950bd in read () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fa33132aa88 in _IO_file_underflow () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fa33132bace in _IO_default_uflow () from /lib/x86_64-linux-gnu/libc.so.6
--Type <return> to continue, or q <return> to quit--
#3 0x00007fa33131fc1a in _IO_getline_info () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007fa33131eafb in fgets () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007fa330b97b19 in fgets (__stream=<optimized out>, __n=<optimized out>, __s=<optimized out>) at /usr/include/bits/stdio2.h:255
#6 check_stdin_thread (arg=<optimized out>) at extensions/daemon/stdin_check.c:37
#7 0x00007fa331674e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007fa3313a24bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7fa32b448700 (LWP 1219)):
#0 0x00007fa3316790fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fa32ca932e8 in wait (tv=..., this=<optimized out>) at src/syncobject.hh:58
#2 IdleTask::run (this=0x1730120, d=...) at src/dispatcher.cc:336
#3 0x00007fa32ca9629a in Dispatcher::run (this=0x5fd9880) at src/dispatcher.cc:173
#4 0x00007fa32ca96b9b in launch_dispatcher_thread (arg=0x5fd98d4) at src/dispatcher.cc:28
#5 0x00007fa331674e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fa3313a24bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7fa32ac47700 (LWP 1220)):
#0 0x00007fa3316790fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fa32ca932e8 in wait (tv=..., this=<optimized out>) at src/syncobject.hh:58
#2 IdleTask::run (this=0x1730360, d=...) at src/dispatcher.cc:336
#3 0x00007fa32ca9629a in Dispatcher::run (this=0x5fd96c0) at src/dispatcher.cc:173
#4 0x00007fa32ca96b9b in launch_dispatcher_thread (arg=0x5fd9714) at src/dispatcher.cc:28
#5 0x00007fa331674e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fa3313a24bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7fa32a446700 (LWP 1221)):
#0 0x00007fa3316790fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fa32ca932e8 in wait (tv=..., this=<optimized out>) at src/syncobject.hh:58
#2 IdleTask::run (this=0x17302d0, d=...) at src/dispatcher.cc:336
#3 0x00007fa32ca9629a in Dispatcher::run (this=0x5fd9500) at src/dispatcher.cc:173
#4 0x00007fa32ca96b9b in launch_dispatcher_thread (arg=0x5fd9554) at src/dispatcher.cc:28
#5 0x00007fa331674e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fa3313a24bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7fa328442700 (LWP 1462)):
#0 0x00007fa3316790fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fa32ca932e8 in wait (tv=..., this=<optimized out>) at src/syncobject.hh:58
#2 IdleTask::run (this=0xaefca20, d=...) at src/dispatcher.cc:336
#3 0x00007fa32ca9629a in Dispatcher::run (this=0x1e8a36c0) at src/dispatcher.cc:173
#4 0x00007fa32ca96b9b in launch_dispatcher_thread (arg=0x1e8a3714) at src/dispatcher.cc:28
#5 0x00007fa331674e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fa3313a24bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7fa32d568700 (LWP 1214)):
#0 0x00007fa3313a2b53 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fa332200f36 in epoll_dispatch (base=0x5f8c780, tv=<optimized out>) at epoll.c:404
#2 0x00007fa3321ec394 in event_base_loop (base=0x5f8c780, flags=<optimized out>) at event.c:1558
#3 0x0000000000414b94 in worker_libevent (arg=0x172b7e0) at daemon/thread.c:301
#4 0x00007fa331674e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fa3313a24bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7fa32f56c700 (LWP 1210)):
--Type <return> to continue, or q <return> to quit--
#0 0x00007fa3313a2b53 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fa332200f36 in epoll_dispatch (base=0x5f8c500, tv=<optimized out>) at epoll.c:404
#2 0x00007fa3321ec394 in event_base_loop (base=0x5f8c500, flags=<optimized out>) at event.c:1558
#3 0x0000000000414b94 in worker_libevent (arg=0x172b400) at daemon/thread.c:301
#4 0x00007fa331674e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fa3313a24bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7fa327440700 (LWP 1464)):
#0 0x00007fa3316790fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fa32cab3faf in wait (tv=..., this=<optimized out>) at src/syncobject.hh:58
#2 wait (secs=<optimized out>, this=<optimized out>) at src/syncobject.hh:74
#3 wait (previousCounter=<optimized out>, howlong=<optimized out>, this=<optimized out>) at src/tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x5fe5600) at src/ep_engine.cc:3382
#5 0x00007fa32cab4093 in EvpNotifyPendingConns (arg=0x5fe5600) at src/ep_engine.cc:1128
#6 0x00007fa331674e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7 0x00007fa3313a24bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7fa327c41700 (LWP 1463)):
#0 0x00007fa3316790fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fa32ca932e8 in wait (tv=..., this=<optimized out>) at src/syncobject.hh:58
#2 IdleTask::run (this=0xaefc3f0, d=...) at src/dispatcher.cc:336
#3 0x00007fa32ca9629a in Dispatcher::run (this=0x1e8a3500) at src/dispatcher.cc:173
#4 0x00007fa32ca96b9b in launch_dispatcher_thread (arg=0x1e8a3554) at src/dispatcher.cc:28
#5 0x00007fa331674e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fa3313a24bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fa328c43700 (LWP 1461)):
#0 0x00007fa3312e6445 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fa3312e9bab in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fa3312df10e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007fa3312df1b2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007fa32cb1495b in CouchKVStore::fetchDoc (this=0x5ff8b00, db=0x131eb960, docinfo=0x131eb5e0, docValue=..., vbId=637, metaOnly=104) at src/couch-kvstore/couch-kvstore.cc:1348
#5 0x00007fa32cb1508b in CouchKVStore::get (this=0x5ff8b00, key=..., vb=<optimized out>, cb=...) at src/couch-kvstore/couch-kvstore.cc:394
#6 0x00007fa32caa0d1f in EventuallyPersistentStore::completeBGFetch (this=0x5fe0d00, key=..., vbucket=637, rowid=18446744073709551615, cookie=<optimized out>, init=<optimized out>,
type=BG_FETCH_METADATA) at src/ep.cc:1109
#7 0x00007fa32cab0bca in BGFetchCallback::callback(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /opt/couchbase/lib/memcached/ep.so
#8 0x00007fa32ca9629a in Dispatcher::run (this=0x1e8a28c0) at src/dispatcher.cc:173
#9 0x00007fa32ca96b9b in launch_dispatcher_thread (arg=0x4af) at src/dispatcher.cc:28
#10 0x00007fa331674e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007fa3313a24bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x0000000000000000 in ?? ()