Port server memcached on node 'ns_1@10.3.121.31' exited with status 134." during xdcr

1. Setup unidirectional replication from cluster1 to cluster2 on bucket default.
2. Load 22M items on cluster1, expect 22M items on cluster2 on xdcr replication.

• thereafter continous load/mutations running on the cluster.

Output
----------------
Memcached crashed on one node on destination cluster ( node 33)

Error
-----------
Port server memcached on node 'ns_1@10.3.121.33' exited with status 134. Restarting. Messages: >57 tried to send an empty chunk (ignored) tap
>57 tried to send an empty chunk (ignored) tap
.
memcached: couch-kvstore/couch-kvstore.cc:1298: couchstore_error_t CouchKVStore::fetchDoc(Db*, DocInfo*, GetValue&, uint16_t, bool): Assertion `docinfo->deleted' failed

Backtrace from node 33
---------------------------------

Thread 18 (Thread 9056):
#0 0x00007f954cf6a90d in fdatasync () from /lib/libc.so.6
#1 0x00007f9548f7908f in couch_sync (handle=<value optimized out>) at src/os.c:139
#2 0x00007f9548f7570f in couchstore_commit (db=0x65ec690) at src/couch_db.c:192
#3 0x00007f9549245bf0 in CouchKVStore::setVBucketState (this=0x6640680, vbucketId=<value optimized out>, vbstate=...,
stateChanged=<value optimized out>, newfile=<value optimized out>) at couch-kvstore/couch-kvstore.cc:785
#4 0x00007f954924624d in CouchKVStore::snapshotVBuckets (this=<value optimized out>, m=<value optimized out>)
at couch-kvstore/couch-kvstore.cc:649
#5 0x00007f95491bf612 in EventuallyPersistentStore::snapshotVBuckets (this=0x6670000, priority=...) at ep.cc:922
#6 0x00007f95491cb3d1 in SnapshotVBucketsCallback::callback(Dispatcher&, std::tr1::shared_ptr<Task>) ()
from /opt/couchbase/lib/memcached/ep.so
#7 0x00007f95491af87f in Task::run (this=<value optimized out>, d=..., t=<value optimized out>) at dispatcher.hh:142
#8 0x00007f95491ae58a in Dispatcher::run (this=0x6632a80) at dispatcher.cc:169
#9 0x00007f95491aedbb in launch_dispatcher_thread (arg=0x3b) at dispatcher.cc:28
#10 0x00007f954d2149ca in start_thread () from /lib/libpthread.so.0
#11 0x00007f954cf71cdd in clone () from /lib/libc.so.6
#12 0x0000000000000000 in ?? ()

Thread 17 (Thread 9054):
#0 0x00007f954cf722d3 in epoll_wait () from /lib/libc.so.6
#1 0x00007f954dd2af36 in epoll_dispatch (base=0x65e6780, tv=<value optimized out>) at epoll.c:404
#2 0x00007f954dd16394 in event_base_loop (base=0x65e6780, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414b64 in worker_libevent (arg=0x21868e0) at daemon/thread.c:301
#4 0x00007f954d2149ca in start_thread () from /lib/libpthread.so.0
#5 0x00007f954cf71cdd in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 16 (Thread 9063):
#0 0x00007f954d21d243 in pread64 () from /lib/libpthread.so.0
#1 0x00007f9548f79190 in pread (handle=<value optimized out>, buf=0x85f8431, nbyte=8192, offset=278528) at /usr/include/bits/unistd.h:100
#2 couch_pread (handle=<value optimized out>, buf=0x85f8431, nbyte=8192, offset=278528) at src/os.c:75
#3 0x00007f9548f77e9c in load_buffer_from (handle=0x9b6fe90, buf=<value optimized out>, nbyte=8, offset=281342) at src/iobuffer.c:153
#4 buffered_pread (handle=0x9b6fe90, buf=<value optimized out>, nbyte=8, offset=281342) at src/iobuffer.c:291
#5 0x00007f9548f7648c in read_skipping_prefixes (db=0x65ec380, pos=281342, ret_ptr=<value optimized out>, header=0)
at src/couch_file_read.c:27
#6 pread_bin_internal (db=0x65ec380, pos=281342, ret_ptr=<value optimized out>, header=0) at src/couch_file_read.c:53
#7 0x00007f9548f76697 in pread_compressed (db=0x30, pos=140477489, ret_ptr=0x7f95453531b8) at src/couch_file_read.c:92
#8 0x00007f9548f73659 in btree_lookup_inner (rq=0x7f9545353360, diskpos=140477489, current=0, end=-1) at src/btree_read.c:22
#9 0x00007f9548f7379a in btree_lookup_inner (rq=0x7f9545353360, diskpos=<value optimized out>, current=0, end=1) at src/btree_read.c:47
#10 0x00007f9548f7379a in btree_lookup_inner (rq=0x7f9545353360, diskpos=<value optimized out>, current=0, end=1) at src/btree_read.c:47
#11 0x00007f9548f744fa in couchstore_changes_since (db=0x65ec380, since=<value optimized out>, options=<value optimized out>,
callback=<value optimized out>, ctx=<value optimized out>) at src/couch_db.c:641
#12 0x00007f9549241488 in CouchKVStore::loadDB (this=0x67e4000, cb=<value optimized out>, keysOnly=<value optimized out>,
vbids=<value optimized out>, options=<value optimized out>) at couch-kvstore/couch-kvstore.cc:1009
#13 0x00007f9549241cad in CouchKVStore::dumpKeys (this=0x67e4000, vbids=<value optimized out>, cb=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at couch-kvstore/couch-kvstore.cc:843
#14 0x00007f954921c015 in Warmup::keyDump (this=0x65ee1e0) at warmup.cc:400
#15 0x00007f954921c6e2 in Warmup::step (this=0x65ee1e0, d=..., t=<value optimized out>) at warmup.cc:550
#16 0x00007f954921d6b9 in WarmupStepper::callback(Dispatcher&, std::tr1::shared_ptr<Task>) () from /opt/couchbase/lib/memcached/ep.so
#17 0x00007f95491af87f in Task::run (this=<value optimized out>, d=..., t=<value optimized out>) at dispatcher.hh:142
#18 0x00007f95491ae58a in Dispatcher::run (this=0x673a540) at dispatcher.cc:169
#19 0x00007f95491aedbb in launch_dispatcher_thread (arg=0x30) at dispatcher.cc:28
#20 0x00007f954d2149ca in start_thread () from /lib/libpthread.so.0
#21 0x00007f954cf71cdd in clone () from /lib/libc.so.6
#22 0x0000000000000000 in ?? ()

Thread 15 (Thread 9050):
#0 0x00007f954cf722d3 in epoll_wait () from /lib/libc.so.6
#1 0x00007f954dd2af36 in epoll_dispatch (base=0x65e6500, tv=<value optimized out>) at epoll.c:404
#2 0x00007f954dd16394 in event_base_loop (base=0x65e6500, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414b64 in worker_libevent (arg=0x2186500) at daemon/thread.c:301
#4 0x00007f954d2149ca in start_thread () from /lib/libpthread.so.0
#5 0x00007f954cf71cdd in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 14 (Thread 9055):
#0 0x00007f954cf3569d in nanosleep () from /lib/libc.so.6
#1 0x00007f954cf6adf4 in usleep () from /lib/libc.so.6
#2 0x00007f95491ef135 in updateStatsThread (arg=<value optimized out>) at memory_tracker.cc:31
#3 0x00007f954d2149ca in start_thread () from /lib/libpthread.so.0
#4 0x00007f954cf71cdd in clone () from /lib/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 13 (Thread 9053):
#0 0x00007f954cf722d3 in epoll_wait () from /lib/libc.so.6
#1 0x00007f954dd2af36 in epoll_dispatch (base=0x65e6a00, tv=<value optimized out>) at epoll.c:404
#2 0x00007f954dd16394 in event_base_loop (base=0x65e6a00, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414b64 in worker_libevent (arg=0x21867e8) at daemon/thread.c:301
#4 0x00007f954d2149ca in start_thread () from /lib/libpthread.so.0
#5 0x00007f954cf71cdd in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 12 (Thread 9049):
#0 0x00007f954cf63a6d in read () from /lib/libc.so.6

#1 0x00007f954cefe598 in _IO_file_underflow () from /lib/libc.so.6
#2 0x00007f954cf0013e in _IO_default_uflow () from /lib/libc.so.6
#3 0x00007f954cef6ecb in getc () from /lib/libc.so.6
#4 0x00007f954c75e9c9 in check_stdin_thread (arg=0x403880) at extensions/daemon/stdin_check.c:19
#5 0x00007f954d2149ca in start_thread () from /lib/libpthread.so.0
#6 0x00007f954cf71cdd in clone () from /lib/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 11 (Thread 9051):
#0 0x00007f954cf722d3 in epoll_wait () from /lib/libc.so.6
#1 0x00007f954dd2af36 in epoll_dispatch (base=0x65e6280, tv=<value optimized out>) at epoll.c:404
#2 0x00007f954dd16394 in event_base_loop (base=0x65e6280, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414b64 in worker_libevent (arg=0x21865f8) at daemon/thread.c:301
#4 0x00007f954d2149ca in start_thread () from /lib/libpthread.so.0
#5 0x00007f954cf71cdd in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 10 (Thread 9066):
#0 0x00007f954d219bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00007f95491ceeff in SyncObject::wait (this=0x6601400) at syncobject.hh:47
#2 SyncObject::wait (this=0x6601400) at syncobject.hh:63
#3 TapConnMap::wait (this=0x6601400) at tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x6601400) at ep_engine.cc:3445
#5 0x00007f95491cefe3 in EvpNotifyPendingConns (arg=0x6601400) at ep_engine.cc:1114
#6 0x00007f954d2149ca in start_thread () from /lib/libpthread.so.0
#7 0x00007f954cf71cdd in clone () from /lib/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 9 (Thread 9065):
#0 0x00007f954d219bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00007f95491ab030 in SyncObject::wait (this=0x66a2880, d=...) at syncobject.hh:47
#2 IdleTask::run (this=0x66a2880, d=...) at dispatcher.cc:341
#3 0x00007f95491ae58a in Dispatcher::run (this=0x673b180) at dispatcher.cc:169
#4 0x00007f95491aedbb in launch_dispatcher_thread (arg=0x673b1d4) at dispatcher.cc:28
#5 0x00007f954d2149ca in start_thread () from /lib/libpthread.so.0
#6 0x00007f954cf71cdd in clone () from /lib/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 8 (Thread 9052):
#0 0x00007f954cf722d3 in epoll_wait () from /lib/libc.so.6
#1 0x00007f954dd2af36 in epoll_dispatch (base=0x65e6c80, tv=<value optimized out>) at epoll.c:404
#2 0x00007f954dd16394 in event_base_loop (base=0x65e6c80, flags=<value optimized out>) at event.c:1558

#3 0x0000000000414b64 in worker_libevent (arg=0x21866f0) at daemon/thread.c:301
#4 0x00007f954d2149ca in start_thread () from /lib/libpthread.so.0
#5 0x00007f954cf71cdd in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 7 (Thread 9048):
#0 0x00007f954cf722d3 in epoll_wait () from /lib/libc.so.6
#1 0x00007f954dd2af36 in epoll_dispatch (base=0x65e6000, tv=<value optimized out>) at epoll.c:404
#2 0x00007f954dd16394 in event_base_loop (base=0x65e6000, flags=<value optimized out>) at event.c:1558
#3 0x000000000040c315 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7920

Thread 6 (Thread 9059):
#0 0x00007f954d219bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00007f95491ab030 in SyncObject::wait (this=0x218c300, d=...) at syncobject.hh:47
#2 IdleTask::run (this=0x218c300, d=...) at dispatcher.cc:341
#3 0x00007f95491ae58a in Dispatcher::run (this=0x6633500) at dispatcher.cc:169
#4 0x00007f95491aedbb in launch_dispatcher_thread (arg=0x6633554) at dispatcher.cc:28
#5 0x00007f954d2149ca in start_thread () from /lib/libpthread.so.0
#6 0x00007f954cf71cdd in clone () from /lib/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 5 (Thread 9062):
#0 0x00007f954d219bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00007f95491ab030 in SyncObject::wait (this=0x66a2b00, d=...) at syncobject.hh:47
#2 IdleTask::run (this=0x66a2b00, d=...) at dispatcher.cc:341
#3 0x00007f95491ae58a in Dispatcher::run (this=0x673a700) at dispatcher.cc:169
#4 0x00007f95491aedbb in launch_dispatcher_thread (arg=0x673a754) at dispatcher.cc:28
#5 0x00007f954d2149ca in start_thread () from /lib/libpthread.so.0
#6 0x00007f954cf71cdd in clone () from /lib/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 4 (Thread 9060):
#0 0x00007f954d219bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00007f95491ceeff in SyncObject::wait (this=0x6600a00) at syncobject.hh:47
#2 SyncObject::wait (this=0x6600a00) at syncobject.hh:63
#3 TapConnMap::wait (this=0x6600a00) at tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x6600a00) at ep_engine.cc:3445
#5 0x00007f95491cefe3 in EvpNotifyPendingConns (arg=0x6600a00) at ep_engine.cc:1114
#6 0x00007f954d2149ca in start_thread () from /lib/libpthread.so.0
#7 0x00007f954cf71cdd in clone () from /lib/libc.so.6
#8 0x0000000000000000 in ?? ()
Thread 3 (Thread 9064):
#0 0x00007f954d21985c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00007f95491ae7c7 in SyncObject::wait (this=0x673b340) at syncobject.hh:36
#2 Dispatcher::run (this=0x673b340) at dispatcher.cc:135
#3 0x00007f95491aedbb in launch_dispatcher_thread (arg=0x673b394) at dispatcher.cc:28
#4 0x00007f954d2149ca in start_thread () from /lib/libpthread.so.0
#5 0x00007f954cf71cdd in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 2 (Thread 9058):
#0 0x00007f954d21985c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00007f95491ae7c7 in SyncObject::wait (this=0x66336c0) at syncobject.hh:36
#2 Dispatcher::run (this=0x66336c0) at dispatcher.cc:135
#3 0x00007f95491aedbb in launch_dispatcher_thread (arg=0x6633714) at dispatcher.cc:28
#4 0x00007f954d2149ca in start_thread () from /lib/libpthread.so.0
#5 0x00007f954cf71cdd in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 1 (Thread 9057):
#0 0x00007f954cebea75 in raise () from /lib/libc.so.6
#1 0x00007f954cec25c0 in abort () from /lib/libc.so.6
#2 0x00007f954ceb7941 in __assert_fail () from /lib/libc.so.6
#3 0x00007f9549244cd2 in CouchKVStore::fetchDoc (this=0x6640d00, db=0x65ec460, docinfo=0x7c8ff800, docValue=..., vbId=777, metaOnly=209)
at couch-kvstore/couch-kvstore.cc:1298
#4 0x00007f95492453df in CouchKVStore::get (this=0x6640d00, key=..., vb=<value optimized out>, cb=...) at couch-kvstore/couch-kvstore.cc:389
#5 0x00007f95491c0409 in EventuallyPersistentStore::completeBGFetch (this=0x6670000, key=<value optimized out>, vbucket=777,
rowid=18446744073709551615, cookie=<value optimized out>, init=<value optimized out>, type=BG_FETCH_METADATA) at ep.cc:1100
#6 0x00007f95491cc59a in BGFetchCallback::callback(Dispatcher&, std::tr1::shared_ptr<Task>) () from /opt/couchbase/lib/memcached/ep.so
#7 0x00007f95491af87f in Task::run (this=<value optimized out>, d=..., t=<value optimized out>) at dispatcher.hh:142
#8 0x00007f95491ae58a in Dispatcher::run (this=0x6633880) at dispatcher.cc:169
#9 0x00007f95491aedbb in launch_dispatcher_thread (arg=0x2358) at dispatcher.cc:28
#10 0x00007f954d2149ca in start_thread () from /lib/libpthread.so.0
#11 0x00007f954cf71cdd in clone () from /lib/libc.so.6
#12 0x0000000000000000 in ?? ()