memcached exited with status 139 after restart couchbase server in failover add back test.

Create a cluster of 3 nodes couchbase server 2.0.0r-266 on ec2.
Use mcsoda to load 200000 items to cluster.
Do some failover and add back test
Shutdown couchbase server in all nodes and restart again
See lost item at master node
Check log, see memcached exit with status 139

Thread 12 (process 7070):
#0 0x00002aaaabce4018 in epoll_wait () from /lib64/libc.so.6
#1 0x00002aaaaaeeec28 in epoll_dispatch (base=0x6ce000, tv=<value optimized out>) at epoll.c:404
#2 0x00002aaaaaedda4c in event_base_loop (base=0x6ce000, flags=0) at event.c:1558
#3 0x000000000040930b in main (argc=15, argv=0x7fff7c93d328) at daemon/memcached.c:7531
Current language: auto; currently c++

Thread 11 (process 7078):
#0 0x00002aaaabcd5f3b in read () from /lib64/libc.so.6
#1 0x00002aaaabc7cc07 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x00002aaaabc7d5ce in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x00002aaaabc78e8b in getc () from /lib64/libc.so.6
#4 0x00002aaaac478875 in check_stdin_thread (arg=0x403610) at extensions/daemon/stdin_check.c:19
#5 0x00002aaaab9fb617 in start_thread () from /lib64/libpthread.so.0
#6 0x00002aaaabce3c2d in clone () from /lib64/libc.so.6

Thread 10 (process 7079):
#0 0x00002aaaabcaa0b1 in nanosleep () from /lib64/libc.so.6
#1 0x00002aaaabca9ed4 in sleep () from /lib64/libc.so.6
#2 0x0000000000415068 in check_isasl_db_thread (arg=<value optimized out>) at daemon/isasl.c:233
#3 0x00002aaaab9fb617 in start_thread () from /lib64/libpthread.so.0
#4 0x00002aaaabce3c2d in clone () from /lib64/libc.so.6

Thread 9 (process 7080):
#0 0x00002aaaabce4018 in epoll_wait () from /lib64/libc.so.6
#1 0x00002aaaaaeeec28 in epoll_dispatch (base=0x6ce500, tv=<value optimized out>) at epoll.c:404
#2 0x00002aaaaaedda4c in event_base_loop (base=0x6ce500, flags=0) at event.c:1558
#3 0x00000000004134a4 in worker_libevent (arg=0x6b0500) at daemon/thread.c:304
#4 0x00002aaaab9fb617 in start_thread () from /lib64/libpthread.so.0
#5 0x00002aaaabce3c2d in clone () from /lib64/libc.so.6

Thread 8 (process 7081):
#0 0x00002aaaabce4018 in epoll_wait () from /lib64/libc.so.6
#1 0x00002aaaaaeeec28 in epoll_dispatch (base=0x6ce280, tv=<value optimized out>) at epoll.c:404
#2 0x00002aaaaaedda4c in event_base_loop (base=0x6ce280, flags=0) at event.c:1558
#3 0x00000000004134a4 in worker_libevent (arg=0x6b0600) at daemon/thread.c:304
#4 0x00002aaaab9fb617 in start_thread () from /lib64/libpthread.so.0
#5 0x00002aaaabce3c2d in clone () from /lib64/libc.so.6

Thread 7 (process 7082):
#0 0x00002aaaabcca937 in sched_yield () from /lib64/libc.so.6
#1 0x00002aaaac98795b in SpinLock::acquire (this=0x76c078) at atomic.hh:236
#2 0x00002aaaac987986 in SpinLockHolder::lock (this=0x43202c30) at atomic.hh:263
#3 0x00002aaaac9879b4 in SpinLockHolder (this=0x43202c30, theLock=0x76c078) at atomic.hh:255
#4 0x00002aaaac9b4ad2 in RCPtr<VBucketHolder>::gimme (this=0x76c070) at atomic.hh:375
#5 0x00002aaaac9b4b85 in RCPtr (this=0x43202c90, other=@0x76c070) at atomic.hh:313
#6 0x00002aaaac9b70b6 in VBucketMap::getBucket (this=0x76c070, id=218) at vbucket.hh:510
#7 0x00002aaaac9a00f4 in EventuallyPersistentStore::queueDirty (this=0x76c000, key=@0x1439a90, vbid=218, op=queue_op_set, value=
{value = {<Atomic<Blob*>> =

, <No data fields>}, lock = {lock = 0}}, flags=0, exptime=0, cas=79495244093595,
seqno=1, rowid=-1, tapBackfill=true) at ep.cc:2447
#8 0x00002aaaac9a132f in EventuallyPersistentStore::addTAPBackfillItem (this=0x76c000, itm=@0x1439a80, meta=false) at ep.cc:997
#9 0x00002aaaac9d20f3 in EventuallyPersistentEngine::tapNotify (this=0x74a000, cookie=0x6e3b88, engine_specific=0x144f020, nengine=22,
tap_flags=0, tap_event=TAP_MUTATION, tap_seqno=4316, key=0x144f03e, nkey=23, flags=0, exptime=0, data=0x144f055, ndata=453, vbucket=218)
at ep_engine.cc:2026
#10 0x00002aaaac9d37b0 in EvpTapNotify (handle=0x74a000, cookie=0x6e3b88, engine_specific=0x144f020, nengine=22, ttl=254 '?', tap_flags=0,
tap_event=TAP_MUTATION, tap_seqno=4316, key=0x144f03e, nkey=23, flags=0, exptime=0, cas=13576151416751, data=0x144f055, ndata=453,
vbucket=218) at ep_engine.cc:960
#11 0x00002aaaac67d56b in bucket_tap_notify (handle=0x2aaaac881ba0, cookie=0x6e3b88, engine_specific=0x144f020, nengine=22, ttl=254 '?',
tap_flags=0, tap_event=TAP_MUTATION, tap_seqno=4316, key=0x144f03e, nkey=23, flags=0, exptime=0, cas=13576151416751, data=0x144f055,
ndata=453, vbucket=218) at bucket_engine.c:1912
#12 0x0000000000406a11 in process_bin_tap_packet (event=21207680, c=0x6e3b88) at daemon/memcached.c:2744
#13 0x0000000000410c05 in complete_nread_binary (c=0x76c078) at daemon/memcached.c:3427
#14 0x00000000004118a1 in conn_nread (c=0x6e3b88) at daemon/memcached.c:3509
#15 0x0000000000407564 in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x6e3b88) at daemon/memcached.c:5620
#16 0x00002aaaaaedddf9 in event_base_loop (base=0x6cec80, flags=0) at event.c:1308
#17 0x00000000004134a4 in worker_libevent (arg=0x6b0700) at daemon/thread.c:304
#18 0x00002aaaab9fb617 in start_thread () from /lib64/libpthread.so.0
#19 0x00002aaaabce3c2d in clone () from /lib64/libc.so.6

Thread 6 (process 7083):
#0 0x00002aaaabcca937 in sched_yield () from /lib64/libc.so.6
#1 0x00002aaaac98795b in SpinLock::acquire (this=0x76c078) at atomic.hh:236
#2 0x00002aaaac987986 in SpinLockHolder::lock (this=0x43c03da0) at atomic.hh:263
#3 0x00002aaaac9879b4 in SpinLockHolder (this=0x43c03da0, theLock=0x76c078) at atomic.hh:255
#4 0x00002aaaac9b4ad2 in RCPtr<VBucketHolder>::gimme (this=0x76c070) at atomic.hh:375
#5 0x00002aaaac9b4b85 in RCPtr (this=0x43c03e00, other=@0x76c070) at atomic.hh:313
#6 0x00002aaaac9b70b6 in VBucketMap::getBucket (this=0x76c070, id=132) at vbucket.hh:510
#7 0x00002aaaaca0c6e1 in TapConsumer::isBackfillPhase (this=0x6d3180, vbucket=132) at tapconnection.cc:1139
#8 0x00002aaaac9d20d4 in EventuallyPersistentEngine::tapNotify (this=0x74a000, cookie=0x1fb82c8, engine_specific=0x6f0230, nengine=22,
tap_flags=0, tap_event=TAP_MUTATION, tap_seqno=3643, key=0x6f024e, nkey=23, flags=0, exptime=0, data=0x6f0265, ndata=438, vbucket=132)
at ep_engine.cc:2025
#9 0x00002aaaac9d37b0 in EvpTapNotify (handle=0x74a000, cookie=0x1fb82c8, engine_specific=0x6f0230, nengine=22, ttl=254 '?', tap_flags=0,
tap_event=TAP_MUTATION, tap_seqno=3643, key=0x6f024e, nkey=23, flags=0, exptime=0, cas=445054831872, data=0x6f0265, ndata=438,
vbucket=132) at ep_engine.cc:960
#10 0x00002aaaac67d56b in bucket_tap_notify (handle=0x2aaaac881ba0, cookie=0x1fb82c8, engine_specific=0x6f0230, nengine=22, ttl=254 '?',
tap_flags=0, tap_event=TAP_MUTATION, tap_seqno=3643, key=0x6f024e, nkey=23, flags=0, exptime=0, cas=445054831872, data=0x6f0265,
ndata=438, vbucket=132) at bucket_engine.c:1912
#11 0x0000000000406a11 in process_bin_tap_packet (event=7156096, c=0x1fb82c8) at daemon/memcached.c:2744
#12 0x0000000000410c05 in complete_nread_binary (c=0x76c078) at daemon/memcached.c:3427
#13 0x00000000004118a1 in conn_nread (c=0x1fb82c8) at daemon/memcached.c:3509
#14 0x0000000000407564 in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x1fb82c8) at daemon/memcached.c:5620
#15 0x00002aaaaaedddf9 in event_base_loop (base=0x6cea00, flags=0) at event.c:1308
#16 0x00000000004134a4 in worker_libevent (arg=0x6b0800) at daemon/thread.c:304
#17 0x00002aaaab9fb617 in start_thread () from /lib64/libpthread.so.0
#18 0x00002aaaabce3c2d in clone () from /lib64/libc.so.6

Thread 5 (process 7084):
#0 0x00002aaaabcca937 in sched_yield () from /lib64/libc.so.6
#1 0x00002aaaac98795b in SpinLock::acquire (this=0x76c078) at atomic.hh:236
#2 0x00002aaaac987986 in SpinLockHolder::lock (this=0x44605070) at atomic.hh:263
#3 0x00002aaaac9879b4 in SpinLockHolder (this=0x44605070, theLock=0x76c078) at atomic.hh:255
#4 0x00002aaaac9b4ad2 in RCPtr<VBucketHolder>::gimme (this=0x76c070) at atomic.hh:375
#5 0x00002aaaac9b4b85 in RCPtr (this=0x446050d0, other=@0x76c070) at atomic.hh:313
#6 0x00002aaaac9b70b6 in VBucketMap::getBucket (this=0x76c070, id=9) at vbucket.hh:510
#7 0x00002aaaac99e154 in EventuallyPersistentStore::getVBucket (this=0x76c000, vbucket=9) at ep.cc:755
#8 0x00002aaaac9a6d32 in EventuallyPersistentStore::get (this=0x76c000, key=@0x265eef8, vbucket=9, cookie=0x6e2dc8, queueBG=false,
honorStates=false) at ep.cc:1294
#9 0x00002aaaac9edc89 in EventuallyPersistentEngine::doWalkTapQueue (this=0x74a000, cookie=0x6e2dc8, itm=0x446057a8, es=0x446057a0,
nes=0x446057bc, ttl=0x446057bf "?", flags=0x446057ba, seqno=0x446057b4, vbucket=0x446057b8, connection=0x2364000, retry=@0x44605601)
at ep_engine.cc:1603
#10 0x00002aaaac9d1241 in EventuallyPersistentEngine::walkTapQueue (this=0x74a000, cookie=0x6e2dc8, itm=0x446057a8, es=0x446057a0,
nes=0x446057bc, ttl=0x446057bf "?", flags=0x446057ba, seqno=0x446057b4, vbucket=0x446057b8) at ep_engine.cc:1762
#11 0x00002aaaac9d390e in EvpTapIterator (handle=0x74a000, cookie=0x6e2dc8, itm=0x446057a8, es=0x446057a0, nes=0x446057bc,
ttl=0x446057bf "?", flags=0x446057ba, seqno=0x446057b4, vbucket=0x446057b8) at ep_engine.cc:969
#12 0x00002aaaac67d660 in bucket_tap_iterator_shim (handle=0x2aaaac881ba0, cookie=0x6e2dc8, itm=0x446057a8, engine_specific=0x446057a0,
nengine_specific=0x446057bc, ttl=0x446057bf "?", flags=0x446057ba, seqno=0x446057b4, vbucket=0x446057b8) at bucket_engine.c:1941
#13 0x000000000040fad2 in conn_ship_log (c=0x6e2dc8) at daemon/memcached.c:2378
#14 0x0000000000407556 in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x6e2dc8) at daemon/memcached.c:5620
#15 0x00002aaaaaedddf9 in event_base_loop (base=0x6ce780, flags=0) at event.c:1308
#16 0x00000000004134a4 in worker_libevent (arg=0x6b0900) at daemon/thread.c:304
#17 0x00002aaaab9fb617 in start_thread () from /lib64/libpthread.so.0
#18 0x00002aaaabce3c2d in clone () from /lib64/libc.so.6

Thread 4 (process 7105):
#0 0x00002aaaabc8c634 in _wordcopy_bwd_dest_aligned () from /lib64/libc.so.6
#1 0x00002aaaabc8a92b in memmove () from /lib64/libc.so.6
#2 0x00002aaaaca3c398 in MemcachedEngine::processInput (this=0x762100) at mc-kvstore/mc-engine.cc:961
#3 0x00002aaaaca3c547 in MemcachedEngine::wait (this=0x762100) at mc-kvstore/mc-engine.cc:1033
#4 0x00002aaaaca3d28e in MemcachedEngine::tap (this=0x762100, cb={_M_ptr = 0x45a07ac0, _M_refcount = {_M_pi = 0x762100}})
at mc-kvstore/mc-engine.cc:1266
#5 0x00002aaaaca4243f in MCKVStore::dump (this=0x6af840, cb={_M_ptr = 0x45a07be0, _M_refcount = {_M_pi = 0x6af840}})
at mc-kvstore/mc-kvstore.cc:150
#6 0x00002aaaac99fd33 in EventuallyPersistentStore::warmup (this=0x76c000, state=@0x7472d0, keysOnly=false) at ep.cc:2528
#7 0x00002aaaac9f3e3c in Flusher::step (this=0x747240, d=@0x755000, tid={_M_ptr = 0x45a07eb0, _M_refcount = {_M_pi = 0x755000}})
at flusher.cc:210
#8 0x00002aaaac9f4487 in FlusherStepper::callback (this=0x6aecc0, d=@0x755000, t={_M_ptr = 0x45a07f10, _M_refcount = {_M_pi = 0x755000}})
at flusher.cc:23
#9 0x00002aaaac99b56a in Task::run (this=0x759280, d=@0x755000, t={_M_ptr = 0x45a08030, _M_refcount = {_M_pi = 0x755000}})
at dispatcher.hh:139
#10 0x00002aaaac99a118 in Dispatcher::run (this=0x755000) at dispatcher.cc:119
#11 0x00002aaaac99a678 in launch_dispatcher_thread (arg=0x755000) at dispatcher.cc:28
#12 0x00002aaaab9fb617 in start_thread () from /lib64/libpthread.so.0
#13 0x00002aaaabce3c2d in clone () from /lib64/libc.so.6

Thread 3 (process 7106):
#0 0x00002aaaab9fff70 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaac99cad8 in SyncObject::wait (this=0x783c50, tv=@0x759088) at syncobject.hh:42
#2 0x00002aaaac999013 in IdleTask::run (this=0x759080, d=@0x783c40) at dispatcher.cc:244
#3 0x00002aaaac99a118 in Dispatcher::run (this=0x783c40) at dispatcher.cc:119
#4 0x00002aaaac99a678 in launch_dispatcher_thread (arg=0x783c40) at dispatcher.cc:28
#5 0x00002aaaab9fb617 in start_thread () from /lib64/libpthread.so.0
#6 0x00002aaaabce3c2d in clone () from /lib64/libc.so.6

Thread 2 (process 7107):
#0 0x00002aaaab9fff70 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaac99cad8 in SyncObject::wait (this=0x6d0200, tv=@0x46e0a060) at syncobject.hh:42
#2 0x00002aaaac9e7422 in SyncObject::wait (this=0x6d0200, secs=1) at syncobject.hh:58
#3 0x00002aaaac9e745d in TapConnMap::wait (this=0x6d0200, howlong=1) at tapconnmap.hh:209
#4 0x00002aaaac9ca957 in EventuallyPersistentEngine::notifyTapIoThread (this=0x74a000) at ep_engine.cc:3398
#5 0x00002aaaac9d3ba2 in EvpNotifyTapIo (arg=0x74a000) at ep_engine.cc:1037
#6 0x00002aaaab9fb617 in start_thread () from /lib64/libpthread.so.0
#7 0x00002aaaabce3c2d in clone () from /lib64/libc.so.6

Thread 1 (process 7104):
#0 0x00002aaaaca408d7 in TapResponseHandler::request (this=0x78d8c0, req=0xcce000) at mc-kvstore/mc-engine.cc:434
#1 0x00002aaaaca3ba25 in MemcachedEngine::handleRequest (this=0x762100, req=0xcce000) at mc-kvstore/mc-engine.cc:631
#2 0x00002aaaaca3c323 in MemcachedEngine::processInput (this=0x762100) at mc-kvstore/mc-engine.cc:951
#3 0x00002aaaaca3c5cc in MemcachedEngine::maybeProcessInput (this=0x762100) at mc-kvstore/mc-engine.cc:924
#4 0x00002aaaaca3c814 in MemcachedEngine::sendCommand (this=0x762100, rh=0x1af5d00) at mc-kvstore/mc-engine.cc:829
#5 0x00002aaaaca3d015 in MemcachedEngine::tap (this=0x762100, vbids=@0x45006db0, full=true, cb=
{_M_ptr = 0x45006e00, _M_refcount = {_M_pi = 0x2a01ac998835}}) at mc-kvstore/mc-engine.cc:1326
#6 0x00002aaaaca422b6 in MCKVStore::dump (this=0x6af840, vb=0, cb={_M_ptr = 0x45006ea0, _M_refcount = {_M_pi = 0x2418000}})
at mc-kvstore/mc-kvstore.cc:161
#7 0x00002aaaac9840d4 in BackfillDiskLoad::callback (this=0x1bde180, d=@0x783e00, t={_M_ptr = 0x45006f10, _M_refcount = {_M_pi = 0x783e00}})
at backfill.cc:56
#8 0x00002aaaac99b56a in Task::run (this=0x14c6780, d=@0x783e00, t={_M_ptr = 0x45007030, _M_refcount = {_M_pi = 0x783e00}})
at dispatcher.hh:139
#9 0x00002aaaac99a118 in Dispatcher::run (this=0x783e00) at dispatcher.cc:119
#10 0x00002aaaac99a678 in launch_dispatcher_thread (arg=0x783e00) at dispatcher.cc:28
#11 0x00002aaaab9fb617 in start_thread () from /lib64/libpthread.so.0
#12 0x00002aaaabce3c2d in clone () from /lib64/libc.so.6
--------------------------------------------------------------------------------
Module information:
/opt/couchbase/lib/memcached/libmemcached_utilities.so.0:
lrwxrwxrwx 1 bin bin 31 Nov 30 21:57 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0
13c89aae52d83681cef2a67531dbbc65 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
/opt/couchbase/lib/libevent-2.0.so.5:
lrwxrwxrwx 1 bin bin 21 Nov 30 21:57 /opt/couchbase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0
3a4104b298d23d059d6b2fbc5746ef98 /opt/couchbase/lib/libevent-2.0.so.5
/lib64/libdl.so.2:
lrwxrwxrwx 1 root root 12 Dec 28 2010 /lib64/libdl.so.2 -> libdl-2.5.so
42da5b9baf02f0dbfd22a0e5dc551ab1 /lib64/libdl.so.2
/lib64/libm.so.6:
lrwxrwxrwx 1 root root 11 Dec 28 2010 /lib64/libm.so.6 -> libm-2.5.so
c66f87d26bf6454880f397ff9def80ac /lib64/libm.so.6
/lib64/librt.so.1:
lrwxrwxrwx 1 root root 12 Dec 28 2010 /lib64/librt.so.1 -> librt-2.5.so
3abae2f6e2d27511ebd853ec5819d6e6 /lib64/librt.so.1
/opt/couchbase/lib/libtcmalloc_minimal.so.0:
lrwxrwxrwx 1 bin bin 28 Nov 30 21:57 /opt/couchbase/lib/libtcmalloc_minimal.so.0 -> libtcmalloc_minimal.so.0.1.0
4c88156c56aabbd2a6474b31a6fe21ef /opt/couchbase/lib/libtcmalloc_minimal.so.0
/lib64/libpthread.so.0:
lrwxrwxrwx 1 root root 17 Dec 28 2010 /lib64/libpthread.so.0 -> libpthread-2.5.so
4ae5c4594f5b1181af9cc3e79f88c739 /lib64/libpthread.so.0
/lib64/libc.so.6:
lrwxrwxrwx 1 root root 11 Dec 28 2010 /lib64/libc.so.6 -> libc-2.5.so
493d8a405542e7244fe84f2d60e0582e /lib64/libc.so.6
/lib64/ld-linux-x86-64.so.2:
lrwxrwxrwx 1 root root 9 Dec 28 2010 /lib64/ld-linux-x86-64.so.2 -> ld-2.5.so
6087211498c7c29096b74f8b6491c686 /lib64/ld-linux-x86-64.so.2
/usr/lib64/libstdc++.so.6:
lrwxrwxrwx 1 root root 18 Dec 28 2010 /usr/lib64/libstdc+.so.6 -> libstdc+.so.6.0.8
51c64250a754e77d9d55588febe560dc /usr/lib64/libstdc++.so.6
/lib64/libgcc_s.so.1:
lrwxrwxrwx 1 root root 28 Dec 28 2010 /lib64/libgcc_s.so.1 -> libgcc_s-4.1.2-20080825.so.1
f60cde42a19e5b0628c2c8e53f878629 /lib64/libgcc_s.so.1
/opt/couchbase/lib/memcached/stdin_term_handler.so:
-rwxr-xr-x 1 bin bin 15410 Nov 29 19:28 /opt/couchbase/lib/memcached/stdin_term_handler.so
61e9fec1f8429c34a8a105df0c928631 /opt/couchbase/lib/memcached/stdin_term_handler.so
/opt/couchbase/lib/memcached/bucket_engine.so:
lrwxrwxrwx 1 bin bin 22 Nov 30 21:57 /opt/couchbase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0
9f536ac339f17a605f874c4739c79fae /opt/couchbase/lib/memcached/bucket_engine.so
/opt/couchbase/lib/memcached/ep.so:
lrwxrwxrwx 1 bin bin 11 Nov 30 21:57 /opt/couchbase/lib/memcached/ep.so -> ep.so.0.0.0
def49ed5d17ed413302d42faee6b5a24 /opt/couchbase/lib/memcached/ep.so
1.7.2_190_g160cb1d
CEP engine v1.7.2_190_g160cb1d