System Test : Continous memcached crash after disk full, exited with status 139

2 buckets, default and saslbucket
1 ddoc, 2 views, continuos querying
30M and 70M items per bucket.

1. Seeing continuous memcached cores on node 10.6.2.45

2. Error logger shows errors as "
#5 0x00007efff9ac4e02 in logger_log (severity=EXTENSION_LOG_WARNING, client_cookie=<value optimized out>, fmt=0x7efff5a808bf "%s")
at extensions/loggers/file_logger.c:231
"

Diags at https://s3.amazonaws.com/bugdb/bug-memcached/10.6.2.45-8091-diag.txt.gz

Stack trace shows
[root@localhost data]# gdb /opt/couchbase/bin/memcached /data/core.memcached.6922
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 7772]
[New Thread 7773]
[New Thread 6930]
[New Thread 6935]
[New Thread 7774]
[New Thread 6946]
[New Thread 7776]
[New Thread 6937]
[New Thread 6944]
[New Thread 6947]
[New Thread 7775]
[New Thread 6922]
[New Thread 6931]
[New Thread 6948]
[New Thread 6933]
[New Thread 6936]
[New Thread 6934]
[New Thread 6942]
[New Thread 6945]
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000000052841a40 in ?? ()
Missing separate debuginfos, use: debuginfo-install couchbase-server-2.0.1-125.x86_64
(gdb) t a a bt

Thread 19 (Thread 0x7f21b10e7700 (LWP 6945)):
#0 0x00007f21b82ed7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f21b2b35f28 in wait (this=0x65f6090, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0x65f6090, d=...) at src/dispatcher.cc:336
#3 0x00007f21b2b385ea in Dispatcher::run (this=0x65bea80) at src/dispatcher.cc:173
#4 0x00007f21b2b38eeb in launch_dispatcher_thread (arg=0x65bea80) at src/dispatcher.cc:28
#5 0x00007f21b82e9851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f21b803711d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f21b26f7700 (LWP 6942)):
#0 0x00007f21b7ffb3bd in nanosleep () from /lib64/libc.so.6
#1 0x00007f21b8030574 in usleep () from /lib64/libc.so.6
#2 0x00007f21b2b75765 in updateStatsThread (arg=0x1d084c0) at src/memory_tracker.cc:31
#3 0x00007f21b82e9851 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f21b803711d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f21b560e700 (LWP 6934)):
#0 0x00007f21b8037713 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f21b8dfb576 in epoll_dispatch (base=0x6574280, tv=<value optimized out>) at epoll.c:404
#2 0x00007f21b8de6e44 in event_base_loop (base=0x6574280, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414504 in worker_libevent (arg=0x1d0b9f8) at daemon/thread.c:301
#4 0x00007f21b82e9851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f21b803711d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f21b420c700 (LWP 6936)):
#0 0x00007f21b8037713 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f21b8dfb576 in epoll_dispatch (base=0x6574a00, tv=<value optimized out>) at epoll.c:404
#2 0x00007f21b8de6e44 in event_base_loop (base=0x6574a00, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414504 in worker_libevent (arg=0x1d0bbe8) at daemon/thread.c:301
#4 0x00007f21b82e9851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f21b803711d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f21b600f700 (LWP 6933)):
#0 0x00007f21b8037713 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f21b8dfb576 in epoll_dispatch (base=0x6574500, tv=<value optimized out>) at epoll.c:404
#2 0x00007f21b8de6e44 in event_base_loop (base=0x6574500, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414504 in worker_libevent (arg=0x1d0b900) at daemon/thread.c:301
#4 0x00007f21b82e9851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f21b803711d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f21af2e4700 (LWP 6948)):
#0 0x00007f21b82ed7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f21b2b54d2f in wait (this=0x65c8000) at src/syncobject.hh:58
#2 wait (this=0x65c8000) at src/syncobject.hh:74
#3 wait (this=0x65c8000) at src/tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x65c8000) at src/ep_engine.cc:3420
#5 0x00007f21b2b54e13 in EvpNotifyPendingConns (arg=0x65c8000) at src/ep_engine.cc:1145
#6 0x00007f21b82e9851 in start_thread () from /lib64/libpthread.so.0
--Type <return> to continue, or q <return> to quit--
#7 0x00007f21b803711d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f21b6c1c700 (LWP 6931)):
#0 0x00007f21b82ed7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f21b6c204d6 in logger_thead_main (arg=0x1d08040) at extensions/loggers/file_logger.c:368
#2 0x00007f21b82e9851 in start_thread () from /lib64/libpthread.so.0
#3 0x00007f21b803711d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f21b9436720 (LWP 6922)):
#0 0x00007f21b8037713 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f21b8dfb576 in epoll_dispatch (base=0x6574000, tv=<value optimized out>) at epoll.c:404
#2 0x00007f21b8de6e44 in event_base_loop (base=0x6574000, flags=<value optimized out>) at event.c:1558
#3 0x0000000000409742 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7918

Thread 11 (Thread 0x7f21acae0700 (LWP 7775)):
#0 0x00007f21b82ed7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f21b2b35f28 in wait (this=0x149ffb90, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0x149ffb90, d=...) at src/dispatcher.cc:336
#3 0x00007f21b2b385ea in Dispatcher::run (this=0x76348c0) at src/dispatcher.cc:173
#4 0x00007f21b2b38eeb in launch_dispatcher_thread (arg=0x76348c0) at src/dispatcher.cc:28
#5 0x00007f21b82e9851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f21b803711d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f21afce5700 (LWP 6947)):
#0 0x00007f21b82ed7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f21b2b35f28 in wait (this=0x65f6240, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0x65f6240, d=...) at src/dispatcher.cc:336
#3 0x00007f21b2b385ea in Dispatcher::run (this=0x65bf6c0) at src/dispatcher.cc:173
#4 0x00007f21b2b38eeb in launch_dispatcher_thread (arg=0x65bf6c0) at src/dispatcher.cc:28
#5 0x00007f21b82e9851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f21b803711d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f21b1ae8700 (LWP 6944)):
#0 0x00007f21b26fb41e in snappy::SnappyDecompressor::RefillTag (this=0x7f21b1ae6f70) at snappy.cc:732
#1 0x00007f21b26fc98a in DecompressAllTags<snappy::SnappyArrayWriter> (compressed=<value optimized out>, uncompressed=0xf432800 "\200_-\n")
at snappy.cc:661
#2 InternalUncompress<snappy::SnappyArrayWriter> (compressed=<value optimized out>, uncompressed=0xf432800 "\200_-\n") at snappy.cc:788
#3 snappy::RawUncompress (compressed=<value optimized out>, uncompressed=0xf432800 "\200_-\n") at snappy.cc:956
#4 0x00007f21b26fcc62 in snappy::RawUncompress (compressed=<value optimized out>, n=<value optimized out>, uncompressed=<value optimized out>)
at snappy.cc:951
#5 0x00007f21b26fd49e in snappy_uncompress (compressed=0xa2d5180 "w", <incomplete sequence \310>, compressed_length=115,
uncompressed=0xf432800 "\200_-\n", uncompressed_length=0x7f21b1ae7070) at snappy-c.cc:58
#6 0x00007f21b2908fc6 in pread_compressed (db=<value optimized out>, pos=<value optimized out>, ret_ptr=0x7f21b1ae7118) at src/couch_file_read.c:108
#7 0x00007f21b290496a in modify_node (rq=0x7f21b1ae71f0, nptr=0x14a1cea40, start=0, end=8, dst=0x399480010) at src/btree_modify.c:317
#8 0x00007f21b2905317 in modify_btree (rq=0x7f21b1ae71f0, root=0x14a1cea40, errcode=0x7f21b1ae72fc) at src/btree_modify.c:607
#9 0x00007f21b29080d5 in update_indexes (db=0x657bea0, seqs=0xd4d5d810, seqvals=0xd4d5d890, ids=<value optimized out>, idvals=0xd4d5d8d0, numdocs=4)
at src/couch_save.c:204
#10 0x00007f21b2908bb0 in couchstore_save_documents (db=0x657bea0, docs=0x3343e4cf8, infos=<value optimized out>, numdocs=4, options=1)
at src/couch_save.c:359
--Type <return> to continue, or q <return> to quit--
#11 0x00007f21b2bb2949 in CouchKVStore::saveDocs (this=0x65e0000, vbid=770, rev=15, docs=0x213286320, docinfos=0x2132865c0, docCount=4)
at src/couch-kvstore/couch-kvstore.cc:1557
#12 0x00007f21b2bb30c1 in CouchKVStore::commit2couchstore (this=0x65e0000) at src/couch-kvstore/couch-kvstore.cc:1492
#13 0x00007f21b2bb33d9 in queueItem (this=0x65e0000, itm=..., cb=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:1625
#14 CouchKVStore::set (this=0x65e0000, itm=..., cb=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:344
#15 0x00007f21b2b410d4 in EventuallyPersistentStore::flushOneDelOrSet (this=0x65c4480, qi=...,
rejectQueue=std::queue wrapping: std::deque with 0 elements, vb=...) at src/ep.cc:2420
#16 0x00007f21b2b4139b in EventuallyPersistentStore::flushOne (this=0x65c4480, queue=<value optimized out>,
rejectQueue=std::queue wrapping: std::deque with 0 elements, vb=...) at src/ep.cc:2468
#17 0x00007f21b2b44e95 in EventuallyPersistentStore::flushVBQueue (this=0x65c4480, vb=...,
vb_queue=std::queue wrapping: std::deque with 1 elements =

, vbid=771, data_age=0) at src/ep.cc:2022
#18 0x00007f21b2b460ec in EventuallyPersistentStore::flushOutgoingQueue (this=0x65c4480, flushQueue=0x65c4748, flushPhase=@0x65c2570, nextVbid=@0x65c2578)
at src/ep.cc:1964
#19 0x00007f21b2b6f7ec in Flusher::doFlush (this=0x65c2480) at src/flusher.cc:245
#20 0x00007f21b2b70625 in Flusher::step (this=0x65c2480, d=..., tid=...) at src/flusher.cc:158
#21 0x00007f21b2b385ea in Dispatcher::run (this=0x65bec40) at src/dispatcher.cc:173
#22 0x00007f21b2b38eeb in launch_dispatcher_thread (arg=0x65bec40) at src/dispatcher.cc:28
#23 0x00007f21b82e9851 in start_thread () from /lib64/libpthread.so.0
#24 0x00007f21b803711d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f21b380b700 (LWP 6937)):
#0 0x00007f21b8037713 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f21b8dfb576 in epoll_dispatch (base=0x6574780, tv=<value optimized out>) at epoll.c:404
#2 0x00007f21b8de6e44 in event_base_loop (base=0x6574780, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414504 in worker_libevent (arg=0x1d0bce0) at daemon/thread.c:301
#4 0x00007f21b82e9851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f21b803711d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f21ac0df700 (LWP 7776)):
#0 0x00007f21b82ed7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f21b2b54d2f in wait (this=0x7636000) at src/syncobject.hh:58
#2 wait (this=0x7636000) at src/syncobject.hh:74
#3 wait (this=0x7636000) at src/tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x7636000) at src/ep_engine.cc:3420
#5 0x00007f21b2b54e13 in EvpNotifyPendingConns (arg=0x7636000) at src/ep_engine.cc:1145
#6 0x00007f21b82e9851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f21b803711d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f21b06e6700 (LWP 6946)):
#0 0x00007f21b82ed7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f21b2b35f28 in wait (this=0x65f62d0, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0x65f62d0, d=...) at src/dispatcher.cc:336
#3 0x00007f21b2b385ea in Dispatcher::run (this=0x65bf880) at src/dispatcher.cc:173
#4 0x00007f21b2b38eeb in launch_dispatcher_thread (arg=0x65bf880) at src/dispatcher.cc:28
#5 0x00007f21b82e9851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f21b803711d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f21ad4e1700 (LWP 7774)):
#0 0x00007f21b82ed7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
--Type <return> to continue, or q <return> to quit--
#1 0x00007f21b2b35f28 in wait (this=0x149ffc20, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0x149ffc20, d=...) at src/dispatcher.cc:336
#3 0x00007f21b2b385ea in Dispatcher::run (this=0x7634a80) at src/dispatcher.cc:173
#4 0x00007f21b2b38eeb in launch_dispatcher_thread (arg=0x7634a80) at src/dispatcher.cc:28
#5 0x00007f21b82e9851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f21b803711d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f21b4c0d700 (LWP 6935)):
#0 0x00007f21b8037713 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f21b8dfb576 in epoll_dispatch (base=0x6574c80, tv=<value optimized out>) at epoll.c:404
#2 0x00007f21b8de6e44 in event_base_loop (base=0x6574c80, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414504 in worker_libevent (arg=0x1d0baf0) at daemon/thread.c:301
#4 0x00007f21b82e9851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f21b803711d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f21b7831700 (LWP 6930)):
#0 0x00007f21b8029e1d in read () from /lib64/libc.so.6
#1 0x00007f21b7fc0798 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x00007f21b7fc229e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x00007f21b7fb6a8a in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x00007f21b7fb58e9 in fgets () from /lib64/libc.so.6
#5 0x00007f21b7832939 in check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
#6 0x00007f21b82e9851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f21b803711d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f21adee2700 (LWP 7773)):
#0 0x00007f21b82ed7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f21b2b35f28 in wait (this=0x149ffcb0, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0x149ffcb0, d=...) at src/dispatcher.cc:336
#3 0x00007f21b2b385ea in Dispatcher::run (this=0x7634c40) at src/dispatcher.cc:173
#4 0x00007f21b2b38eeb in launch_dispatcher_thread (arg=0x7634c40) at src/dispatcher.cc:28
#5 0x00007f21b82e9851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f21b803711d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f21ae8e3700 (LWP 7772)):
#0 0x0000000052841a40 in ?? ()
#1 0x00000000065c6288 in ?? ()
#2 0x00007f21b2b3bf15 in TransactionContext::commit (this=0x144de900) at src/ep.cc:2789
#3 0x00007f21b2b46200 in EventuallyPersistentStore::flushOutgoingQueue (this=0x65c5f80, flushQueue=0x65c6248, flushPhase=@0x65c3650, nextVbid=@0x65c3658)
at src/ep.cc:1975
#4 0x00007f21b2b6f7ec in Flusher::doFlush (this=0x65c3560) at src/flusher.cc:245
#5 0x00007f21b2b70625 in Flusher::step (this=0x65c3560, d=..., tid=...) at src/flusher.cc:158
#6 0x00007f21b2b385ea in Dispatcher::run (this=0x7634e00) at src/dispatcher.cc:173
#7 0x00007f21b2b38eeb in launch_dispatcher_thread (arg=0x7634e00) at src/dispatcher.cc:28
#8 0x00007f21b82e9851 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f21b803711d in clone () from /lib64/libc.so.6