[longevity] memcahced crash during warmup in large cluster test

Create 6 nodes cluster running couchbase server 2.0.0-1458, 2.0.0-1471 and 2.0.0-1482
with IP
10.3.121.13
10.3.121.14
10.3.121.15
10.3.121.16
10.3.121.17
10.3.121.20

Load 45 million items to default bucket and keep load running with mutate, delete and expire items.
Do swap rebalance in and out node during logevity test.
Then node 20 was down and memcached crashed several times during warming up.

Basic crash dump analysis of /data/tmp/core.memcached.16479.

Please send the file to support@couchbase.com

--------------------------------------------------------------------------------
File information:
-rwxr-xr-x. 1 bin bin 1628447 Jul 19 16:25 /opt/couchbase/bin/memcached
754fb25e44f568841832d1e7111f766e /opt/couchbase/bin/memcached
memcached 1.4.4_562_g0fe284e
memcached 1.4.4_562_g0fe284e
VERSION 1.4.4_562_g0fe284e
rw------. 1 couchbase couchbase 2540474368 Jul 24 09:27 /data/tmp/core.memcached.16479
ad2bfbd8a5133c5126e305b1a65b2aed /data/tmp/core.memcached.16479
version1.4.4_562_g0fe284e
--------------------------------------------------------------------------------
Core file callstacks:
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 16496]
[New Thread 16499]
[New Thread 16498]
[New Thread 16491]
[New Thread 16497]
[New Thread 16486]
[New Thread 16492]
[New Thread 16479]
[New Thread 16494]
[New Thread 16490]
[New Thread 16493]
[New Thread 16489]
[New Thread 16495]
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0 0x0000003971e32885 in raise () from /lib64/libc.so.6

Thread 13 (Thread 0x7fee8bdf6700 (LWP 16495)):
#0 0x000000397260b75b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fee8ce9b3d0 in wait (this=0x54bc000, d=...) at syncobject.hh:47
#2 IdleTask::run (this=0x54bc000, d=...) at dispatcher.cc:295
#3 0x00007fee8ce9d9c0 in Dispatcher::run (this=0x54a0c40) at dispatcher.cc:123
#4 0x00007fee8ce9e24b in launch_dispatcher_thread (arg=0x54a0c40) at dispatcher.cc:28
#5 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7fee90441700 (LWP 16489)):
#0 0x0000003971ee62c3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fee914ce576 in epoll_dispatch (base=0x5456500, tv=<value optimized out>) at epoll.c:404
#2 0x00007fee914b9e44 in event_base_loop (base=0x5456500, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414484 in worker_libevent (arg=0xff2500) at daemon/thread.c:301
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fee8dc3d700 (LWP 16493)):
#0 0x0000003971ee62c3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fee914ce576 in epoll_dispatch (base=0x5456780, tv=<value optimized out>) at epoll.c:404
#2 0x00007fee914b9e44 in event_base_loop (base=0x5456780, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414484 in worker_libevent (arg=0xff28e0) at daemon/thread.c:301
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fee8fa40700 (LWP 16490)):
#0 0x0000003971ee62c3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fee914ce576 in epoll_dispatch (base=0x5456280, tv=<value optimized out>) at epoll.c:404
#2 0x00007fee914b9e44 in event_base_loop (base=0x5456280, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414484 in worker_libevent (arg=0xff25f8) at daemon/thread.c:301
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fee8ca10700 (LWP 16494)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007fee8ceda6d2 in updateStatsThread (arg=0xff04c0) at memory_tracker.cc:31
#3 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fee91251720 (LWP 16479)):
#0 0x0000003971ee62c3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fee914ce576 in epoll_dispatch (base=0x5456000, tv=<value optimized out>) at epoll.c:404
#2 0x00007fee914b9e44 in event_base_loop (base=0x5456000, flags=<value optimized out>) at event.c:1558
#3 0x0000000000409746 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7912

Thread 7 (Thread 0x7fee8e63e700 (LWP 16492)):
#0 0x0000003971ee62c3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fee914ce576 in epoll_dispatch (base=0x5456a00, tv=<value optimized out>) at epoll.c:404
#2 0x00007fee914b9e44 in event_base_loop (base=0x5456a00, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414484 in worker_libevent (arg=0xff27e8) at daemon/thread.c:301
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fee9104e700 (LWP 16486)):
#0 0x0000003971ed89cd in read () from /lib64/libc.so.6
#1 0x0000003971e71128 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x0000003971e72c2e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x0000003971e6e11b in getc () from /lib64/libc.so.6
#4 0x00007fee9104f879 in check_stdin_thread (arg=0x403420) at extensions/daemon/stdin_check.c:19
#5 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fee8a9f4700 (LWP 16497)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fee8ce9dc17 in wait (this=0x54a1880) at syncobject.hh:36
#2 Dispatcher::run (this=0x54a1880) at dispatcher.cc:89
#3 0x00007fee8ce9e24b in launch_dispatcher_thread (arg=0x54a1880) at dispatcher.cc:28
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fee8f03f700 (LWP 16491)):
#0 0x0000003971ee62c3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fee914ce576 in epoll_dispatch (base=0x5456c80, tv=<value optimized out>) at epoll.c:404
#2 0x00007fee914b9e44 in event_base_loop (base=0x5456c80, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414484 in worker_libevent (arg=0xff26f0) at daemon/thread.c:301
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fee89ff3700 (LWP 16498)):
#0 0x000000397260b75b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fee8ce9b3d0 in wait (this=0x54bc280, d=...) at syncobject.hh:47
#2 IdleTask::run (this=0x54bc280, d=...) at dispatcher.cc:295
#3 0x00007fee8ce9d9c0 in Dispatcher::run (this=0x54a16c0) at dispatcher.cc:123
#4 0x00007fee8ce9e24b in launch_dispatcher_thread (arg=0x54a16c0) at dispatcher.cc:28
#5 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fee895f2700 (LWP 16499)):
#0 0x000000397260b75b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fee8cebac0f in wait (this=0x546ea00) at syncobject.hh:47
#2 wait (this=0x546ea00) at syncobject.hh:63
#3 wait (this=0x546ea00) at tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x546ea00) at ep_engine.cc:3453
#5 0x00007fee8cebacf3 in EvpNotifyPendingConns (arg=0x546ea00) at ep_engine.cc:1112
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fee8b3f5700 (LWP 16496)):
#0 0x0000003971e32885 in raise () from /lib64/libc.so.6
#1 0x0000003971e34065 in abort () from /lib64/libc.so.6
#2 0x0000003971e2b9fe in __assert_fail_base () from /lib64/libc.so.6
#3 0x0000003971e2bac0 in __assert_fail () from /lib64/libc.so.6
#4 0x00007fee8cf30061 in batchWarmupCallback (vbId=103, fetches=std::vector of length 42163, capacity 65536 =

, arg=0x7fee8b3f4840) at couch-kvstore/couch-kvstore.cc:171
#5 0x00007fee8cf08a86 in MutationLogHarvester::apply (this=0x7fee8b3f4780, arg=0x7fee8b3f4840, mlc=0x7fee8cf2fad0 <batchWarmupCallback(uint16_t, std::vector<std::pair<std::basic_string<char
, std::char_traits<char>, std::allocator<char> >, unsigned long>, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long> > >&, void*)
>) at mutation_log.cc:652
#6 0x00007fee8cf305b1 in CouchKVStore::warmup (this=0x5458c00, lf=<value optimized out>, vbmap=<value optimized out>, cb=..., estimate=...) at couch-kvstore/couch-kvstore.cc:1784
#7 0x00007fee8cf04288 in Warmup::loadingAccessLog (this=0x54ac000) at warmup.cc:458
#8 0x00007fee8cf04862 in Warmup::step (this=0x54ac000, d=<value optimized out>, t=<value optimized out>) at warmup.cc:554
#9 0x00007fee8cf06049 in WarmupStepper::callback(Dispatcher&, std::tr1::shared_ptr<Task>) () from /opt/couchbase/lib/memcached/ep.so
#10 0x00007fee8ce9eb0f in Task::run (this=<value optimized out>, d=<value optimized out>, t=<value optimized out>) at dispatcher.hh:139
#11 0x00007fee8ce9d9c0 in Dispatcher::run (this=0x54a0a80) at dispatcher.cc:123
#12 0x00007fee8ce9e24b in launch_dispatcher_thread (arg=0x54a0a80) at dispatcher.cc:28
#13 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#14 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
--------------------------------------------------------------------------------
Module information:
/opt/couchbase/lib/memcached/libmemcached_utilities.so.0:
lrwxrwxrwx. 1 bin bin 31 Jul 19 17:37 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0
cb808b1bcffb486f1e5884034707c69c /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
/opt/couchbase/lib/libevent-2.0.so.5:
lrwxrwxrwx. 1 bin bin 21 Jul 19 17:37 /opt/couchbase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0
f0dc63a3e615aaf24088f10273a0a3ed /opt/couchbase/lib/libevent-2.0.so.5
/lib64/libdl.so.2:
lrwxrwxrwx. 1 root root 13 May 22 14:11 /lib64/libdl.so.2 -> libdl-2.12.so
11652c5d0ba3bf86eb3580c27e5d52c3 /lib64/libdl.so.2
/lib64/libm.so.6:
lrwxrwxrwx. 1 root root 12 May 22 14:11 /lib64/libm.so.6 -> libm-2.12.so
211b8a166904bf77588bfe7b82546f82 /lib64/libm.so.6
/lib64/librt.so.1:
lrwxrwxrwx. 1 root root 13 May 22 14:11 /lib64/librt.so.1 -> librt-2.12.so
91b86595d72c65a9221e3a0f1628a31e /lib64/librt.so.1
/opt/couchbase/lib/libtcmalloc_minimal.so.4:
lrwxrwxrwx. 1 bin bin 28 Jul 19 17:37 /opt/couchbase/lib/libtcmalloc_minimal.so.4 -> libtcmalloc_minimal.so.4.1.0
fdf2985982a30333348f979c0794239a /opt/couchbase/lib/libtcmalloc_minimal.so.4
/lib64/libpthread.so.0:
lrwxrwxrwx. 1 root root 18 May 22 14:11 /lib64/libpthread.so.0 -> libpthread-2.12.so
d0f807f3fa3c1f95f7098c8af9442088 /lib64/libpthread.so.0
/lib64/libc.so.6:
lrwxrwxrwx. 1 root root 12 May 22 14:11 /lib64/libc.so.6 -> libc-2.12.so
23802125c461a5ec145f892f9da03c25 /lib64/libc.so.6
/lib64/ld-linux-x86-64.so.2:
lrwxrwxrwx. 1 root root 10 May 22 14:11 /lib64/ld-linux-x86-64.so.2 -> ld-2.12.so
150469f984599704562c67f0d7048c6b /lib64/ld-linux-x86-64.so.2
/usr/lib64/libstdc++.so.6:
lrwxrwxrwx. 1 root root 19 May 21 14:29 /usr/lib64/libstdc+.so.6 -> libstdc+.so.6.0.13
743a00f53ada7152287fcd37f3ae0152 /usr/lib64/libstdc++.so.6
/lib64/libgcc_s.so.1:
lrwxrwxrwx. 1 root root 28 May 21 14:28 /lib64/libgcc_s.so.1 -> libgcc_s-4.4.6-20110824.so.1
1761953c9a7d0f10c78055fbb4df6f99 /lib64/libgcc_s.so.1
/opt/couchbase/lib/memcached/stdin_term_handler.so:
-rwxr-xr-x. 1 bin bin 47522 Jul 19 16:25 /opt/couchbase/lib/memcached/stdin_term_handler.so
f9d3c3ec35235a11e597c6d29affecbd /opt/couchbase/lib/memcached/stdin_term_handler.so
/opt/couchbase/lib/memcached/bucket_engine.so:
lrwxrwxrwx. 1 bin bin 22 Jul 19 17:37 /opt/couchbase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0
2ad99bc8c3981cc61b1b5d2af1022e68 /opt/couchbase/lib/memcached/bucket_engine.so
/opt/couchbase/lib/memcached/ep.so:
lrwxrwxrwx. 1 bin bin 11 Jul 19 17:37 /opt/couchbase/lib/memcached/ep.so -> ep.so.0.0.0
dd599625186613b7d3df1f5d71bc34bd /opt/couchbase/lib/memcached/ep.so
/opt/couchbase/lib/libcouchstore.so.1:
lrwxrwxrwx. 1 bin bin 22 Jul 19 17:37 /opt/couchbase/lib/libcouchstore.so.1 -> libcouchstore.so.1.0.0
6af412b2d1ef23e8e692e630d488ae08 /opt/couchbase/lib/libcouchstore.so.1
/opt/couchbase/lib/libsnappy.so.1:
lrwxrwxrwx. 1 bin bin 18 Jul 19 17:37 /opt/couchbase/lib/libsnappy.so.1 -> libsnappy.so.1.1.2
2a0e36dfd064475a6b227b1238fd285d /opt/couchbase/lib/libsnappy.so.1
/lib64/libnss_files.so.2:
lrwxrwxrwx. 1 root root 20 May 22 14:11 /lib64/libnss_files.so.2 -> libnss_files-2.12.so
753e9a8fc403825892be0109ebb7f32a /lib64/libnss_files.so.2

Diags of 6 nodes is here
https://s3.amazonaws.com/packages.couchbase/diag-logs/large_cluster_2_0/6ndoes-memcached-crashed-warmup.tgz