memcached crashed in CouchKVStore::setVBucketState

http://qa.hq.northscale.net/job/cen-ubu-64-2.0-new-rebalance-mixed-cluster-P1/9/consoleFull

1.8.1 & 2.0.2-807 ( mixed cluster)
1_8_1:
10.5.2.11
10.6.2.81
10.6.2.82

2.0.2-807-rel:
10.3.121.119
10.3.121.121
10.3.121.122
10.3.121.123

on 10.3.121.121( root/couchbase):

[root@localhost tmp]# ls -la
total 1345100
drwxrwxrwt 10 root root 4096 May 20 01:09 .
drwxr-xr-x 24 root root 4096 May 18 09:40 ..
drwx------ 2 root root 4096 May 20 00:00 atop.d
drwxr-xr-x 2 root root 4096 May 18 03:24 backup
rw------ 1 couchbase couchbase 698519552 May 19 17:13 core.memcached.14586
rw------ 1 couchbase couchbase 947982336 May 19 13:57 core.memcached.3876
rw-rr- 1 root root 129834728 May 18 00:32 couchbase-server-enterprise_x86_64_2.0.2-807-rel.rpm
drwxrwxrwt 2 root root 4096 May 18 09:40 .font-unix
rw------ 1 root root 66 May 18 00:33 .gdm6DAMUW
srw-rw-rw- 1 root root 0 May 18 09:40 .gdm_socket
drwxrwxrwt 2 root root 4096 May 18 09:40 .ICE-unix
drwx------ 2 root root 4096 Apr 2 12:32 keyring-iUmHcT
drwx------ 2 couchbase couchbase 4096 Apr 11 2012 keyring-JYEPdw
drwx------ 2 root root 4096 May 18 09:40 vmware-root
rrr- 1 root root 11 May 18 09:40 .X0-lock
drwxrwxrwt 2 root root 4096 May 18 09:40 .X11-unix
[root@localhost tmp]# gdb /opt/couchbase/bin/memcached core.memcached.14586
GNU gdb (GDB) CentOS (7.0.1-45.el5.centos)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 15231]
[New Thread 20939]
[New Thread 20938]
[New Thread 20937]
[New Thread 15236]
[New Thread 15235]
[New Thread 15234]
[New Thread 15233]
[New Thread 15232]
[New Thread 15230]
[New Thread 14593]
[New Thread 14592]
[New Thread 14591]
[New Thread 14590]
[New Thread 14589]
[New Thread 14588]
[New Thread 14587]
[New Thread 14586]

warning: .dynamic section for "/usr/lib64/libstdc++.so.6" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib64/libgcc_s.so.1" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff41d5f000
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0 0x0000003018630265 in raise () from /lib64/libc.so.6
(gdb) t a a bt

Thread 18 (Thread 0x2b6944d5f220 (LWP 14586)):
#0 0x00000030186d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b69448ef576 in epoll_dispatch (base=0x13772000, tv=<value optimized out>) at epoll.c:404
#2 0x00002b69448dae44 in event_base_loop (base=0x13772000, flags=<value optimized out>) at event.c:1558
#3 0x00000000004097d6 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7926

Thread 17 (Thread 14587):
#0 0x00000030186c678b in read () from /lib64/libc.so.6
#1 0x000000301866cd57 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x000000301866d71e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x0000003018662804 in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x00000030186616a9 in fgets () from /lib64/libc.so.6
#5 0x00002b6944d60939 in check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
#6 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#7 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 16 (Thread 14588):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaaae4d6 in logger_thead_main (arg=0xef0a040) at extensions/loggers/file_logger.c:368
#2 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#3 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 15 (Thread 14589):
#0 0x00000030186d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b69448ef576 in epoll_dispatch (base=0x13772500, tv=<value optimized out>) at epoll.c:404
#2 0x00002b69448dae44 in event_base_loop (base=0x13772500, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0xef0d900) at daemon/thread.c:301
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 14 (Thread 14590):
#0 0x00000030186d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b69448ef576 in epoll_dispatch (base=0x13772280, tv=<value optimized out>) at epoll.c:404
#2 0x00002b69448dae44 in event_base_loop (base=0x13772280, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0xef0d9f8) at daemon/thread.c:301
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 13 (Thread 14591):
#0 0x00000030186d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b69448ef576 in epoll_dispatch (base=0x13772c80, tv=<value optimized out>) at epoll.c:404
#2 0x00002b69448dae44 in event_base_loop (base=0x13772c80, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0xef0daf0) at daemon/thread.c:301
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 12 (Thread 14592):
#0 0x00000030186d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b69448ef576 in epoll_dispatch (base=0x13772a00, tv=<value optimized out>) at epoll.c:404
#2 0x00002b69448dae44 in event_base_loop (base=0x13772a00, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0xef0dbe8) at daemon/thread.c:301
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

--Type <return> to continue, or q <return> to quit--
Thread 11 (Thread 14593):
#0 0x00000030186d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b69448ef576 in epoll_dispatch (base=0x13772780, tv=<value optimized out>) at epoll.c:404
#2 0x00002b69448dae44 in event_base_loop (base=0x13772780, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414604 in worker_libevent (arg=0xef0dce0) at daemon/thread.c:301
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 10 (Thread 15230):
#0 0x000000301869a541 in nanosleep () from /lib64/libc.so.6
#1 0x00000030186cded4 in usleep () from /lib64/libc.so.6
#2 0x00002aaaaaf32805 in updateStatsThread (arg=0xef0a4c0) at src/memory_tracker.cc:31
#3 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#4 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 9 (Thread 15232):

#0 0x000000301920d524 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003019208e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x0000003019208cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00002aaaaaf336fa in Mutex::acquire (this=0x137ed110) at src/mutex.cc:79
#4 0x00002aaaaaf79dc3 in lock (this=0x137ed000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:48
#5 LockHolder (this=0x137ed000, vbs=..., file_version=1, header_offset=4096, cb=...) at ./src/locks.hh:26
#6 CouchNotifier::notify_update (this=0x137ed000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:753
#7 0x00002aaaaaf70163 in CouchKVStore::setVBucketState (this=0x195f6300, vbucketId=1, vbstate=..., vb_change_type=1, newfile=96, notify=true) at src/couch-kvstore/couch-kvstore.cc:745
#8 0x00002aaaaaf71069 in CouchKVStore::snapshotVBuckets (this=0x195f6300, vbstates=Traceback (most recent call last):
File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 288, in children
nodetype = gdb.lookup_type('std::_Rb_tree_node< std::pair< %s, %s > >' % (keytype, valuetype))
RuntimeError: No type named std::_Rb_tree_node< std::pair< const unsigned short, vbucket_state > >.
std::map with 1 elements) at src/couch-kvstore/couch-kvstore.cc:596
#9 0x00002aaaaaf021f3 in EventuallyPersistentStore::snapshotVBuckets (this=0x2ca5b800, priority=..., shardId=<value optimized out>) at src/ep.cc:760
#10 0x00002aaaaaf51cff in VBSnapshotTask::run (this=<value optimized out>) at src/tasks.cc:78
#11 0x00002aaaaaf371c1 in ExecutorThread::run (this=0x1381da00) at src/scheduler.cc:153
#12 0x00002aaaaaf378dd in launch_executor_thread (arg=<value optimized out>) at src/scheduler.cc:34
#13 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#14 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 8 (Thread 15233):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf37051 in wait (this=0x1381d860) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x1381d860) at src/scheduler.cc:139
#3 0x00002aaaaaf378dd in launch_executor_thread (arg=<value optimized out>) at src/scheduler.cc:34
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 7 (Thread 15234):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf37051 in wait (this=0x13844ea0) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x13844ea0) at src/scheduler.cc:139
#3 0x00002aaaaaf378dd in launch_executor_thread (arg=<value optimized out>) at src/scheduler.cc:34
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 6 (Thread 15235):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf37051 in wait (this=0x13844d00) at src/syncobject.hh:57
--Type <return> to continue, or q <return> to quit--
#2 ExecutorThread::run (this=0x13844d00) at src/scheduler.cc:139
#3 0x00002aaaaaf378dd in launch_executor_thread (arg=<value optimized out>) at src/scheduler.cc:34
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 5 (Thread 15236):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf37051 in wait (this=0x13844b60) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x13844b60) at src/scheduler.cc:139
#3 0x00002aaaaaf378dd in launch_executor_thread (arg=<value optimized out>) at src/scheduler.cc:34
#4 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 4 (Thread 20937):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaf0ebdf in wait (this=0x13770400) at src/syncobject.hh:57
#2 wait (this=0x13770400) at src/syncobject.hh:73
#3 wait (this=0x13770400) at src/tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x13770400) at src/ep_engine.cc:3377
#5 0x00002aaaaaf0ecc3 in EvpNotifyPendingConns (arg=0x13770400) at src/ep_engine.cc:1153
#6 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#7 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 3 (Thread 20938):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef31c8 in wait (this=0x1428d440, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x1428d440, d=...) at src/dispatcher.cc:342
#3 0x00002aaaaaef5d2a in Dispatcher::run (this=0x137b2a80) at src/dispatcher.cc:184
#4 0x00002aaaaaef64ed in launch_dispatcher_thread (arg=<value optimized out>) at src/dispatcher.cc:28
#5 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#6 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 2 (Thread 20939):
#0 0x000000301920b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaaef31c8 in wait (this=0x1428ccf0, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x1428ccf0, d=...) at src/dispatcher.cc:342
#3 0x00002aaaaaef5d2a in Dispatcher::run (this=0x137b2fc0) at src/dispatcher.cc:184
#4 0x00002aaaaaef64ed in launch_dispatcher_thread (arg=<value optimized out>) at src/dispatcher.cc:28
#5 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#6 0x00000030186d44bd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x4651e940 (LWP 15231)):
#0 0x0000003018630265 in raise () from /lib64/libc.so.6
#1 0x0000003018631d10 in abort () from /lib64/libc.so.6
#2 0x00002aaaaaf7a779 in BinaryPacketHandler::implicitResponse (this=0x38fa) at src/couch-kvstore/couch-notifier.cc:41
#3 0x00002aaaaaf77009 in CouchNotifier::handleResponse (this=0x137ed000, res=0x137fc000) at src/couch-kvstore/couch-notifier.cc:216
#4 0x00002aaaaaf794eb in CouchNotifier::processInput (this=0x137ed000) at src/couch-kvstore/couch-notifier.cc:571
#5 0x00002aaaaaf79185 in waitOnce (this=0x137ed000) at src/couch-kvstore/couch-notifier.cc:677
#6 CouchNotifier::selectBucket (this=0x137ed000) at src/couch-kvstore/couch-notifier.cc:725
#7 0x00002aaaaaf7966f in CouchNotifier::processInput (this=0x137ed000) at src/couch-kvstore/couch-notifier.cc:608
#8 0x00002aaaaaf79f75 in waitOnce (this=0x137ed000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:677
#9 CouchNotifier::notify_update (this=0x137ed000, vbs=..., file_version=1, header_offset=4096, cb=...) at src/couch-kvstore/couch-notifier.cc:755
#10 0x00002aaaaaf70163 in CouchKVStore::setVBucketState (this=0x195f6000, vbucketId=0, vbstate=..., vb_change_type=1, newfile=96, notify=true) at src/couch-kvstore/couch-kvstore.cc:745
#11 0x00002aaaaaf71069 in CouchKVStore::snapshotVBuckets (this=0x195f6000, vbstates=Traceback (most recent call last):
File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 288, in children
--Type <return> to continue, or q <return> to quit--
nodetype = gdb.lookup_type('std::_Rb_tree_node< std::pair< %s, %s > >' % (keytype, valuetype))
RuntimeError: No type named std::_Rb_tree_node< std::pair< const unsigned short, vbucket_state > >.
std::map with 1 elements) at src/couch-kvstore/couch-kvstore.cc:596
#12 0x00002aaaaaf021f3 in EventuallyPersistentStore::snapshotVBuckets (this=0x2ca5b800, priority=..., shardId=<value optimized out>) at src/ep.cc:760
#13 0x00002aaaaaf51cff in VBSnapshotTask::run (this=<value optimized out>) at src/tasks.cc:78
#14 0x00002aaaaaf371c1 in ExecutorThread::run (this=0x1381dba0) at src/scheduler.cc:153
#15 0x00002aaaaaf378dd in launch_executor_thread (arg=<value optimized out>) at src/scheduler.cc:34
#16 0x000000301920673d in start_thread () from /lib64/libpthread.so.0
#17 0x00000030186d44bd in clone () from /lib64/libc.so.6
(gdb) quit