memcached crashed (Segmentation fault) when upgrade from 2.0.0, 2.5.0 to 3.0.0

Install couchbase server 2.0.0 GA
Create bucket
Shutdown couchbase server
Upgrade couchbase server to 3.0.0-433
Upgrade failed due to memcached crashed

Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 139.

Basic crash dump analysis of /tmp/core.memcached.4312.

Please send the file to support@couchbase.com

--------------------------------------------------------------------------------
File information:
-rwxr-xr-x 1 couchbase couchbase 4250336 Mar 13 13:29 /opt/couchbase/bin/memcached
b81b2d8643d61a06c695a2ee6e4619ec /opt/couchbase/bin/memcached
rw------ 1 couchbase couchbase 241614848 Mar 14 16:27 /tmp/core.memcached.4312
f01130efe7a1267edd1ad7404ab117db /tmp/core.memcached.4312
--------------------------------------------------------------------------------
Core file callstacks:
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New LWP 4340]
[New LWP 4312]
[New LWP 4325]
[New LWP 4326]
[New LWP 4334]
[New LWP 4335]
[New LWP 4336]
[New LWP 4337]
[New LWP 4338]
[New LWP 4339]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fb769832000
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fb767618321 in ?? () from /lib/x86_64-linux-gnu/libc.so.6

Thread 10 (Thread 0x7fb7608d7700 (LWP 4339)):
#0 0x00007fb767f890fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb76900319b in cb_cond_timedwait (cond=0xbf9cde0, mutex=0xbf9cda8, ms=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
#2 0x00007fb7644adc3f in SyncObject::wait (this=0xbf9cda0, tv=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/syncobject.h:74
#3 0x00007fb7644a9c40 in ExecutorPool::trySleep (this=0xbf9cd80, t=..., now=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:177
#4 0x00007fb7644a9f84 in ExecutorPool::nextTask (this=0xbf9cd80, t=..., tick=8 '\b') at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:151
#5 0x00007fb7644b9d27 in ExecutorThread::run (this=0xbf58550) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:77
#6 0x00007fb7644ba12d in launch_executor_thread (arg=0xbf9cde4) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
#7 0x00007fb7690030af in platform_thread_wrap (arg=0x2bc8b60) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#8 0x00007fb767f84e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007fb7675a93fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7fb7610d8700 (LWP 4338)):
#0 0x00007fb767f890fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb76900319b in cb_cond_timedwait (cond=0xbf9cde0, mutex=0xbf9cda8, ms=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
#2 0x00007fb7644adc3f in SyncObject::wait (this=0xbf9cda0, tv=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/syncobject.h:74
#3 0x00007fb7644a9c40 in ExecutorPool::trySleep (this=0xbf9cd80, t=..., now=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:177
#4 0x00007fb7644a9f84 in ExecutorPool::nextTask (this=0xbf9cd80, t=..., tick=24 '\030') at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:151
#5 0x00007fb7644b9d27 in ExecutorThread::run (this=0xbf585a0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:77
#6 0x00007fb7644ba12d in launch_executor_thread (arg=0xbf9cde4) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
#7 0x00007fb7690030af in platform_thread_wrap (arg=0x2bc8b50) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#8 0x00007fb767f84e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007fb7675a93fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7fb7618d9700 (LWP 4337)):
#0 0x00007fb767f890fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb76900319b in cb_cond_timedwait (cond=0xbf9cde0, mutex=0xbf9cda8, ms=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
#2 0x00007fb7644adc3f in SyncObject::wait (this=0xbf9cda0, tv=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/syncobject.h:74
#3 0x00007fb7644a9c40 in ExecutorPool::trySleep (this=0xbf9cd80, t=..., now=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:177
#4 0x00007fb7644a9f84 in ExecutorPool::nextTask (this=0xbf9cd80, t=..., tick=0 '\000') at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:151
#5 0x00007fb7644b9d27 in ExecutorThread::run (this=0xbf585f0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:77
#6 0x00007fb7644ba12d in launch_executor_thread (arg=0xbf9cde4) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
#7 0x00007fb7690030af in platform_thread_wrap (arg=0x2bc8b70) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#8 0x00007fb767f84e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007fb7675a93fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7fb7620da700 (LWP 4336)):
#0 0x00007fb76757508d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fb7675a2eb4 in usleep () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fb7644b86d5 in updateStatsThread (arg=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/memory_tracker.cc:36
#3 0x00007fb7690030af in platform_thread_wrap (arg=0x2bc80f0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007fb767f84e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fb7675a93fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7fb764f71700 (LWP 4335)):
#0 0x00007fb7675a9a93 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fb76879c5a6 in epoll_dispatch (base=0xbf48280, tv=<optimized out>) at epoll.c:404
#2 0x00007fb768787a04 in event_base_loop (base=0xbf48280, flags=<optimized out>) at event.c:1558
#3 0x00007fb7690030af in platform_thread_wrap (arg=0x2bc80d0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007fb767f84e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fb7675a93fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7fb765772700 (LWP 4334)):
#0 0x00007fb767f88d84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb76447d26c in wait (this=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/syncobject.h:39
#2 wait (this=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/ep.cc:293
#3 EventuallyPersistentStore::initialize (this=0xbf64480) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/ep.cc:323
#4 0x00007fb764494df7 in EventuallyPersistentEngine::initialize (this=0xbf8a000, config=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/ep_engine.cc:1874
#5 0x00007fb7644951b6 in EvpInitialize (handle=0xbf8a000,
config_str=0xbf64023 "ht_size=3079;ht_locks=5;tap_noop_interval=20;max_size=621805568;tap_keepalive=300;dbname=/opt/couchbase/var/lib/couchbase/data/default;allow_data_loss_during_shutdown=true;backend=couchdb;couch_bucket"...)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/ep_engine.cc:133
#6 0x00007fb76577864a in create_bucket_UNLOCKED (e=<optimized out>, bucket_name=0x2bc40e0 "default", path=0xbf64000 "/opt/couchbase/lib/memcached/ep.so", config=<optimized out>, e_out=<optimized out>, msg=0x7fb7657717f0 "",
msglen=1024) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/engines/bucket_engine/bucket_engine.c:834
#7 0x00007fb7657788be in handle_create_bucket (handle=0x7fb76597f220, cookie=0xbeed200, request=<optimized out>, response=0x409e10 <binary_response_handler>)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/engines/bucket_engine/bucket_engine.c:2767
#8 0x00007fb765778d79 in bucket_unknown_command (handle=0x7fb76597f220, cookie=0xbeed200, request=0xbf16800, response=0x409e10 <binary_response_handler>)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/engines/bucket_engine/bucket_engine.c:3107
#9 0x0000000000416d91 in process_bin_unknown_packet (c=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:2786
#10 process_bin_packet (c=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:4793
#11 complete_nread (c=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:5350
#12 conn_nread (c=0xbeed200) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:6521
#13 0x0000000000409a2d in event_handler (fd=<optimized out>, which=<optimized out>, arg=0xbeed200) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:6794
#14 0x00007fb768787afc in event_process_active_single_queue (activeq=<optimized out>, base=<optimized out>) at event.c:1308
#15 event_process_active (base=<optimized out>) at event.c:1375
#16 event_base_loop (base=0xbf48500, flags=<optimized out>) at event.c:1572
#17 0x00007fb7690030af in platform_thread_wrap (arg=0x2bc8060) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#18 0x00007fb767f84e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#19 0x00007fb7675a93fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#20 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7fb766180700 (LWP 4326)):
#0 0x00007fb767f890fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb76900319b in cb_cond_timedwait (cond=0x7fb766384200, mutex=0x7fb7663841c0, ms=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
#2 0x00007fb76618304d in logger_thead_main (arg=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/extensions/loggers/file_logger.c:363
#3 0x00007fb7690030af in platform_thread_wrap (arg=0x2bc8090) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007fb767f84e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fb7675a93fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7fb766b85700 (LWP 4325)):
#0 0x00007fb76759bffd in read () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fb76752fff8 in _IO_file_underflow () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fb76753103e in _IO_default_uflow () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007fb76752518a in _IO_getline_info () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007fb76752406b in fgets () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007fb766b86aa1 in fgets (__stream=<optimized out>, __n=<optimized out>, __s=<optimized out>) at /usr/include/bits/stdio2.h:255
#6 check_stdin_thread (arg=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/extensions/daemon/stdin_check.c:38
#7 0x00007fb7690030af in platform_thread_wrap (arg=0x2bc8080) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#8 0x00007fb767f84e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007fb7675a93fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7fb76981e740 (LWP 4312)):
#0 0x00007fb7675a9a93 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fb76879c5a6 in epoll_dispatch (base=0xbf48000, tv=<optimized out>) at epoll.c:404
#2 0x00007fb768787a04 in event_base_loop (base=0xbf48000, flags=<optimized out>) at event.c:1558
#3 0x000000000040eb19 in main (argc=<optimized out>, argv=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:8250

Thread 1 (Thread 0x7fb7600d6700 (LWP 4340)):
#0 0x00007fb767618321 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fb76450d288 in length (__s=<optimized out>) at /usr/include/c++/4.4/bits/char_traits.h:263
#2 assign (__s=<optimized out>, this=<optimized out>) at /usr/include/c++/4.4/bits/basic_string.h:970
#3 CouchKVStore::readVBState (db=0xc05d100, vbId=<optimized out>, vbState=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-kvstore.cc:1865
#4 0x00007fb7645115c8 in CouchKVStore::listPersistedVbuckets (this=0xbfc2340) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-kvstore.cc:569
#5 0x00007fb764476016 in EventuallyPersistentStore::loadVBucketState (this=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/ep.cc:2633
#6 0x00007fb7644fd2c9 in Warmup::initialize (this=0xbf700f0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/warmup.cc:420
#7 0x00007fb7644fee90 in WarmupInitialize::run() () from /opt/couchbase/lib/memcached/ep.so
#8 0x00007fb7644b9dc6 in ExecutorThread::run (this=0xbf58500) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:93
#9 0x00007fb7644ba12d in launch_executor_thread (arg=0x0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
#10 0x00007fb7690030af in platform_thread_wrap (arg=0x2bc8b40) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#11 0x00007fb767f84e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#12 0x00007fb7675a93fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#13 0x0000000000000000 in ?? ()
--------------------------------------------------------------------------------
Module information: