{UPR}:: Memcached crashing with Segmentation fault during rebalance

Jenkins Ref Link:
http://qa.hq.northscale.net/job/centos_x64--44_04--observe-P0/23/consoleFull
http://qa.hq.northscale.net/job/ubuntu_x64--10_01--bucket_flush-P0/31/consoleFull

Test failed:
./testrunner -i /tmp/bucket-flush.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False -t flush.bucketflush.BucketFlushTests.bucketflush,items=20000,sasl_buckets=2,nodes_in=1,GROUP=P1

*This test will not deterministically reproduce the crash. Observed same crash with some other tests too.

Crash Logs:
gdb /opt/couchbase/bin/memcached core.memcached.5038
GNU gdb (GDB) 7.1-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 5050]
[New Thread 5077]
[New Thread 5053]
[New Thread 5051]
[New Thread 5038]
[New Thread 5044]
[New Thread 5080]
[New Thread 5049]
[New Thread 5079]
[New Thread 5081]
[New Thread 5078]
[New Thread 5046]
[New Thread 5052]

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0
Reading symbols from /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1...done.
Loaded symbols for /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1
Reading symbols from /opt/couchbase/bin/../lib/libplatform.so.0.1.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libplatform.so.0.1.0
Reading symbols from /opt/couchbase/bin/../lib/libcJSON.so.1.0.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libcJSON.so.1.0.0
Reading symbols from /opt/couchbase/bin/../lib/libJSON_checker.so...done.
Loaded symbols for /opt/couchbase/bin/../lib/libJSON_checker.so
Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1
Reading symbols from /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4
Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5...done.
Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.0.so.5
Reading symbols from /lib/libssl.so.0.9.8...(no debugging symbols found)...done.
Loaded symbols for /lib/libssl.so.0.9.8
Reading symbols from /lib/libcrypto.so.0.9.8...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypto.so.0.9.8
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so
Reading symbols from /opt/couchbase/lib/libdirutils.so.0.1.0...done.
Loaded symbols for /opt/couchbase/lib/libdirutils.so.0.1.0
Reading symbols from /opt/couchbase/lib/libv8.so...done.
Loaded symbols for /opt/couchbase/lib/libv8.so
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000001a00000001 in ?? ()
((gdb) t a a bt

Thread 13 (Thread 5052):
#0 0x00007f4a77723d03 in epoll_wait () from /lib/libc.so.6
#1 0x00007f4a78d9e5a6 in epoll_dispatch (base=0x68aaa00, tv=<value optimized out>) at epoll.c:404
#2 0x00007f4a78d89a04 in event_base_loop (base=0x68aaa00, flags=<value optimized out>) at event.c:1558
#3 0x00007f4a7980809f in platform_thread_wrap (arg=0x1ff6170) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f4a785809ca in start_thread () from /lib/libpthread.so.0
#5 0x00007f4a7772370d in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 12 (Thread 5046):
#0 0x00007f4a78585bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00007f4a7980818b in cb_cond_timedwait (cond=0x7f4a76a22240, mutex=0x7f4a76a22200, ms=<value optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f4a768211e8 in logger_thead_main (arg=<value optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/extensions/loggers/file_logger.c:372
#3 0x00007f4a7980809f in platform_thread_wrap (arg=0x1ff6090) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f4a785809ca in start_thread () from /lib/libpthread.so.0
#5 0x00007f4a7772370d in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 11 (Thread 5078):
#0 0x00007f4a78585bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00007f4a7980818b in cb_cond_timedwait (cond=0x690cde0, mutex=0x690cda8, ms=<value optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f4a72b41fcf in SyncObject::wait (this=0x690cda0, tv=<value optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 0x00007f4a72b3db60 in ExecutorPool::trySleep (this=0x690cd80, t=..., now=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:177
#4 0x00007f4a72b3dda8 in ExecutorPool::nextTask (this=0x690cd80, t=..., tick=<value optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:135
#5 0x00007f4a72b4e3b7 in ExecutorThread::run (this=0x68c0780) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
#6 0x00007f4a72b4e7dd in launch_executor_thread (arg=0x690cde4) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#7 0x00007f4a7980809f in platform_thread_wrap (arg=0x1ff6c10) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x00007f4a785809ca in start_thread () from /lib/libpthread.so.0
#9 0x00007f4a7772370d in clone () from /lib/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 10 (Thread 5081):
#0 ObjectRegistry::memoryDeallocated (mem=<value optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/objectregistry.cc:211
#1 0x00007f4a78fc1251 in MallocHook::InvokeDeleteHookSlow (p=0x6d6e6c0) at src/malloc_hook.cc:529
#2 0x00007f4a78fc7215 in MallocHook::InvokeDeleteHook (p=0x6d6e6c0) at src/malloc_hook-inl.h:176
#3 tc_delete (p=0x6d6e6c0) at src/tcmalloc.cc:1618
#4 0x00007f4a72b85543 in __gnu_cxx::new_allocator<UprResponse**>::deallocate (this=<value optimized out>) at /usr/include/c++/4.4/ext/new_allocator.h:95
#5 std::_Deque_base<UprResponse*, std::allocator<UprResponse*> >::_M_deallocate_map (this=<value optimized out>) at /usr/include/c++/4.4/bits/stl_deque.h:459
#6 ~_Deque_base (this=<value optimized out>) at /usr/include/c++/4.4/bits/stl_deque.h:478
#7 ~deque (this=<value optimized out>) at /usr/include/c++/4.4/bits/stl_deque.h:790
#8 ~queue (this=<value optimized out>) at /usr/include/c++/4.4/bits/stl_queue.h:90
#9 PassiveStream::processBufferedMessages (this=<value optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/upr-stream.cc:935
#10 0x00007f4a72b7781d in UprConsumer::processBufferedItems (this=0x68dbc00) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/upr-consumer.cc:553
#11 0x00007f4a72b7791d in Processer::run (this=0x6e65a70) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/upr-consumer.cc:48
#12 0x00007f4a72b4e456 in ExecutorThread::run (this=0x68c0aa0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:94
#13 0x00007f4a72b4e7dd in launch_executor_thread (arg=0x6) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#14 0x00007f4a7980809f in platform_thread_wrap (arg=0x1ff6be0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#15 0x00007f4a785809ca in start_thread () from /lib/libpthread.so.0
#16 0x00007f4a7772370d in clone () from /lib/libc.so.6
#17 0x0000000000000000 in ?? ()

Thread 9 (Thread 5079):
#0 0x00007f4a78585bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00007f4a7980818b in cb_cond_timedwait (cond=0x690cde0, mutex=0x690cda8, ms=<value optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
--Type <return> to continue, or q <return> to quit--
#2 0x00007f4a72b41fcf in SyncObject::wait (this=0x690cda0, tv=<value optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 0x00007f4a72b3db60 in ExecutorPool::trySleep (this=0x690cd80, t=..., now=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:177
#4 0x00007f4a72b3dea4 in ExecutorPool::nextTask (this=0x690cd80, t=..., tick=24 '\030') at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:151
#5 0x00007f4a72b4e3b7 in ExecutorThread::run (this=0x68c0730) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
#6 0x00007f4a72b4e7dd in launch_executor_thread (arg=0x690cde4) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#7 0x00007f4a7980809f in platform_thread_wrap (arg=0x1ff6bf0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x00007f4a785809ca in start_thread () from /lib/libpthread.so.0
#9 0x00007f4a7772370d in clone () from /lib/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 8 (Thread 5049):
#0 0x00007f4a77723d03 in epoll_wait () from /lib/libc.so.6
#1 0x00007f4a78d9e5a6 in epoll_dispatch (base=0x68aa500, tv=<value optimized out>) at epoll.c:404
#2 0x00007f4a78d89a04 in event_base_loop (base=0x68aa500, flags=<value optimized out>) at event.c:1558
#3 0x00007f4a7980809f in platform_thread_wrap (arg=0x1ff61a0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f4a785809ca in start_thread () from /lib/libpthread.so.0
#5 0x00007f4a7772370d in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 7 (Thread 5080):
#0 0x00007f4a78585bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00007f4a7980818b in cb_cond_timedwait (cond=0x690cde0, mutex=0x690cda8, ms=<value optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f4a72b41fcf in SyncObject::wait (this=0x690cda0, tv=<value optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 0x00007f4a72b3db60 in ExecutorPool::trySleep (this=0x690cd80, t=..., now=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:177
#4 0x00007f4a72b3dea4 in ExecutorPool::nextTask (this=0x690cd80, t=..., tick=8 '\b') at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:151
#5 0x00007f4a72b4e3b7 in ExecutorThread::run (this=0x68c06e0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
#6 0x00007f4a72b4e7dd in launch_executor_thread (arg=0x690cde4) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#7 0x00007f4a7980809f in platform_thread_wrap (arg=0x1ff6c00) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x00007f4a785809ca in start_thread () from /lib/libpthread.so.0
#9 0x00007f4a7772370d in clone () from /lib/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 6 (Thread 5044):
#0 0x00007f4a777154bd in read () from /lib/libc.so.6
#1 0x00007f4a776b0348 in _IO_file_underflow () from /lib/libc.so.6
#2 0x00007f4a776b1eee in _IO_default_uflow () from /lib/libc.so.6
#3 0x00007f4a776a643e in _IO_getline_info () from /lib/libc.so.6
#4 0x00007f4a776a5329 in fgets () from /lib/libc.so.6
#5 0x00007f4a77224a91 in fgets (arg=<value optimized out>) at /usr/include/bits/stdio2.h:255
#6 check_stdin_thread (arg=<value optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/extensions/daemon/stdin_check.c:38
#7 0x00007f4a7980809f in platform_thread_wrap (arg=0x1ff6080) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x00007f4a785809ca in start_thread () from /lib/libpthread.so.0
#9 0x00007f4a7772370d in clone () from /lib/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 5 (Thread 5038):
#0 0x00007f4a77723d03 in epoll_wait () from /lib/libc.so.6
#1 0x00007f4a78d9e5a6 in epoll_dispatch (base=0x68aa000, tv=<value optimized out>) at epoll.c:404
#2 0x00007f4a78d89a04 in event_base_loop (base=0x68aa000, flags=<value optimized out>) at event.c:1558
#3 0x00000000004103d3 in main (argc=<value optimized out>, argv=<value optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/daemon/memcached.c:8786

Thread 4 (Thread 5051):
#0 0x00007f4a77723d03 in epoll_wait () from /lib/libc.so.6
#1 0x00007f4a78d9e5a6 in epoll_dispatch (base=0x68aac80, tv=<value optimized out>) at epoll.c:404
#2 0x00007f4a78d89a04 in event_base_loop (base=0x68aac80, flags=<value optimized out>) at event.c:1558
--Type <return> to continue, or q <return> to quit--
#3 0x00007f4a7980809f in platform_thread_wrap (arg=0x1ff6180) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f4a785809ca in start_thread () from /lib/libpthread.so.0
#5 0x00007f4a7772370d in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 3 (Thread 5053):
#0 0x00007f4a77723d03 in epoll_wait () from /lib/libc.so.6
#1 0x00007f4a78d9e5a6 in epoll_dispatch (base=0x68aa780, tv=<value optimized out>) at epoll.c:404
#2 0x00007f4a78d89a04 in event_base_loop (base=0x68aa780, flags=<value optimized out>) at event.c:1558
#3 0x00007f4a7980809f in platform_thread_wrap (arg=0x1ff6160) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f4a785809ca in start_thread () from /lib/libpthread.so.0
#5 0x00007f4a7772370d in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 2 (Thread 5077):
#0 0x00007f4a776e739d in nanosleep () from /lib/libc.so.6
#1 0x00007f4a7771c844 in usleep () from /lib/libc.so.6
#2 0x00007f4a72b4cc35 in updateStatsThread (arg=<value optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/memory_tracker.cc:36
#3 0x00007f4a7980809f in platform_thread_wrap (arg=0x1ff6210) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f4a785809ca in start_thread () from /lib/libpthread.so.0
#5 0x00007f4a7772370d in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 1 (Thread 5050):
#0 0x0000001a00000001 in ?? ()
#1 0x00007f4a72b77a2f in UprConsumer::getNextItem (this=0x68dbc00) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/upr-consumer.cc:577
#2 0x00007f4a72b77b9b in UprConsumer::step (this=0x68dbc00, producers=0x6225c0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/upr-consumer.cc:374
#3 0x00007f4a72b21149 in EvpUprStep (handle=<value optimized out>, cookie=0x67ab200, producers=0x6225c0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/ep_engine.cc:1352
#4 0x00007f4a75e19296 in upr_step (handle=<value optimized out>, cookie=0x67ab200, producers=0x6225c0)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:2215
#5 0x000000000041484d in ship_upr_log (c=0x67ab200) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/daemon/memcached.c:3455
#6 conn_ship_log (c=0x67ab200) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/daemon/memcached.c:6833
#7 0x000000000040a92d in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x67ab200) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/daemon/memcached.c:7276
#8 0x00007f4a78d89afc in event_process_active_single_queue (base=0x68aa280, flags=<value optimized out>) at event.c:1308
#9 event_process_active (base=0x68aa280, flags=<value optimized out>) at event.c:1375
#10 event_base_loop (base=0x68aa280, flags=<value optimized out>) at event.c:1572
#11 0x00007f4a7980809f in platform_thread_wrap (arg=0x1ff6190) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#12 0x00007f4a785809ca in start_thread () from /lib/libpthread.so.0
#13 0x00007f4a7772370d in clone () from /lib/libc.so.6
#14 0x0000000000000000 in ?? ()

Uploading Logs.