{UPR} :: Memcached crashed during Views + Rebalance test

Jenkins Ref Link:
http://qa.sc.couchbase.com/job/centos_x64--29_01--create_view_all-P1/85/consoleFull

Test to Reproduce:
./testrunner -i /tmp/new-viewtests-all.ini get-cbcollect-info=True,get-logs=True, -t view.createdeleteview.CreateDeleteViewTests.rebalance_in_and_out_with_ddoc_ops,ddoc_ops=update,test_with_view=True,num_ddocs=2,num_views_per_ddoc=3,items=200000,default_bucket=False,sasl_buckets=1,standard_buckets=1

Tests Failed:
rebalance_in_and_out_with_ddoc_ops,ddoc_ops=delete,test_with_view=False,num_ddocs=2,num_views_per_ddoc=3,items=200000
pending_removal_with_ddoc_ops,ddoc_ops=delete,test_with_view=False,num_ddocs=4,num_views_per_ddoc=3,items=200000,nodes_out=1,sasl_buckets=1,standard_buckets=1
ddoc_ops_during_failover,ddoc_ops=create,test_with_view=False,num_ddocs=3,num_views_per_ddoc=2,items=200000,nodes_out=2,replicas=2,default_bucket=False,sasl_buckets=1,standard_buckets=1
ddoc_ops_during_failover,ddoc_ops=update,test_with_view=False,num_ddocs=2,num_views_per_ddoc=2,items=200000,nodes_out=2,replicas=2
ddoc_ops_during_failover,ddoc_ops=delete,test_with_view=False,num_ddocs=2,num_views_per_ddoc=3,items=200000,nodes_out=1,sasl_buckets=1,standard_buckets=1

[user:info,2014-06-06T0:41:05.067,ns_1@172.23.107.24:<0.1440.0>:ns_orchestrator:handle_info:480]Rebalance exited with reason {unexpected_exit,
{'EXIT',<0.27026.4>,
{wait_seqno_persisted_failed,
"standard_bucket0",385,0,
[{'ns_1@172.23.107.25',
{'EXIT',
badmatch,{error,closed,
{gen_server,call,
[

{'janitor_agent-standard_bucket0', 'ns_1@172.23.107.25'}

,
{if_rebalance,<0.6288.4>,
{wait_seqno_persisted,385,0}},
infinity]}}}}]}}}

Crash Logs:
root@cherry-s10119:/tmp# gdb /opt/couchbase/bin/memcached core.memcached.7533
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New LWP 7676]
[New LWP 7543]
[New LWP 7544]
[New LWP 7546]
[New LWP 7542]
[New LWP 7548]
[New LWP 7533]
[New LWP 7672]
[New LWP 7545]
[New LWP 7547]
[New LWP 7674]
[New LWP 7673]
[New LWP 7675]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
Program terminated with signal 6, Aborted.
#0 0x00007ff2153df425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) t a a bt

Thread 13 (Thread 0x7ff20cc7a700 (LWP 7675)):
#0 0x00007ff215c8a0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007ff216f4b695 in cb_cond_timedwait (cond=0x64f3940, mutex=0x64f3908, ms=<optimized out>) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007ff210198060 in wait (tv=..., this=0x64f3900) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 ExecutorPool::trySleep (this=0x64f38e0, t=..., now=...) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/executorpool.cc:177
#4 0x00007ff2101983d1 in ExecutorPool::nextTask (this=0x64f38e0, t=..., tick=<optimized out>) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/executorpool.cc:135
#5 0x00007ff2101aa070 in ExecutorThread::run (this=0x1c28480) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
#6 0x00007ff2101aa699 in launch_executor_thread (arg=<optimized out>) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#7 0x00007ff216f4b34f in platform_thread_wrap (arg=0x1bd0be0) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x00007ff215c85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007ff21549ccbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7ff20dc7c700 (LWP 7673)):
#0 0x00007ff215c8a0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007ff216f4b695 in cb_cond_timedwait (cond=0x64f3940, mutex=0x64f3908, ms=<optimized out>) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007ff210198060 in wait (tv=..., this=0x64f3900) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 ExecutorPool::trySleep (this=0x64f38e0, t=..., now=...) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/executorpool.cc:177
#4 0x00007ff2101984fd in ExecutorPool::nextTask (this=0x64f38e0, t=..., tick=<optimized out>) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/executorpool.cc:151
#5 0x00007ff2101aa070 in ExecutorThread::run (this=0x1c28360) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
#6 0x00007ff2101aa699 in launch_executor_thread (arg=<optimized out>) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#7 0x00007ff216f4b34f in platform_thread_wrap (arg=0x1bd0bd0) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x00007ff215c85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007ff21549ccbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7ff20d47b700 (LWP 7674)):
#0 0x00007ff215c8a0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007ff216f4b695 in cb_cond_timedwait (cond=0x64f3940, mutex=0x64f3908, ms=<optimized out>) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007ff210198060 in wait (tv=..., this=0x64f3900) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 ExecutorPool::trySleep (this=0x64f38e0, t=..., now=...) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/executorpool.cc:177
#4 0x00007ff2101984fd in ExecutorPool::nextTask (this=0x64f38e0, t=..., tick=<optimized out>) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/executorpool.cc:151
#5 0x00007ff2101aa070 in ExecutorThread::run (this=0x1c283f0) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
#6 0x00007ff2101aa699 in launch_executor_thread (arg=<optimized out>) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#7 0x00007ff216f4b34f in platform_thread_wrap (arg=0x1bd0bf0) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x00007ff215c85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007ff21549ccbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7ff211c6f700 (LWP 7547)):
#0 0x00007ff21549d353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff2164e1337 in epoll_dispatch (base=0x6490a00, tv=<optimized out>) at epoll.c:404
#2 0x00007ff2164cc081 in event_base_loop (base=0x6490a00, flags=<optimized out>) at event.c:1558
#3 0x00007ff216f4b34f in platform_thread_wrap (arg=0x1bd0120) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007ff215c85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007ff21549ccbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7ff212c71700 (LWP 7545)):
#0 0x00007ff21549d353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff2164e1337 in epoll_dispatch (base=0x6490500, tv=<optimized out>) at epoll.c:404
#2 0x00007ff2164cc081 in event_base_loop (base=0x6490500, flags=<optimized out>) at event.c:1558
#3 0x00007ff216f4b34f in platform_thread_wrap (arg=0x1bd0100) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007ff215c85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
--Type <return> to continue, or q <return> to quit--
#5 0x00007ff21549ccbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7ff210c6d700 (LWP 7672)):
#0 0x00007ff21546883d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff215496774 in usleep () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ff2101a8ec5 in updateStatsThread (arg=0x1c20200) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/memory_tracker.cc:36
#3 0x00007ff216f4b34f in platform_thread_wrap (arg=0x1bd0210) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007ff215c85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007ff21549ccbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7ff217768740 (LWP 7533)):
#0 0x00007ff21549d353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff2164e1337 in epoll_dispatch (base=0x6490000, tv=<optimized out>) at epoll.c:404
#2 0x00007ff2164cc081 in event_base_loop (base=0x6490000, flags=<optimized out>) at event.c:1558
#3 0x0000000000405f61 in main (argc=<optimized out>, argv=<optimized out>) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/memcached/daemon/memcached.c:8785

Thread 6 (Thread 0x7ff21146e700 (LWP 7548)):
#0 0x00007ff21549d353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff2164e1337 in epoll_dispatch (base=0x6490c80, tv=<optimized out>) at epoll.c:404
#2 0x00007ff2164cc081 in event_base_loop (base=0x6490c80, flags=<optimized out>) at event.c:1558
#3 0x00007ff216f4b34f in platform_thread_wrap (arg=0x1bd0130) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007ff215c85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007ff21549ccbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7ff214887700 (LWP 7542)):
#0 0x00007ff21548f8bd in read () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff215423ff8 in _IO_file_underflow () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ff21542503e in _IO_default_uflow () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007ff21541918a in _IO_getline_info () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007ff21541806b in fgets () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007ff214888901 in fgets (__stream=<optimized out>, __n=80, __s=0x7ff214886e10 "\005") at /usr/include/x86_64-linux-gnu/bits/stdio2.h:255
#6 check_stdin_thread (arg=<optimized out>) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/memcached/extensions/daemon/stdin_check.c:38
#7 0x00007ff216f4b34f in platform_thread_wrap (arg=0x1bd0060) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x00007ff215c85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007ff21549ccbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7ff212470700 (LWP 7546)):
#0 0x00007ff21549d353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff2164e1337 in epoll_dispatch (base=0x6490780, tv=<optimized out>) at epoll.c:404
#2 0x00007ff2164cc081 in event_base_loop (base=0x6490780, flags=<optimized out>) at event.c:1558
#3 0x00007ff216f4b34f in platform_thread_wrap (arg=0x1bd0110) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007ff215c85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007ff21549ccbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7ff213472700 (LWP 7544)):
#0 0x00007ff21549d353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff2164e1337 in epoll_dispatch (base=0x6490280, tv=<optimized out>) at epoll.c:404
#2 0x00007ff2164cc081 in event_base_loop (base=0x6490280, flags=<optimized out>) at event.c:1558
#3 0x00007ff216f4b34f in platform_thread_wrap (arg=0x1bd00f0) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007ff215c85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
--Type <return> to continue, or q <return> to quit--
#5 0x00007ff21549ccbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7ff213e82700 (LWP 7543)):
#0 0x00007ff215c8a0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007ff216f4b695 in cb_cond_timedwait (cond=0x7ff214086200, mutex=0x7ff2140861c0, ms=<optimized out>)
at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007ff213e84bb0 in logger_thead_main (arg=0x1c20900) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/memcached/extensions/loggers/file_logger.c:372
#3 0x00007ff216f4b34f in platform_thread_wrap (arg=0x1bd0070) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007ff215c85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007ff21549ccbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7ff20c479700 (LWP 7676)):
#0 0x00007ff2153df425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ff2153e2b8b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ff2153d80ee in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007ff2153d8192 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007ff2101e129a in ~PassiveStream (this=<optimized out>, __in_chrg=<optimized out>) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/upr-stream.cc:829
#5 PassiveStream::~PassiveStream (this=0x67c3c00, __in_chrg=<optimized out>) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/upr-stream.cc:825
#6 0x00007ff2101e12d9 in PassiveStream::~PassiveStream (this=0x67c3c00, __in_chrg=<optimized out>) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/upr-stream.cc:830
#7 0x00007ff2101d3c31 in ~SingleThreadedRCPtr (this=0x6f5f940, __in_chrg=<optimized out>) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/atomic.h:387
#8 ~SingleThreadedRCPtr (this=0x6f5f940, __in_chrg=<optimized out>) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/upr-consumer.cc:89
#9 UprConsumer::~UprConsumer (this=0x64c5a40, __in_chrg=<optimized out>) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/upr-consumer.cc:90
#10 0x00007ff2101d3dd9 in UprConsumer::~UprConsumer (this=0x64c5a40, __in_chrg=<optimized out>) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/upr-consumer.cc:91
#11 0x00007ff2101c008a in UprConnMap::manageConnections (this=0x64fc000) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/tapconnmap.cc:1041
#12 0x00007ff2101c482d in run (this=0x1c29170) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/tapconnmap.cc:132
#13 ConnManager::run (this=0x1c29170) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/tapconnmap.cc:128
#14 0x00007ff2101aa111 in ExecutorThread::run (this=0x1c28510) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/executorthread.cc:94
#15 0x00007ff2101aa699 in launch_executor_thread (arg=<optimized out>) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#16 0x00007ff216f4b34f in platform_thread_wrap (arg=0x1bd0c00) at /home/buildbot/buildbot_slave/ubuntu-1204-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#17 0x00007ff215c85e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#18 0x00007ff21549ccbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#19 0x0000000000000000 in ?? ()

Uploading Logs