{UPR} :: Memcached crashes with Segmentation fault during Views + Rebalance test

Jenkins Link:
http://qa.sc.couchbase.com/job/centos_x64--29_01--new_view_all-P1/63/console

Test to Reproduce:
./testrunner -i <yourfile.ini> get-cbcollect-info=True,get-logs=True, -t view.createdeleteview.CreateDeleteViewTests.rebalance_in_and_out_with_ddoc_ops,ddoc_ops=create,test_with_view=True,num_ddocs=3,num_views_per_ddoc=2,items=200000,sasl_buckets=1

Tests Failed:

Crash Logs:
root@cherry-s10120:/tmp# gdb /opt/couchbase/bin/memcached core.memcached.1792
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New LWP 2070]
[New LWP 1815]
[New LWP 1804]
[New LWP 1792]
[New LWP 1817]
[New LWP 1819]
[New LWP 1816]
[New LWP 1818]
[New LWP 2067]
[New LWP 2068]
[New LWP 1805]
[New LWP 2066]
[New LWP 2069]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007ffb45cd1dbb in ActiveStream::getItemsRemaining (this=0x5e8d760) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/upr-stream.cc:636
636 /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/upr-stream.cc: No such file or directory.
(gdb)
(gdb) t a a bt

Thread 13 (Thread 0x7ffb426e3700 (LWP 2069)):
#0 0x00007ffb4b7760fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007ffb4c9f318b in cb_cond_timedwait (cond=0x5eaede0, mutex=0x5eaeda8, ms=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007ffb45c94f5f in SyncObject::wait (this=0x5eaeda0, tv=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 0x00007ffb45c90b10 in ExecutorPool::trySleep (this=0x5eaed80, t=..., now=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:177
#4 0x00007ffb45c90e54 in ExecutorPool::nextTask (this=0x5eaed80, t=..., tick=8 '\b') at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:151
#5 0x00007ffb45ca12fa in ExecutorThread::run (this=0x5e626e0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
#6 0x00007ffb45ca16fd in launch_executor_thread (arg=0x5eaede4) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#7 0x00007ffb4c9f309f in platform_thread_wrap (arg=0x1598bf0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x00007ffb4b771e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007ffb4a880cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7ffb4675b700 (LWP 2066)):
#0 0x00007ffb4a84c83d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffb4a87a774 in usleep () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ffb45c9fb85 in updateStatsThread (arg=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/memory_tracker.cc:36
#3 0x00007ffb4c9f309f in platform_thread_wrap (arg=0x1598200) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007ffb4b771e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007ffb4a880cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7ffb4996e700 (LWP 1805)):
#0 0x00007ffb4b7760fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007ffb4c9f318b in cb_cond_timedwait (cond=0x7ffb49b72240, mutex=0x7ffb49b72200, ms=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007ffb499711e8 in logger_thead_main (arg=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/extensions/loggers/file_logger.c:372
#3 0x00007ffb4c9f309f in platform_thread_wrap (arg=0x1598090) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007ffb4b771e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007ffb4a880cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7ffb42ee4700 (LWP 2068)):
#0 0x00007ffb4b7760fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007ffb4c9f318b in cb_cond_timedwait (cond=0x5eaede0, mutex=0x5eaeda8, ms=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007ffb45c94f5f in SyncObject::wait (this=0x5eaeda0, tv=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 0x00007ffb45c90b10 in ExecutorPool::trySleep (this=0x5eaed80, t=..., now=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:177
#4 0x00007ffb45c90e54 in ExecutorPool::nextTask (this=0x5eaed80, t=..., tick=24 '\030') at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:151
#5 0x00007ffb45ca12fa in ExecutorThread::run (this=0x5e62730) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
#6 0x00007ffb45ca16fd in launch_executor_thread (arg=0x5eaede4) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#7 0x00007ffb4c9f309f in platform_thread_wrap (arg=0x1598be0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x00007ffb4b771e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007ffb4a880cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7ffb436e5700 (LWP 2067)):
#0 0x00007ffb4b7760fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007ffb4c9f318b in cb_cond_timedwait (cond=0x5eaede0, mutex=0x5eaeda8, ms=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007ffb45c94f5f in SyncObject::wait (this=0x5eaeda0, tv=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 0x00007ffb45c90b10 in ExecutorPool::trySleep (this=0x5eaed80, t=..., now=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:177
#4 0x00007ffb45c90e54 in ExecutorPool::nextTask (this=0x5eaed80, t=..., tick=0 '\000') at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorpool.cc:151
#5 0x00007ffb45ca12fa in ExecutorThread::run (this=0x5e62780) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
#6 0x00007ffb45ca16fd in launch_executor_thread (arg=0x5eaede4) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#7 0x00007ffb4c9f309f in platform_thread_wrap (arg=0x1598c00) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x00007ffb4b771e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
--Type <return> to continue, or q <return> to quit--
#9 0x00007ffb4a880cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7ffb4775d700 (LWP 1818)):
#0 0x00007ffb4a881353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffb4bf895a6 in epoll_dispatch (base=0x5e4ca00, tv=<optimized out>) at epoll.c:404
#2 0x00007ffb4bf74a04 in event_base_loop (base=0x5e4ca00, flags=<optimized out>) at event.c:1558
#3 0x00007ffb4c9f309f in platform_thread_wrap (arg=0x1598170) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007ffb4b771e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007ffb4a880cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7ffb4875f700 (LWP 1816)):
#0 0x00007ffb4a881353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffb4bf895a6 in epoll_dispatch (base=0x5e4c280, tv=<optimized out>) at epoll.c:404
#2 0x00007ffb4bf74a04 in event_base_loop (base=0x5e4c280, flags=<optimized out>) at event.c:1558
#3 0x00007ffb4c9f309f in platform_thread_wrap (arg=0x1598190) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007ffb4b771e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007ffb4a880cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7ffb46f5c700 (LWP 1819)):
#0 0x00007ffb4a881353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffb4bf895a6 in epoll_dispatch (base=0x5e4c780, tv=<optimized out>) at epoll.c:404
#2 0x00007ffb4bf74a04 in event_base_loop (base=0x5e4c780, flags=<optimized out>) at event.c:1558
#3 0x00007ffb4c9f309f in platform_thread_wrap (arg=0x1598160) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007ffb4b771e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007ffb4a880cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7ffb47f5e700 (LWP 1817)):
#0 0x00007ffb4a881353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffb4bf895a6 in epoll_dispatch (base=0x5e4cc80, tv=<optimized out>) at epoll.c:404
#2 0x00007ffb4bf74a04 in event_base_loop (base=0x5e4cc80, flags=<optimized out>) at event.c:1558
#3 0x00007ffb4c9f309f in platform_thread_wrap (arg=0x1598180) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007ffb4b771e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007ffb4a880cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7ffb4d213740 (LWP 1792)):
#0 0x00007ffb4a881353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffb4bf895a6 in epoll_dispatch (base=0x5e4c000, tv=<optimized out>) at epoll.c:404
#2 0x00007ffb4bf74a04 in event_base_loop (base=0x5e4c000, flags=<optimized out>) at event.c:1558
#3 0x00000000004101d3 in main (argc=<optimized out>, argv=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/daemon/memcached.c:8724

Thread 3 (Thread 0x7ffb4a373700 (LWP 1804)):
#0 0x00007ffb4a8738bd in read () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffb4a807ff8 in _IO_file_underflow () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ffb4a80903e in _IO_default_uflow () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007ffb4a7fd18a in _IO_getline_info () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007ffb4a7fc06b in fgets () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007ffb4a374a91 in fgets (__stream=<optimized out>, __n=<optimized out>, __s=<optimized out>) at /usr/include/bits/stdio2.h:255
#6 check_stdin_thread (arg=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/extensions/daemon/stdin_check.c:38
#7 0x00007ffb4c9f309f in platform_thread_wrap (arg=0x1598080) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x00007ffb4b771e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
--Type <return> to continue, or q <return> to quit--
#9 0x00007ffb4a880cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7ffb48f60700 (LWP 1815)):
#0 0x00007ffb4a881353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffb4bf895a6 in epoll_dispatch (base=0x5e4c500, tv=<optimized out>) at epoll.c:404
#2 0x00007ffb4bf74a04 in event_base_loop (base=0x5e4c500, flags=<optimized out>) at event.c:1558
#3 0x00007ffb4c9f309f in platform_thread_wrap (arg=0x15981a0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007ffb4b771e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007ffb4a880cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7ffb41ee2700 (LWP 2070)):
#0 0x00007ffb45cd1dbb in ActiveStream::getItemsRemaining (this=0x5e8d760) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/upr-stream.cc:636
#1 0x00007ffb45cd0812 in getItemsRemaining_UNLOCKED (this=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/upr-producer.cc:590
#2 UprProducer::addStats (this=0x8df3c80, add_stat=0x7ffb45c68c50 <add_stat(char const*, uint16_t, char const*, uint32_t, void const*)>, c=0x7ffb41ee1c30)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/upr-producer.cc:372
#3 0x00007ffb45c77109 in operator() (tc=..., this=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/ep_engine.cc:3545
#4 for_each<std::List_iterator<SingleThreadedRCPtr<ConnHandler> >, ConnStatBuilder> (_last=..., __first=..., __f=...) at /usr/include/c++/4.4/bits/stl_algo.h:4200
#5 each_UNLOCKED<ConnStatBuilder> (f=..., this=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/tapconnmap.h:146
#6 each<ConnStatBuilder> (f=..., this=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/tapconnmap.h:138
#7 EventuallyPersistentEngine::doUprStats (this=<optimized out>, cookie=0x7ffb41ee1c30, add_stat=0x7ffb45c68c50 <add_stat(char const*, uint16_t, char const*, uint32_t, void const*)>)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/ep_engine.cc:3794
#8 0x00007ffb45c877ae in EventuallyPersistentEngine::getStats (this=0x5e9c000, cookie=0x7ffb41ee1c30, stat_key=0x7ffb45d1c85b "upr", nkey=3,
add_stat=0x7ffb45c68c50 <add_stat(char const*, uint16_t, char const*, uint32_t, void const*)>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/ep_engine.cc:4132
#9 0x00007ffb45c61d2e in EventuallyPersistentStore::snapshotStats (this=0x5e6e480) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/ep.cc:1175
#10 0x00007ffb45cc0db4 in StatSnap::run (this=0x15e4b40) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/tasks.cc:58
#11 0x00007ffb45ca1399 in ExecutorThread::run (this=0x5e62aa0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:93
#12 0x00007ffb45ca16fd in launch_executor_thread (arg=0x7ffb41ee13b0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#13 0x00007ffb4c9f309f in platform_thread_wrap (arg=0x1598bd0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#14 0x00007ffb4b771e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#15 0x00007ffb4a880cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#16 0x0000000000000000 in ?? ()

Logs:
[user:info,2014-05-16T0:33:25.555,ns_1@172.23.107.24:<0.8941.0>:ns_orchestrator:handle_info:480]Rebalance exited with reason {unexpected_exit,
{'EXIT',<0.19354.2>,
{wait_seqno_persisted_failed,"bucket0",329,
179,
[{'ns_1@172.23.107.27',
{'EXIT',
{{badmatch,wrong_rebalancer_pid},
[

{janitor_agent,wait_seqno_persisted,5},
{ns_single_vbucket_mover, '-wait_seqno_persisted_many/5-fun-0-',5},
{misc,'-parallel_map/3-fun-0-',4}]}}}]}}}

[ns_server:debug,2014-05-16T0:33:25.555,ns_1@172.23.107.24:<0.18256.2>:ns_pubsub:do_subscribe_link:136]Parent process of subscription {master_activity_events,<0.18246.2>} exited with reason {unexpected_exit,
{'EXIT',
<0.19354.2>,
{wait_seqno_persisted_failed,
"bucket0",
329,
179,
[{'ns_1@172.23.107.27',
{'EXIT',
{{badmatch,
wrong_rebalancer_pid},
[{janitor_agent, wait_seqno_persisted, 5},
{ns_single_vbucket_mover, '-wait_seqno_persisted_many/5-fun-0-', 5},
{misc, '-parallel_map/3-fun-0-', 4}]}}}]}}}
[ns_server:debug,2014-05-16T0:33:25.555,ns_1@172.23.107.24:upr_consumer_conn-bucket0-ns_1@172.23.107.25<0.26132.0>:upr_proxy:terminate:78]Terminating. Disconnecting from socket #Port<0.15451>
[ns_server:debug,2014-05-16T0:33:25.555,ns_1@172.23.107.24:<0.26134.0>:upr_proxy:terminate:78]Terminating. Disconnecting from socket #Port<0.15453>
[ns_server:debug,2014-05-16T0:33:25.556,ns_1@172.23.107.24:capi_set_view_manager-bucket0<0.3378.0>:capi_set_view_manager:handle_info:306]doing replicate_newnodes_docs
[ns_server:info,2014-05-16T0:33:25.556,ns_1@172.23.107.24:<0.19494.2>:diag_handler:log_all_tap_and_checkpoint_stats:125]logging tap & checkpoint stats
[ns_server:debug,2014-05-16T0:33:25.557,ns_1@172.23.107.24:<0.19497.2>:upr_proxy:nuke_connection:176]Nuke UPR connection "ns_server:ns_1@172.23.107.25->ns_1@172.23.107.24:bucket0" type consumer on node 'ns_1@172.23.107.24'
[ns_server:debug,2014-05-16T0:33:25.557,ns_1@172.23.107.24:ns_config_log<0.278.0>:ns_config_log:log_common:134]config change:
counters ->
[{'_vclock',[{<<"c3c3fa0e717f561e3f6b499fd50237fa">>,{4,63567444805}}]},
{rebalance_fail,1},
{rebalance_start,2},
{rebalance_success,1}]
[ns_server:debug,2014-05-16T0:33:25.557,ns_1@172.23.107.24:<0.19498.2>:upr_proxy:nuke_connection:176]Nuke UPR connection "ns_server:ns_1@172.23.107.25->ns_1@172.23.107.24:bucket0" type producer on node 'ns_1@172.23.107.25'
[error_logger:error,2014-05-16T0:33:25.557,ns_1@172.23.107.24:error_logger<0.6.0>:ale_error_logger_handler:do_log:207]
=========================CRASH REPORT=========================
crasher:
initial call: ns_single_vbucket_mover:mover/6
pid: <0.19445.2>
registered_name: []
exception exit: {unexpected_exit,
{'EXIT',<0.18433.2>,
{unexpected_exit,
{'EXIT',<0.19354.2>,
{wait_seqno_persisted_failed,"bucket0",329,179,
[{'ns_1@172.23.107.27',
{'EXIT',
{{badmatch,wrong_rebalancer_pid},
[{janitor_agent,wait_seqno_persisted,5}

,

{ns_single_vbucket_mover, '-wait_seqno_persisted_many/5-fun-0-',5}

,

{misc,'-parallel_map/3-fun-0-',4}]}}}]}}}}}
in function ns_single_vbucket_mover:spawn_and_wait/1
in call from ns_single_vbucket_mover:mover_inner_upr/6
in call from ns_single_vbucket_mover:'mover/6-fun-2'/7
in call from misc:try_with_maybe_ignorant_after/2
in call from ns_single_vbucket_mover:mover/6
ancestors: [<0.18433.2>,<0.18223.2>,<0.8941.0>,mb_master_sup,mb_master,
ns_server_sup,'ns_server_sup-wrapper',ns_server_cluster_sup,
<0.58.0>]
messages: [{'EXIT',<0.18433.2>,
{unexpected_exit,
{'EXIT',<0.19354.2>,
{wait_seqno_persisted_failed,"bucket0",329,179,
[{'ns_1@172.23.107.27',
{'EXIT',
{{badmatch,wrong_rebalancer_pid},
[{janitor_agent,wait_seqno_persisted,5},
{ns_single_vbucket_mover, '-wait_seqno_persisted_many/5-fun-0-',5},
{misc,'-parallel_map/3-fun-0-',4}

]}}}]}}}}]
links: [<0.18433.2>]
dictionary: [

{cleanup_list,[<0.19449.2>]}

]
trap_exit: true
status: running
heap_size: 987
stack_size: 24
reductions: 1915
neighbours:

Uploading logs