Broken Simple tests: Memcached crashes with Segmentation fault during view queries test

Build Tested - 3.0.0-502-rel

Jenkins Reference Link:
http://qa.sc.couchbase.com/view/3.0.0/job/ubuntu_x64--65_02--view_query_extended-P1/48/consoleFull

Test Failed:
./testrunner -i myfile.ini -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_out,num_nodes_to_add=1,limit=1000
test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,limit=1,skip_rebalance=true,GROUP=P1
test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,limit=10000,skip_rebalance=true,GROUP=P1
test_employee_dataset_startkey_endkey_queries_rebalance_out,num_nodes_to_add=1,limit=1,GROUP=P1

root@mulberry-s10708:/tmp# gdb /opt/couchbase/bin/memcached core.memcached.14315
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New LWP 14343]
[New LWP 14344]
[New LWP 14326]
[New LWP 14325]
[New LWP 16985]
[New LWP 16983]
[New LWP 16984]
[New LWP 16987]
[New LWP 16986]
[New LWP 14315]
[New LWP 14340]
[New LWP 14342]
[New LWP 14341]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007eff4f7bf762 in UprProducer::getNextItem (this=0x1b60300) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/upr-producer.cc:340
340 /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/upr-producer.cc: No such file or directory.
(gdb) t a a bt

Thread 13 (Thread 0x7eff5224d700 (LWP 14341)):
#0 0x00007eff54885353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007eff55a775a6 in epoll_dispatch (base=0xae82280, tv=<optimized out>) at epoll.c:404
#2 0x00007eff55a62a04 in event_base_loop (base=0xae82280, flags=<optimized out>) at event.c:1558
#3 0x00007eff562de0af in platform_thread_wrap (arg=0x1b02190) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007eff5525fe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007eff54884cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7eff51a4c700 (LWP 14342)):
#0 0x00007eff54885353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007eff55a775a6 in epoll_dispatch (base=0xae82c80, tv=<optimized out>) at epoll.c:404
#2 0x00007eff55a62a04 in event_base_loop (base=0xae82c80, flags=<optimized out>) at event.c:1558
#3 0x00007eff562de0af in platform_thread_wrap (arg=0x1b02180) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007eff5525fe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007eff54884cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7eff52a4e700 (LWP 14340)):
#0 0x00007eff54885353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007eff55a775a6 in epoll_dispatch (base=0xae82500, tv=<optimized out>) at epoll.c:404
#2 0x00007eff55a62a04 in event_base_loop (base=0xae82500, flags=<optimized out>) at event.c:1558
#3 0x00007eff562de0af in platform_thread_wrap (arg=0x1b021a0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007eff5525fe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007eff54884cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7eff56afe740 (LWP 14315)):
#0 0x00007eff54885353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007eff55a775a6 in epoll_dispatch (base=0xae82000, tv=<optimized out>) at epoll.c:404
#2 0x00007eff55a62a04 in event_base_loop (base=0xae82000, flags=<optimized out>) at event.c:1558
#3 0x000000000040eb49 in main (argc=<optimized out>, argv=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:8284

Thread 9 (Thread 0x7eff4c3af700 (LWP 16986)):
#0 0x00007eff552640fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007eff562de19b in cb_cond_timedwait (cond=0xaee6de0, mutex=0xaee6da8, ms=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
#2 0x00007eff4f78500f in SyncObject::wait (this=0xaee6da0, tv=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/syncobject.h:74
#3 0x00007eff4f781010 in ExecutorPool::trySleep (this=0xaee6d80, t=..., now=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:177
#4 0x00007eff4f781354 in ExecutorPool::nextTask (this=0xaee6d80, t=..., tick=8 '\b') at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:151
#5 0x00007eff4f7910f7 in ExecutorThread::run (this=0xae98730) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:77
#6 0x00007eff4f7914fd in launch_executor_thread (arg=0xaee6de4) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
#7 0x00007eff562de0af in platform_thread_wrap (arg=0x1b02c20) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#8 0x00007eff5525fe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007eff54884cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7eff4bbae700 (LWP 16987)):
#0 0x00007eff552640fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007eff562de19b in cb_cond_timedwait (cond=0xaee6de0, mutex=0xaee6da8, ms=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
#2 0x00007eff4f78500f in SyncObject::wait (this=0xaee6da0, tv=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/syncobject.h:74
#3 0x00007eff4f781010 in ExecutorPool::trySleep (this=0xaee6d80, t=..., now=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:177
#4 0x00007eff4f781354 in ExecutorPool::nextTask (this=0xaee6d80, t=..., tick=16 '\020') at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:151
#5 0x00007eff4f7910f7 in ExecutorThread::run (this=0xae986e0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:77
#6 0x00007eff4f7914fd in launch_executor_thread (arg=0xaee6de4) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
--Type <return> to continue, or q <return> to quit--
#7 0x00007eff562de0af in platform_thread_wrap (arg=0x1b02c00) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#8 0x00007eff5525fe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007eff54884cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7eff4d3b1700 (LWP 16984)):
#0 0x00007eff552640fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007eff562de19b in cb_cond_timedwait (cond=0xaee6de0, mutex=0xaee6da8, ms=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
#2 0x00007eff4f78500f in SyncObject::wait (this=0xaee6da0, tv=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/syncobject.h:74
#3 0x00007eff4f781010 in ExecutorPool::trySleep (this=0xaee6d80, t=..., now=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:177
#4 0x00007eff4f781354 in ExecutorPool::nextTask (this=0xaee6d80, t=..., tick=0 '\000') at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:151
#5 0x00007eff4f7910f7 in ExecutorThread::run (this=0xae987d0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:77
#6 0x00007eff4f7914fd in launch_executor_thread (arg=0xaee6de4) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
#7 0x00007eff562de0af in platform_thread_wrap (arg=0x1b02c30) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#8 0x00007eff5525fe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007eff54884cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7eff50249700 (LWP 16983)):
#0 0x00007eff5485083d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007eff5487e774 in usleep () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007eff4f78faa5 in updateStatsThread (arg=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/memory_tracker.cc:36
#3 0x00007eff562de0af in platform_thread_wrap (arg=0x1b021b0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007eff5525fe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007eff54884cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7eff4cbb0700 (LWP 16985)):
#0 0x00007eff5526689c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007eff552692b8 in _L_cond_lock_874 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007eff55269088 in __pthread_mutex_cond_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007eff552641b2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007eff562de19b in cb_cond_timedwait (cond=0xaee6de0, mutex=0xaee6da8, ms=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
#5 0x00007eff4f78500f in SyncObject::wait (this=0xaee6da0, tv=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/syncobject.h:74
#6 0x00007eff4f781010 in ExecutorPool::trySleep (this=0xaee6d80, t=..., now=...) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:177
#7 0x00007eff4f781354 in ExecutorPool::nextTask (this=0xaee6d80, t=..., tick=24 '\030') at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:151
#8 0x00007eff4f7910f7 in ExecutorThread::run (this=0xae98780) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:77
#9 0x00007eff4f7914fd in launch_executor_thread (arg=0xaee6da8) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
#10 0x00007eff562de0af in platform_thread_wrap (arg=0x1b02c10) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#11 0x00007eff5525fe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#12 0x00007eff54884cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#13 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7eff53e61700 (LWP 14325)):
#0 0x00007eff548778bd in read () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007eff5480bff8 in _IO_file_underflow () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007eff5480d03e in _IO_default_uflow () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007eff5480118a in _IO_getline_info () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007eff5480006b in fgets () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007eff53e62aa1 in fgets (__stream=<optimized out>, __n=<optimized out>, __s=<optimized out>) at /usr/include/bits/stdio2.h:255
#6 check_stdin_thread (arg=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/extensions/daemon/stdin_check.c:38
#7 0x00007eff562de0af in platform_thread_wrap (arg=0x1b02080) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#8 0x00007eff5525fe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007eff54884cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()
--Type <return> to continue, or q <return> to quit--
Thread 3 (Thread 0x7eff5345c700 (LWP 14326)):
#0 0x00007eff552640fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007eff562de19b in cb_cond_timedwait (cond=0x7eff53660200, mutex=0x7eff536601c0, ms=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
#2 0x00007eff5345f04d in logger_thead_main (arg=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/extensions/loggers/file_logger.c:362
#3 0x00007eff562de0af in platform_thread_wrap (arg=0x1b02090) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007eff5525fe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007eff54884cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7eff50a4a700 (LWP 14344)):
#0 0x00007eff54885353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007eff55a775a6 in epoll_dispatch (base=0xae82780, tv=<optimized out>) at epoll.c:404
#2 0x00007eff55a62a04 in event_base_loop (base=0xae82780, flags=<optimized out>) at event.c:1558
#3 0x00007eff562de0af in platform_thread_wrap (arg=0x1b02160) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007eff5525fe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007eff54884cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7eff5124b700 (LWP 14343)):
#0 0x00007eff4f7bf762 in UprProducer::getNextItem (this=0x1b60300) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/upr-producer.cc:340
#1 0x00007eff4f7bf885 in UprProducer::step (this=0x1b60300, producers=0x61f520) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/upr-producer.cc:148
#2 0x00007eff4f765126 in EvpUprStep (handle=<optimized out>, cookie=0xaddc400, producers=0x61f520) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/ep-engine/src/ep_engine.cc:1286
#3 0x00007eff52a57256 in upr_step (handle=<optimized out>, cookie=0xaddc400, producers=0x61f520)
at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/engines/bucket_engine/bucket_engine.c:2158
#4 0x0000000000412a6d in ship_upr_log (c=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:3553
#5 conn_ship_log (c=0xaddc400) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:6386
#6 0x000000000040999d in event_handler (fd=<optimized out>, which=<optimized out>, arg=0xaddc400) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:6828
#7 0x00007eff55a62afc in event_process_active_single_queue (activeq=<optimized out>, base=<optimized out>) at event.c:1308
#8 event_process_active (base=<optimized out>) at event.c:1375
#9 event_base_loop (base=0xae82a00, flags=<optimized out>) at event.c:1572
#10 0x00007eff562de0af in platform_thread_wrap (arg=0x1b02170) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#11 0x00007eff5525fe9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#12 0x00007eff54884cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#13 0x0000000000000000 in ?? ()
(gdb)

Uploading Logs.