Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-10641

Memcached crashes while running view-queries

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Test Blocker
    • 3.0
    • 3.0
    • couchbase-bucket
    • Security Level: Public
    • None
    • Untriaged
    • Yes
    • June 30 - July 18

    Description

      Test repro:
      ./testrunner -i /tmp/centos64.ini max-dupe-result-count=10,get-cbcollect-info=True,num-tries=60,attempt-num=60,get-delays=True,GROUP=P0 -t view.viewquerytests.ViewQueryTests.test_employee_dataset_stale_queries,GROUP=P0

      Stack:
      [root@tambran-s21714 tmp]# gdb /opt/couchbase/bin/memcached core.memcached.27015
      GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
      Copyright (C) 2010 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law. Type "show copying"
      and "show warranty" for details.
      This GDB was configured as "x86_64-redhat-linux-gnu".
      For bug reporting instructions, please see:
      <http://www.gnu.org/software/gdb/bugs/>...
      Reading symbols from /opt/couchbase/bin/memcached...done.
      [New Thread 27018]
      [New Thread 27101]
      [New Thread 27099]
      [New Thread 27102]
      [New Thread 27020]
      [New Thread 27017]
      [New Thread 27022]
      [New Thread 27098]
      [New Thread 27019]
      [New Thread 27021]
      [New Thread 27016]
      [New Thread 27100]
      [New Thread 27015]
      Reading symbols from /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0...done.
      Loaded symbols for /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0
      Reading symbols from /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1...done.
      Loaded symbols for /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1
      Reading symbols from /opt/couchbase/bin/../lib/libplatform.so.0.1.0...done.
      Loaded symbols for /opt/couchbase/bin/../lib/libplatform.so.0.1.0
      Reading symbols from /opt/couchbase/bin/../lib/libcJSON.so.1.0.0...done.
      Loaded symbols for /opt/couchbase/bin/../lib/libcJSON.so.1.0.0
      Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...done.
      Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1
      Reading symbols from /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4...done.
      Loaded symbols for /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4
      Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5...done.
      Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.0.so.5
      Reading symbols from /usr/lib64/libssl.so.6...(no debugging symbols found)...done.
      Loaded symbols for /usr/lib64/libssl.so.6
      Reading symbols from /usr/lib64/libcrypto.so.6...(no debugging symbols found)...done.
      Loaded symbols for /usr/lib64/libcrypto.so.6
      Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
      [Thread debugging using libthread_db enabled]
      Loaded symbols for /lib64/libpthread.so.0
      Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libdl.so.2
      Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib64/librt.so.1
      Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libm.so.6
      Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libc.so.6
      Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
      Loaded symbols for /usr/lib64/libstdc++.so.6
      Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libgcc_s.so.1
      Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libgssapi_krb5.so.2
      Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libkrb5.so.3
      Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libcom_err.so.2
      Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libk5crypto.so.3
      Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libz.so.1
      Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib64/ld-linux-x86-64.so.2
      Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libkrb5support.so.0
      Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libkeyutils.so.1
      Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libresolv.so.2
      Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib64/libselinux.so.1
      Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
      Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
      Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
      Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
      Loaded symbols for /opt/couchbase/lib/memcached/ep.so
      Reading symbols from /opt/couchbase/lib/libcouchstore.so...done.
      Loaded symbols for /opt/couchbase/lib/libcouchstore.so
      Reading symbols from /opt/couchbase/lib/libdirutils.so.0.1.0...done.
      Loaded symbols for /opt/couchbase/lib/libdirutils.so.0.1.0
      Reading symbols from /opt/couchbase/lib/libsigar.so...done.
      Loaded symbols for /opt/couchbase/lib/libsigar.so
      Reading symbols from /opt/couchbase/lib/libv8.so...done.
      Loaded symbols for /opt/couchbase/lib/libv8.so
      Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
      Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
      Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
      Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
      Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
      Loaded symbols for /opt/couchbase/lib/libicudata.so.44
      Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
      Program terminated with signal 6, Aborted.
      #0 0x00007f2945a178a5 in raise () from /lib64/libc.so.6
      Missing separate debuginfos, use: debuginfo-install couchbase-server-3.0.0-510.x86_64
      (gdb) t a a bt

      Thread 13 (Thread 0x7f2947c6f7e0 (LWP 27015)):
      #0 0x00007f2945acdf03 in epoll_wait () from /lib64/libc.so.6
      #1 0x00007f2946bf0376 in epoll_dispatch (base=0xb176000, tv=<value optimized out>) at epoll.c:404
      #2 0x00007f2946bdbc44 in event_base_loop (base=0xb176000, flags=<value optimized out>) at event.c:1558
      #3 0x000000000040dbeb in main (argc=<value optimized out>, argv=<value optimized out>)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:8284

      Thread 12 (Thread 0x7f293b871700 (LWP 27100)):
      #0 0x00007f2945ac4253 in poll () from /lib64/libc.so.6
      #1 0x00007f293e8a2010 in CouchNotifier::waitForReadable (this=0xb208000, tryOnce=true)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-notifier.cc:719
      #2 0x00007f293e8a27ad in waitOnce (this=0xb208000, vbs=..., file_version=12, header_offset=1204224, cb=...)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-notifier.cc:769
      #3 CouchNotifier::notify_update (this=0xb208000, vbs=..., file_version=12, header_offset=1204224, cb=...)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-notifier.cc:848
      #4 0x00007f293e896d4d in CouchKVStore::saveDocs (this=0xb1de000, vbid=48, rev=<value optimized out>, docs=0x1dee4a8, docinfos=0x1dee5f0,
      docCount=<value optimized out>, kvctx=...)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-notifier.h:172
      #5 0x00007f293e897ad6 in CouchKVStore::commit2couchstore (this=0xb1de000, cb=0x7f293b870990)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-kvstore.cc:1594
      #6 0x00007f293e897df5 in CouchKVStore::commit (this=0x7f293b870490, cb=0x1)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-kvstore.cc:1067
      #7 0x00007f293e7f149f in EventuallyPersistentStore::flushVBucket (this=0xb194480, vbid=<value optimized out>)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/ep.cc:2492
      #8 0x00007f293e82c1ac in Flusher::flushVB (this=0xb212000)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/flusher.cc:282
      #9 0x00007f293e82cbc8 in Flusher::step (this=0x7f293b870490, tid=1)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/flusher.cc:172
      #10 0x00007f293e85a014 in FlusherTask::run (this=<value optimized out>)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/tasks.cc:76
      #11 0x00007f293e8359c9 in ExecutorThread::run (this=0x1e4e780)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:93
      #12 0x00007f293e835ec6 in launch_executor_thread (arg=0x7f293b870490)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
      #13 0x00007f2947457b7f in platform_thread_wrap (arg=0x1df2bf0)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
      #14 0x00007f294640f851 in start_thread () from /lib64/libpthread.so.0
      #15 0x00007f2945acd90d in clone () from /lib64/libc.so.6

      --Type <return> to continue, or q <return> to quit--
      Thread 11 (Thread 0x7f2943f10700 (LWP 27016)):
      #0 0x00007f2945ac060d in read () from /lib64/libc.so.6
      #1 0x00007f2945a56f68 in _IO_new_file_underflow () from /lib64/libc.so.6
      #2 0x00007f2945a58a6e in _IO_default_uflow_internal () from /lib64/libc.so.6
      #3 0x00007f2945a4d14a in _IO_getline_info_internal () from /lib64/libc.so.6
      #4 0x00007f2945a4bfa9 in fgets () from /lib64/libc.so.6
      #5 0x00007f2943f118c1 in check_stdin_thread (arg=<value optimized out>)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/memcached/extensions/daemon/stdin_check.c:38
      #6 0x00007f2947457b7f in platform_thread_wrap (arg=0x1df2080)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
      #7 0x00007f294640f851 in start_thread () from /lib64/libpthread.so.0
      #8 0x00007f2945acd90d in clone () from /lib64/libc.so.6

      Thread 10 (Thread 0x7f29408fc700 (LWP 27021)):
      #0 0x00007f2945acdf03 in epoll_wait () from /lib64/libc.so.6
      #1 0x00007f2946bf0376 in epoll_dispatch (base=0xb176a00, tv=<value optimized out>) at epoll.c:404
      #2 0x00007f2946bdbc44 in event_base_loop (base=0xb176a00, flags=<value optimized out>) at event.c:1558
      #3 0x00007f2947457b7f in platform_thread_wrap (arg=0x1df2170)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
      #4 0x00007f294640f851 in start_thread () from /lib64/libpthread.so.0
      #5 0x00007f2945acd90d in clone () from /lib64/libc.so.6

      Thread 9 (Thread 0x7f2941cfe700 (LWP 27019)):
      #0 0x00007f2945acdf03 in epoll_wait () from /lib64/libc.so.6
      #1 0x00007f2946bf0376 in epoll_dispatch (base=0xb176280, tv=<value optimized out>) at epoll.c:404
      #2 0x00007f2946bdbc44 in event_base_loop (base=0xb176280, flags=<value optimized out>) at event.c:1558
      #3 0x00007f2947457b7f in platform_thread_wrap (arg=0x1df2190)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
      #4 0x00007f294640f851 in start_thread () from /lib64/libpthread.so.0
      #5 0x00007f2945acd90d in clone () from /lib64/libc.so.6

      Thread 8 (Thread 0x7f293f4fa700 (LWP 27098)):
      #0 0x00007f2945a91b8d in nanosleep () from /lib64/libc.so.6
      #1 0x00007f2945ac6d64 in usleep () from /lib64/libc.so.6
      #2 0x00007f293e833765 in updateStatsThread (arg=<value optimized out>)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/memory_tracker.cc:36
      #3 0x00007f2947457b7f in platform_thread_wrap (arg=0x1df21d0)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
      #4 0x00007f294640f851 in start_thread () from /lib64/libpthread.so.0
      #5 0x00007f2945acd90d in clone () from /lib64/libc.so.6
      --Type <return> to continue, or q <return> to quit--

      Thread 7 (Thread 0x7f293fefb700 (LWP 27022)):
      #0 0x00007f2945acdf03 in epoll_wait () from /lib64/libc.so.6
      #1 0x00007f2946bf0376 in epoll_dispatch (base=0xb176780, tv=<value optimized out>) at epoll.c:404
      #2 0x00007f2946bdbc44 in event_base_loop (base=0xb176780, flags=<value optimized out>) at event.c:1558
      #3 0x00007f2947457b7f in platform_thread_wrap (arg=0x1df2160)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
      #4 0x00007f294640f851 in start_thread () from /lib64/libpthread.so.0
      #5 0x00007f2945acd90d in clone () from /lib64/libc.so.6

      Thread 6 (Thread 0x7f294330c700 (LWP 27017)):
      #0 0x00007f29464137bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00007f29474578fb in cb_cond_timedwait (cond=0x7f294350f800, mutex=0x7f294350f7c0, ms=<value optimized out>)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
      #2 0x00007f294330ed4d in logger_thead_main (arg=0x1e2ab00)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/memcached/extensions/loggers/file_logger.c:362
      #3 0x00007f2947457b7f in platform_thread_wrap (arg=0x1df2090)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
      #4 0x00007f294640f851 in start_thread () from /lib64/libpthread.so.0
      #5 0x00007f2945acd90d in clone () from /lib64/libc.so.6

      Thread 5 (Thread 0x7f29412fd700 (LWP 27020)):
      #0 0x00007f2945acdf03 in epoll_wait () from /lib64/libc.so.6
      #1 0x00007f2946bf0376 in epoll_dispatch (base=0xb176c80, tv=<value optimized out>) at epoll.c:404
      #2 0x00007f2946bdbc44 in event_base_loop (base=0xb176c80, flags=<value optimized out>) at event.c:1558
      #3 0x00007f2947457b7f in platform_thread_wrap (arg=0x1df2180)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
      #4 0x00007f294640f851 in start_thread () from /lib64/libpthread.so.0
      #5 0x00007f2945acd90d in clone () from /lib64/libc.so.6

      Thread 4 (Thread 0x7f293a46f700 (LWP 27102)):
      #0 0x00007f2946416054 in __lll_lock_wait () from /lib64/libpthread.so.0
      #1 0x00007f2946411388 in _L_lock_854 () from /lib64/libpthread.so.0
      #2 0x00007f2946411257 in pthread_mutex_lock () from /lib64/libpthread.so.0
      #3 0x00007f2947457a09 in cb_mutex_enter (mutex=<value optimized out>)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:85
      #4 0x00007f293e83491d in Mutex::acquire (this=0xb17a900)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/mutex.cc:31
      #5 0x00007f293e85c53c in lock (this=0xb17a900, task=..., waketime=..., taskType=@0x1e4eb38, now=...)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/locks.h:66
      --Type <return> to continue, or q <return> to quit--
      #6 LockHolder (this=0xb17a900, task=..., waketime=..., taskType=@0x1e4eb38, now=...)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/locks.h:44
      #7 TaskQueue::fetchNextTask (this=0xb17a900, task=..., waketime=..., taskType=@0x1e4eb38, now=...)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/taskqueue.cc:61
      #8 0x00007f293e825790 in ExecutorPool::nextTask (this=0xb1ced80, t=..., tick=<value optimized out>)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:146
      #9 0x00007f293e835939 in ExecutorThread::run (this=0x1e4eaf0)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:77
      #10 0x00007f293e835ec6 in launch_executor_thread (arg=0xb17a908)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
      #11 0x00007f2947457b7f in platform_thread_wrap (arg=0x1df2be0)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
      #12 0x00007f294640f851 in start_thread () from /lib64/libpthread.so.0
      #13 0x00007f2945acd90d in clone () from /lib64/libc.so.6

      Thread 3 (Thread 0x7f293c272700 (LWP 27099)):
      #0 0x00007f2946416054 in __lll_lock_wait () from /lib64/libpthread.so.0
      #1 0x00007f2946411388 in _L_lock_854 () from /lib64/libpthread.so.0
      #2 0x00007f2946411257 in pthread_mutex_lock () from /lib64/libpthread.so.0
      #3 0x00007f2947457a09 in cb_mutex_enter (mutex=<value optimized out>)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:85
      #4 0x00007f293e83491d in Mutex::acquire (this=0xb17b080)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/mutex.cc:31
      #5 0x00007f293e85c53c in lock (this=0xb17b080, task=..., waketime=..., taskType=@0x1e4e818, now=...)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/locks.h:66
      #6 LockHolder (this=0xb17b080, task=..., waketime=..., taskType=@0x1e4e818, now=...)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/locks.h:44
      #7 TaskQueue::fetchNextTask (this=0xb17b080, task=..., waketime=..., taskType=@0x1e4e818, now=...)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/taskqueue.cc:61
      #8 0x00007f293e825790 in ExecutorPool::nextTask (this=0xb1ced80, t=..., tick=<value optimized out>)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:146
      #9 0x00007f293e835939 in ExecutorThread::run (this=0x1e4e7d0)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:77
      #10 0x00007f293e835ec6 in launch_executor_thread (arg=0xb17b088)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
      #11 0x00007f2947457b7f in platform_thread_wrap (arg=0x1df2c10)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
      #12 0x00007f294640f851 in start_thread () from /lib64/libpthread.so.0
      #13 0x00007f2945acd90d in clone () from /lib64/libc.so.6

      --Type <return> to continue, or q <return> to quit--
      Thread 2 (Thread 0x7f293ae70700 (LWP 27101)):
      #0 0x00007f2946412659 in pthread_mutex_unlock () from /lib64/libpthread.so.0
      #1 0x00007f2947457999 in cb_mutex_exit (mutex=<value optimized out>)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:99
      #2 0x00007f293e85c5c0 in TaskQueue::fetchNextTask (this=0xb17aa80, task=..., waketime=..., taskType=@0x1e4e778, now=...)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/taskqueue.cc:75
      #3 0x00007f293e825790 in ExecutorPool::nextTask (this=0xb1ced80, t=..., tick=<value optimized out>)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:146
      #4 0x00007f293e835939 in ExecutorThread::run (this=0x1e4e730)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:77
      #5 0x00007f293e835ec6 in launch_executor_thread (arg=0xb1ceda8)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
      #6 0x00007f2947457b7f in platform_thread_wrap (arg=0x1df2c00)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
      #7 0x00007f294640f851 in start_thread () from /lib64/libpthread.so.0
      #8 0x00007f2945acd90d in clone () from /lib64/libc.so.6

      Thread 1 (Thread 0x7f29426ff700 (LWP 27018)):
      #0 0x00007f2945a178a5 in raise () from /lib64/libc.so.6
      #1 0x00007f2945a19085 in abort () from /lib64/libc.so.6
      #2 0x00007f2945a10a1e in __assert_fail_base () from /lib64/libc.so.6
      #3 0x00007f2945a10ae0 in __assert_fail () from /lib64/libc.so.6
      #4 0x00007f293e7e064b in CheckpointManager::queueDirty (this=0xb399b98, vb=..., qi=..., genSeqno=true)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/checkpoint.cc:787
      #5 0x00007f293e7e75fb in EventuallyPersistentStore::queueDirty (this=0xb194480, vb=..., v=0x105dabe0, tapBackfill=false,
      notifyReplicator=true, genBySeqno=true) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/ep.cc:2617
      #6 0x00007f293e7f9a34 in EventuallyPersistentStore::setWithMeta (this=0xb194480, itm=..., cas=0, cookie=0xb52c0e0, force=true,
      allowExisting=true, nru=2 '\002', genBySeqno=true) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/ep.cc:1649
      #7 0x00007f293e83e35c in TapConsumer::mutation (this=0xb52c0e0, opaque=<value optimized out>, key=<value optimized out>,
      nkey=<value optimized out>, value=0xbde0043, nvalue=<value optimized out>, cas=600834636561903, vbucket=75, flags=0, datatype=0 '\000',
      locktime=0, bySeqno=0, revSeqno=1, exptime=0, nru=2 '\002', meta=0x0, nmeta=0)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/tapconnection.cc:2142
      #8 0x00007f293e804de6 in EventuallyPersistentEngine::tapNotify (this=0xb1c0000, cookie=0xb11dc00, engine_specific=0xbde0028, nengine=9,
      ttl=<value optimized out>, tap_flags=1, tap_event=1, tap_seqno=18128, key=0xbde0031, nkey=18, flags=0, exptime=0, cas=600834636561903,
      datatype=0 '\000', data=0xbde0043, ndata=510, vbucket=75)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/ep_engine.cc:2494
      #9 0x00007f293e8055f9 in EvpTapNotify(ENGINE_HANDLE *, const void *, void *, uint16_t, uint8_t, uint16_t, ._156, uint32_t, const void *, size_t, uint32_t, uint32_t, uint64_t, uint8_t, const void *, size_t, uint16_t) (handle=0xb1c0000, cookie=0xb11dc00, engine_specific=0xbde0028,
      nengine=9, ttl=254 '\376', tap_flags=1, tap_event=TAP_MUTATION, tap_seqno=18128, key=0xbde0031, nkey=18, flags=0, exptime=0,
      cas=600834636561903, datatype=0 '\000', data=0xbde0043, ndata=510, vbucket=75)
      --Type <return> to continue, or q <return> to quit--
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/ep_engine.cc:1238
      #10 0x00007f2942704f8d in bucket_tap_notify (handle=0x7f294290bba0, cookie=0xb11dc00, engine_specific=0xbde0028, nengine=9, ttl=254 '\376',
      tap_flags=<value optimized out>, tap_event=TAP_MUTATION, tap_seqno=18128, key=0xbde0031, nkey=18, flags=0, exptime=0, cas=600834636561903,
      datatype=0 '\000', data=0xbde0043, ndata=510, vbucket=75)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/memcached/engines/bucket_engine/bucket_engine.c:2085
      #11 0x0000000000410467 in process_bin_tap_packet (event=TAP_MUTATION, c=0xb11dc00)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:2983
      #12 0x0000000000415ed6 in process_bin_packet (c=0xb11dc00)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:4825
      #13 complete_nread (c=0xb11dc00) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:5384
      #14 conn_nread (c=0xb11dc00) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:6555
      #15 0x0000000000408f6d in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0xb11dc00)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:6828
      #16 0x00007f2946bdbd3c in event_process_active_single_queue (base=0xb176500, flags=<value optimized out>) at event.c:1308
      #17 event_process_active (base=0xb176500, flags=<value optimized out>) at event.c:1375
      #18 event_base_loop (base=0xb176500, flags=<value optimized out>) at event.c:1572
      #19 0x00007f2947457b7f in platform_thread_wrap (arg=0x1df21a0)
      at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
      #20 0x00007f294640f851 in start_thread () from /lib64/libpthread.so.0
      #21 0x00007f2945acd90d in clone () from /lib64/libc.so.6
      (gdb)

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ketaki Ketaki Gangal (Inactive)
            ketaki Ketaki Gangal (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty