Details
-
Triaged
-
Yes
-
June 30 - July 18
Description
Test to repro
./testrunner -i /tmp/centos64.ini max-dupe-result-count=10,get-cbcollect-info=True,num-tries=60,attempt-num=60,get-delays=True -t view.compactionviewtests.CompactionViewTests.test_multiply_compaction,fragmentation_value=10,cycles-num=1,ddocs_num=1
Failed Job:
http://qa.sc.couchbase.com/view/3.0.0/job/centos_x64-00-01-view-compaction-P0/7/consoleFull
Stack from node
[root@centos-64-x64 tmp]# gdb /opt/couchbase/bin/memcached core.memcached.31769
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 32056]
[New Thread 31794]
[New Thread 31797]
[New Thread 31780]
[New Thread 31798]
[New Thread 31769]
[New Thread 31781]
[New Thread 32054]
[New Thread 31796]
[New Thread 31795]
[New Thread 32052]
[New Thread 32055]
[New Thread 32053]
Reading symbols from /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0
Reading symbols from /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1...done.
Loaded symbols for /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1
Reading symbols from /opt/couchbase/bin/../lib/libplatform.so.0.1.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libplatform.so.0.1.0
Reading symbols from /opt/couchbase/bin/../lib/libcJSON.so.1.0.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libcJSON.so.1.0.0
Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1
Reading symbols from /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4
Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5...done.
Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.0.so.5
Reading symbols from /usr/lib64/libssl.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.6
Reading symbols from /usr/lib64/libcrypto.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.6
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so
Reading symbols from /opt/couchbase/lib/libdirutils.so.0.1.0...done.
Loaded symbols for /opt/couchbase/lib/libdirutils.so.0.1.0
Reading symbols from /opt/couchbase/lib/libv8.so...done.
Loaded symbols for /opt/couchbase/lib/libv8.so
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
Program terminated with signal 11, Segmentation fault.
#0 SLL_Next (this=0x165bc18, src=0x165bdf8, cl=<value optimized out>, N=227) at src/linked_list.h:44
44 src/linked_list.h: No such file or directory.
in src/linked_list.h
Missing separate debuginfos, use: debuginfo-install couchbase-server-3.0.0-451.x86_64
(gdb) t a a bt
Thread 13 (Thread 0x7fe9ca442700 (LWP 32053)):
#0 0x00007fe9d3a6e85d in fdatasync () from /lib64/libc.so.6
#1 0x00007fe9cc55fc77 in couch_sync (errinfo=0xdc16020, handle=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/couchstore/src/os.c:133
#2 0x00007fe9cc8483cb in cfs_sync(struct
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-fs-stats.cc:128
#3 0x00007fe9cc5496fa in couchstore_commit (db=0xdc16000)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/couchstore/src/couch_db.cc:203
#4 0x00007fe9cc83f9d0 in CouchKVStore::saveDocs (this=0xb180000, vbid=380, rev=<value optimized out>,
docs=0x1d765b0, docinfos=0x1d76510, docCount=<value optimized out>, kvctx=...)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-kvstore.cc:1695
#5 0x00007fe9cc8408d6 in CouchKVStore::commit2couchstore (this=0xb180000, cb=0x7fe9ca441990)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-kvstore.cc:1587
#6 0x00007fe9cc840bf5 in CouchKVStore::commit (this=0x40, cb=0x40)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-kvstore.cc:1061
#7 0x00007fe9cc798ccf in EventuallyPersistentStore::flushVBucket (this=0xb0e0480, vbid=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/ep.cc:2491
#8 0x00007fe9cc7d4a3c in Flusher::flushVB (this=0xb136280)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/flusher.cc:282
#9 0x00007fe9cc7d5458 in Flusher::step (this=0x40, tid=1)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/flusher.cc:172
#10 0x00007fe9cc802b14 in FlusherTask::run (this=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/tasks.cc:76
#11 0x00007fe9cc7de359 in ExecutorThread::run (this=0x1dd2780)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:93
#12 0x00007fe9cc7de856 in launch_executor_thread (arg=0x40)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
#13 0x00007fe9d53ffb7f in platform_thread_wrap (arg=0x1d6ec30)
--
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#14 0x00007fe9d43b7851 in start_thread () from /lib64/libpthread.so.0
#15 0x00007fe9d3a7590d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fe9c9040700 (LWP 32055)):
#0 0x00007fe9d3a6e85d in fdatasync () from /lib64/libc.so.6
#1 0x00007fe9cc55fc77 in couch_sync (errinfo=0xdae43a0, handle=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/couchstore/src/os.c:133
#2 0x00007fe9cc8483cb in cfs_sync(struct {...}
*, couch_file_handle) (errinfo=0xdae43a0, h=0xc55e5a0)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-fs-stats.cc:128
#3 0x00007fe9cc5496fa in couchstore_commit (db=0xdae4380)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/couchstore/src/couch_db.cc:203
#4 0x00007fe9cc83f9d0 in CouchKVStore::saveDocs (this=0xb1816c0, vbid=290, rev=<value optimized out>,
docs=0x1d6ec80, docinfos=0x1d6ed50, docCount=<value optimized out>, kvctx=...)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-kvstore.cc:1695
#5 0x00007fe9cc8408d6 in CouchKVStore::commit2couchstore (this=0xb1816c0, cb=0x7fe9c903f990)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-kvstore.cc:1587
#6 0x00007fe9cc840bf5 in CouchKVStore::commit (this=0x41, cb=0x41)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-kvstore.cc:1061
#7 0x00007fe9cc798ccf in EventuallyPersistentStore::flushVBucket (this=0xb0e0480, vbid=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/ep.cc:2491
#8 0x00007fe9cc7d4a3c in Flusher::flushVB (this=0xb136640)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/flusher.cc:282
#9 0x00007fe9cc7d5458 in Flusher::step (this=0x41, tid=3)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/flusher.cc:172
#10 0x00007fe9cc802b14 in FlusherTask::run (this=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/tasks.cc:76
#11 0x00007fe9cc7de359 in ExecutorThread::run (this=0x1dd26e0)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:93
--Type <return> to continue, or q <return> to quit--
#12 0x00007fe9cc7de856 in launch_executor_thread (arg=0x41)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
#13 0x00007fe9d53ffb7f in platform_thread_wrap (arg=0x1d6ec20)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#14 0x00007fe9d43b7851 in start_thread () from /lib64/libpthread.so.0
#15 0x00007fe9d3a7590d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fe9cd4a2700 (LWP 32052)):
#0 0x00007fe9d3a39b8d in nanosleep () from /lib64/libc.so.6
#1 0x00007fe9d3a6ed64 in usleep () from /lib64/libc.so.6
#2 0x00007fe9cc7dc0f5 in updateStatsThread (arg=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/memory_tracker.cc:36
#3 0x00007fe9d53ffb7f in platform_thread_wrap (arg=0x1d6e1b0)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007fe9d43b7851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fe9d3a7590d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fe9cfca6700 (LWP 31795)):
#0 0x00007fe9d3a75f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fe9d4b98376 in epoll_dispatch (base=0xb0fa280, tv=<value optimized out>) at epoll.c:404
#2 0x00007fe9d4b83c44 in event_base_loop (base=0xb0fa280, flags=<value optimized out>) at event.c:1558
#3 0x00007fe9d53ffb7f in platform_thread_wrap (arg=0x1d6e180)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007fe9d43b7851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fe9d3a7590d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fe9cf2a5700 (LWP 31796)):
#0 0x00007fe9d3a75f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fe9d4b98376 in epoll_dispatch (base=0xb0fac80, tv=<value optimized out>) at epoll.c:404
--Type <return> to continue, or q <return> to quit--
#2 0x00007fe9d4b83c44 in event_base_loop (base=0xb0fac80, flags=<value optimized out>) at event.c:1558
#3 0x00007fe9d53ffb7f in platform_thread_wrap (arg=0x1d6e170)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007fe9d43b7851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fe9d3a7590d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fe9c9a41700 (LWP 32054)):
#0 0x00007fe9d3a6c253 in poll () from /lib64/libc.so.6
#1 0x00007fe9cc84a6c0 in CouchNotifier::waitForReadable (this=0xb17b000, tryOnce=true)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-notifier.cc:719
#2 0x00007fe9cc84ae5d in waitOnce (this=0xb17b000, vbs=..., file_version=5, header_offset=225280, cb=...)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-notifier.cc:769
#3 CouchNotifier::notify_update (this=0xb17b000, vbs=..., file_version=5, header_offset=225280, cb=...)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-notifier.cc:848
#4 0x00007fe9cc83fb4d in CouchKVStore::saveDocs (this=0xb182700, vbid=523, rev=<value optimized out>,
docs=0xd3bf340, docinfos=0xd3c15a0, docCount=<value optimized out>, kvctx=...)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-notifier.h:172
#5 0x00007fe9cc8408d6 in CouchKVStore::commit2couchstore (this=0xb182700, cb=0x7fe9c9a40990)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-kvstore.cc:1587
#6 0x00007fe9cc840bf5 in CouchKVStore::commit (this=0x7fe9c9a40490, cb=0x1)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/couch-kvstore/couch-kvstore.cc:1061
#7 0x00007fe9cc798ccf in EventuallyPersistentStore::flushVBucket (this=0xb0e0480, vbid=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/ep.cc:2491
#8 0x00007fe9cc7d4a3c in Flusher::flushVB (this=0xb136500)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/flusher.cc:282
#9 0x00007fe9cc7d5458 in Flusher::step (this=0x7fe9c9a40490, tid=4)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/flusher.cc:172
#10 0x00007fe9cc802b14 in FlusherTask::run (this=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/tasks.cc:76
--Type <return> to continue, or q <return> to quit--
#11 0x00007fe9cc7de359 in ExecutorThread::run (this=0x1dd2730)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:93
#12 0x00007fe9cc7de856 in launch_executor_thread (arg=0x7fe9c9a40490)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
#13 0x00007fe9d53ffb7f in platform_thread_wrap (arg=0x1d6ec10)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#14 0x00007fe9d43b7851 in start_thread () from /lib64/libpthread.so.0
#15 0x00007fe9d3a7590d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fe9d12b4700 (LWP 31781)):
#0 0x00007fe9d43bb7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fe9d53ff8fb in cb_cond_timedwait (cond=0x7fe9d14b7800, mutex=0x7fe9d14b77c0, ms=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
#2 0x00007fe9d12b6d4d in logger_thead_main (arg=0x1daeb00)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/memcached/extensions/loggers/file_logger.c:363
#3 0x00007fe9d53ffb7f in platform_thread_wrap (arg=0x1d6e080)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007fe9d43b7851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fe9d3a7590d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fe9d5c177e0 (LWP 31769)):
#0 0x00007fe9d3a75f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fe9d4b98376 in epoll_dispatch (base=0xb0fa000, tv=<value optimized out>) at epoll.c:404
#2 0x00007fe9d4b83c44 in event_base_loop (base=0xb0fa000, flags=<value optimized out>) at event.c:1558
#3 0x000000000040dc6b in main (argc=<value optimized out>, argv=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:8285
Thread 5 (Thread 0x7fe9cdea3700 (LWP 31798)):
#0 0x00007fe9d3a75f03 in epoll_wait () from /lib64/libc.so.6
--Type <return> to continue, or q <return> to quit--
#1 0x00007fe9d4b98376 in epoll_dispatch (base=0xb0fa780, tv=<value optimized out>) at epoll.c:404
#2 0x00007fe9d4b83c44 in event_base_loop (base=0xb0fa780, flags=<value optimized out>) at event.c:1558
#3 0x00007fe9d53ffb7f in platform_thread_wrap (arg=0x1d6e150)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007fe9d43b7851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fe9d3a7590d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fe9d1eb8700 (LWP 31780)):
#0 0x00007fe9d3a6860d in read () from /lib64/libc.so.6
#1 0x00007fe9d39fef68 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x00007fe9d3a00a6e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x00007fe9d39f514a in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x00007fe9d39f3fa9 in fgets () from /lib64/libc.so.6
#5 0x00007fe9d1eb98c1 in check_stdin_thread (arg=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/memcached/extensions/daemon/stdin_check.c:38
#6 0x00007fe9d53ffb7f in platform_thread_wrap (arg=0x1d6e070)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#7 0x00007fe9d43b7851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007fe9d3a7590d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fe9ce8a4700 (LWP 31797)):
#0 0x00007fe9d3a75f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fe9d4b98376 in epoll_dispatch (base=0xb0faa00, tv=<value optimized out>) at epoll.c:404
#2 0x00007fe9d4b83c44 in event_base_loop (base=0xb0faa00, flags=<value optimized out>) at event.c:1558
#3 0x00007fe9d53ffb7f in platform_thread_wrap (arg=0x1d6e160)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007fe9d43b7851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fe9d3a7590d in clone () from /lib64/libc.so.6
--Type <return> to continue, or q <return> to quit--
Thread 2 (Thread 0x7fe9d06a7700 (LWP 31794)):
#0 0x00007fe9d3a16bf8 in memcpy () from /lib64/libc.so.6
#1 0x00007fe9d37241e6 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_M_clone(std::allocator<char> const&, unsigned long) () from /usr/lib64/libstdc++.so.6
#2 0x00007fe9d3724534 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::assign(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /usr/lib64/libstdc++.so.6
#3 0x00007fe9cc812a1f in Item (this=0xd5cd400)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/item.h:307
#4 ActiveStream::nextCheckpointItem (this=0xd5cd400)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/upr-stream.cc:418
#5 0x00007fe9cc81328c in ActiveStream::inMemoryPhase (this=0xd5cd400)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/upr-stream.cc:304
#6 0x00007fe9cc8134aa in ActiveStream::next (this=0xd5cd400)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/upr-stream.cc:196
#7 0x00007fe9cc8134d3 in ActiveStream::next (this=0xd5cd400)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/upr-stream.cc:215
#8 0x00007fe9cc80e10e in UprProducer::getNextItem (this=0xb136a00)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/upr-producer.cc:299
#9 0x00007fe9cc80e239 in UprProducer::step (this=0xb136a00, producers=0x61f1e0)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/upr-producer.cc:125
#10 0x00007fe9cc7ac00f in EvpUprStep (handle=<value optimized out>, cookie=0xb0a0700, producers=0x61f1e0)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/ep_engine.cc:1287
#11 0x00007fe9d06af1e6 in upr_step (handle=<value optimized out>, cookie=0xb0a0700, producers=0x61f1e0)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/memcached/engines/bucket_engine/bucket_engine.c:2159
#12 0x0000000000411a11 in ship_upr_log (c=0xb0a0700)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:3554
#13 conn_ship_log (c=0xb0a0700)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:6387
--Type <return> to continue, or q <return> to quit--
#14 0x0000000000408fed in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0xb0a0700)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:6829
#15 0x00007fe9d4b83d3c in event_process_active_single_queue (base=0xb0fa500, flags=<value optimized out>)
at event.c:1308
#16 event_process_active (base=0xb0fa500, flags=<value optimized out>) at event.c:1375
#17 event_base_loop (base=0xb0fa500, flags=<value optimized out>) at event.c:1572
#18 0x00007fe9d53ffb7f in platform_thread_wrap (arg=0x1d6e190)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#19 0x00007fe9d43b7851 in start_thread () from /lib64/libpthread.so.0
#20 0x00007fe9d3a7590d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fe9c863f700 (LWP 32056)):
#0 SLL_Next (this=0x165bc18, src=0x165bdf8, cl=<value optimized out>, N=227) at src/linked_list.h:44
#1 SLL_PopRange (this=0x165bc18, src=0x165bdf8, cl=<value optimized out>, N=227) at src/linked_list.h:75
#2 PopRange (this=0x165bc18, src=0x165bdf8, cl=<value optimized out>, N=227) at src/thread_cache.h:228
#3 tcmalloc::ThreadCache::ReleaseToCentralCache (this=0x165bc18, src=0x165bdf8, cl=<value optimized out>, N=227)
at src/thread_cache.cc:229
#4 0x00007fe9d4db9ba3 in tcmalloc::ThreadCache::ListTooLong (this=<value optimized out>, list=0x165bdf8,
cl=<value optimized out>) at src/thread_cache.cc:191
#5 0x00007fe9d4dc0ffc in Deallocate (p=<value optimized out>) at src/thread_cache.h:389
#6 do_free_helper (p=<value optimized out>) at src/tcmalloc.cc:1180
#7 do_free_with_callback (p=<value optimized out>) at src/tcmalloc.cc:1210
#8 do_free (p=<value optimized out>) at src/tcmalloc.cc:1219
#9 tc_delete (p=<value optimized out>) at src/tcmalloc.cc:1619
#10 0x00007fe9cc783817 in _M_clear (this=0xdef4500, __in_chrg=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/item.h:193
#11 ~_List_base (this=0xdef4500, __in_chrg=<value optimized out>)
at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_list.h:339
#12 ~list (this=0xdef4500, __in_chrg=<value optimized out>)
--Type <return> to continue, or q <return> to quit--
at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_list.h:399
#13 Checkpoint::~Checkpoint (this=0xdef4500, __in_chrg=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/checkpoint.cc:67
#14 0x00007fe9cc789a01 in CheckpointManager::removeClosedUnrefCheckpoints (this=0xb3ff698, vbucket=...,
newOpenCheckpointCreated=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/checkpoint.cc:677
#15 0x00007fe9cc78dce2 in CheckpointVisitor::visitBucket (this=0xd15f420, vb=...)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/checkpoint_remover.cc:43
#16 0x00007fe9cc78eb33 in VBCBAdaptor::run (this=0xda1c5a0)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/ep.cc:2802
#17 0x00007fe9cc7de359 in ExecutorThread::run (this=0x1dd2690)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:93
#18 0x00007fe9cc7de856 in launch_executor_thread (arg=0x165bc18)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
#19 0x00007fe9d53ffb7f in platform_thread_wrap (arg=0x1d6ec00)
at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#20 0x00007fe9d43b7851 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fe9d3a7590d in clone () from /lib64/libc.so.6
(gdb)