Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
7.2.1
-
Enterprise Edition 7.2.1 build 5902
-
Triaged
-
Centos 64-bit
-
0
-
Yes
Description
Script to Repro
./sequoia -client 172.23.104.27:2375 -provider file:centos_pine.yml -test tests/integration/7.2/test_7.2.yml -scope tests/integration/7.2/scope_7.2_magma.yml -scale 3 -repeat 0 -log_level 0 -version 7.2.1-5902 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=1209600 -show_topology=true
|
Longevity ran for less than 24 hours when we saw this crash. Did not see it on 7.2.1-5893. Hence marking it a regression.
grep CRITICAL on 172.23.99.25
balakumaran.g@Balakumarans-MacBook-Pro-2 cbcollect_info_ns_1@172.23.99.25_20230806-020006 % grep CRITICAL memcached.log
|
2023-08-05T05:11:24.311554-07:00 CRITICAL *** Fatal error encountered during exception handling ***
|
2023-08-05T05:11:24.311626-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): CheckpointManager::queueDirty: Got status:failure:duplicate item when vb:926 is non-active:2, item:[op:pending_sync_write, seqno:108800, key:<ud>cid:0x0:00000000000000003632</ud>], lastBySeqno:108799, openCkpt:[start:108800, end:108800]
|
2023-08-05T05:12:46.385577-07:00 CRITICAL Detected previous crash
|
2023-08-05T05:12:46.385617-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.2.1-5902). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/6c760ec0-e81a-4102-d33723a8-b5f58e42.dmp before terminating.
|
2023-08-05T05:12:46.385627-07:00 CRITICAL Stack backtrace of crashed thread:
|
2023-08-05T05:12:46.385628-07:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x749638]
|
2023-08-05T05:12:46.385634-07:00 CRITICAL #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x79b17a]
|
2023-08-05T05:12:46.385635-07:00 CRITICAL #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x79b4b8]
|
2023-08-05T05:12:46.385636-07:00 CRITICAL #3 /lib64/libpthread.so.0() [0x7f8828f7f000+0xf630]
|
2023-08-05T05:12:46.385637-07:00 CRITICAL #4 /lib64/libc.so.6(gsignal+0x37) [0x7f88267d0000+0x36387]
|
2023-08-05T05:12:46.385640-07:00 CRITICAL #5 /lib64/libc.so.6(abort+0x148) [0x7f88267d0000+0x37a78]
|
2023-08-05T05:12:46.385641-07:00 CRITICAL #6 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f88270b8000+0x9963c]
|
2023-08-05T05:12:46.385642-07:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x753bab]
|
2023-08-05T05:12:46.385642-07:00 CRITICAL #8 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f88270b8000+0xa48f6]
|
2023-08-05T05:12:46.385653-07:00 CRITICAL #9 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f88270b8000+0xa4961]
|
2023-08-05T05:12:46.385653-07:00 CRITICAL #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f88270b8000+0xa4bf4]
|
2023-08-05T05:12:46.385654-07:00 CRITICAL #11 /opt/couchbase/bin/memcached() [0x400000+0xc1f61]
|
2023-08-05T05:12:46.385655-07:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0x33fa09]
|
2023-08-05T05:12:46.385661-07:00 CRITICAL #13 /opt/couchbase/bin/memcached() [0x400000+0x34003c]
|
2023-08-05T05:12:46.385662-07:00 CRITICAL #14 /opt/couchbase/bin/memcached() [0x400000+0x448066]
|
2023-08-05T05:12:46.385683-07:00 CRITICAL #15 /opt/couchbase/bin/memcached() [0x400000+0x33a5ca]
|
2023-08-05T05:12:46.385684-07:00 CRITICAL #16 /opt/couchbase/bin/memcached() [0x400000+0x33abb4]
|
2023-08-05T05:12:46.385685-07:00 CRITICAL #17 /opt/couchbase/bin/memcached() [0x400000+0x33e084]
|
2023-08-05T05:12:46.385703-07:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0x30721a]
|
2023-08-05T05:12:46.385704-07:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0x507cd6]
|
2023-08-05T05:12:46.385706-07:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0x5082f8]
|
2023-08-05T05:12:46.385707-07:00 CRITICAL #21 /opt/couchbase/bin/memcached() [0x400000+0x50872b]
|
2023-08-05T05:12:46.385709-07:00 CRITICAL #22 /opt/couchbase/bin/memcached() [0x400000+0x4f9372]
|
2023-08-05T05:12:46.385710-07:00 CRITICAL #23 /opt/couchbase/bin/memcached() [0x400000+0x4ff937]
|
2023-08-05T05:12:46.385712-07:00 CRITICAL #24 /opt/couchbase/bin/memcached() [0x400000+0x502ee5]
|
2023-08-05T05:12:46.385713-07:00 CRITICAL #25 /opt/couchbase/bin/memcached() [0x400000+0x6c21b9]
|
2023-08-05T05:12:46.385714-07:00 CRITICAL #26 /opt/couchbase/bin/memcached() [0x400000+0x6bb89a]
|
2023-08-05T05:12:46.385715-07:00 CRITICAL #27 /opt/couchbase/bin/memcached() [0x400000+0x6c356e]
|
2023-08-05T05:12:46.385716-07:00 CRITICAL #28 /opt/couchbase/bin/memcached() [0x400000+0x8216d0]
|
2023-08-05T05:12:46.385717-07:00 CRITICAL #29 /opt/couchbase/bin/memcached() [0x400000+0x80c16a]
|
2023-08-05T05:12:46.385718-07:00 CRITICAL #30 /opt/couchbase/bin/memcached() [0x400000+0x824689]
|
2023-08-05T05:12:46.385719-07:00 CRITICAL #31 /opt/couchbase/bin/memcached() [0x400000+0x6bb594]
|
2023-08-05T05:12:46.385721-07:00 CRITICAL #32 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f88270b8000+0xcdd40]
|
2023-08-05T05:12:46.385723-07:00 CRITICAL #33 /lib64/libpthread.so.0() [0x7f8828f7f000+0x7ea5]
|
2023-08-05T05:12:46.385725-07:00 CRITICAL #34 /lib64/libc.so.6(clone+0x6d) [0x7f88267d0000+0xfeb0d]
|
balakumaran.g@Balakumarans-MacBook-Pro-2 cbcollect_info_ns_1@172.23.99.25_20230806-020006 %
|
bt of 6c760ec0-e81a-4102-d33723a8-b5f58e42.dmp on 172.23.99.25
(gdb) bt
|
#0 0x00007f8826806387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
|
#1 0x00007f8826807a78 in __GI_abort () at abort.c:90
|
#2 0x00007f882715163c in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:95
|
#3 0x0000000000b53bab in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:88
|
#4 0x00007f882715c8f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
|
#5 0x00007f882715c961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
|
#6 0x00007f882715cbf4 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7f8788000980, tinfo=0x10a05b0 <typeinfo for std::logic_error>, dest=0x445400 <_ZNSt11logic_errorD1Ev@plt>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95
|
#7 0x00000000004c1f61 in CheckpointManager::queueDirty(SingleThreadedRCPtr<Item, Item*, std::default_delete<Item> >&, GenerateBySeqno, GenerateCas, PreLinkDocumentContext*, std::function<void (long)>) (this=0x7f8739098e80, qi=...,
|
generateBySeqno=<optimized out>, generateCas=<optimized out>, preLinkDocumentContext=<optimized out>, assignedSeqnoCallback=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/checkpoint_manager.cc:915
|
#8 0x000000000073fa09 in VBucket::queueItem(SingleThreadedRCPtr<Item, Item*, std::default_delete<Item> >&, VBQueueItemCtx const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc:1304
|
#9 0x000000000074003c in VBucket::queueDirty(HashTable::HashBucketLock const&, StoredValue&, VBQueueItemCtx const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc:1394
|
#10 0x0000000000848066 in EPVBucket::addNewStoredValue (this=0x7f87390e1600, hbl=..., itm=..., queueItmCtx=..., genRevSeqno=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_vb.cc:716
|
#11 0x000000000073a5ca in VBucket::processSetInner(HashTable::FindUpdateResult&, StoredValue*&, Item&, unsigned long, bool, bool, VBQueueItemCtx const&, cb::StoreIfStatus, bool) ()
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/hash_table.h:741
|
#12 0x000000000073abb4 in VBucket::processSet(HashTable::FindUpdateResult&, StoredValue*&, Item&, unsigned long, bool, bool, VBQueueItemCtx const&, cb::StoreIfStatus, bool) ()
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc:3414
|
#13 0x000000000073e084 in VBucket::prepare(Item&, unsigned long, unsigned long*, CookieIface const*, EventuallyPersistentEngine&, CheckConflicts, bool, GenerateBySeqno, GenerateCas, Collections::VB::CachingReadHandle const&) ()
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc:1969
|
#14 0x000000000070721a in KVBucket::prepare(Item&, CookieIface const*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:1832
|
#15 0x0000000000907cd6 in PassiveStream::processMessage(MutationConsumerMessage*, PassiveStream::MessageType) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/response.h:478
|
#16 0x00000000009082f8 in PassiveStream::processPrepare(MutationConsumerMessage*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/passive_stream.cc:767
|
#17 0x000000000090872b in PassiveStream::processBufferedMessages(unsigned int&, unsigned long) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/passive_stream.cc:499
|
#18 0x00000000008f9372 in DcpConsumer::drainStreamsBufferedItems (this=this@entry=0x7f8738eb1080, stream=..., yieldThreshold=10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/consumer.cc:1276
|
#19 0x00000000008ff937 in DcpConsumer::processBufferedItems() () at /opt/gcc-10.2.0/include/c++/10.2.0/ext/atomicity.h:100
|
#20 0x0000000000902ee5 in DcpConsumerTask::run (this=0x7f8739428e10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/consumer.cc:84
|
#21 0x0000000000ac21b9 in GlobalTask::execute(std::basic_string_view<char, std::char_traits<char> >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98
|
#22 0x0000000000abb89a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f87a5fe9650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309
|
#23 0x0000000000ac356e in operator() (this=0x7f87a5fe9650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42
|
#24 CancellableCPUExecutor::add(GlobalTask*, folly::Function<void ()>)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42
|
#25 0x0000000000c216d0 in operator() (this=0x7f87a5fe9840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416
|
#26 folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f8825151d00, thread=...,
|
task=task@entry=<unknown type in /usr/lib/debug/opt/couchbase/bin/memcached-7.2.1-5902.x86_64.debug, CU 0xaa39e93, DIE 0xaabddc8>)
|
at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97
|
#27 0x0000000000c0c16a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f8825151d00, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265
|
#28 0x0000000000c24689 in __invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__t=<optimized out>,
|
__f=<optimized out>) at /usr/local/include/c++/7.3.0/bits/invoke.h:73
|
#29 __invoke<void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__fn=<optimized out>)
|
at /usr/local/include/c++/7.3.0/bits/invoke.h:95
|
#30 __call<void, 0, 1> (__args=<optimized out>, this=<optimized out>) at /usr/local/include/c++/7.3.0/functional:467
|
#31 operator()<> (this=<optimized out>) at /usr/local/include/c++/7.3.0/functional:551
|
#32 folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401
|
#33 0x0000000000abb594 in operator() (this=0x7f88255e6140) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49
|
#34 operator() (__closure=0x7f88255e6140) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49
|
#35 folly::detail::function::FunctionTraits<void ()>::callBig<CBRegisteredThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401
|
#36 0x00007f8827185d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80
|
#37 0x00007f8828f86ea5 in start_thread (arg=0x7f87a5ffb700) at pthread_create.c:307
|
#38 0x00007f88268ceb0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
|
(gdb)
|
cbcollect_info and other useful info attached.
