Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.6.0
-
7.6.0-1627
-
Untriaged
-
Centos 64-bit
-
0
-
Yes
Description
Script to repro
./sequoia -client 172.23.104.168:2375 -provider file:centos_second_cluster.yml -test tests/integration/7.2/test_7.2.yml -scope tests/integration/7.2/scope_7.2_magma.yml -scale 1 -repeat 0 -log_level 0 -version 7.6.0-1627 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=1209600 -show_topology=true
|
System test ran for just a few hours when we started seeing these minidumps.
grep CRITICAL on 172.23.96.14
2023-10-11T08:56:13.722331-07:00 CRITICAL *** Fatal error encountered during exception handling ***
|
2023-10-11T08:56:13.722345-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure: 'false' at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/collections/vbucket_manifest.cc:325
|
2023-10-11T08:56:13.738801-07:00 CRITICAL (ITEM) Manifest::update vb:47 scopesToModify:true changeScopeWithDataLimitExists:false VB::Manifest: uid:6, scopeWithDataLimitExists:0, dropInProgress:0, defaultCollectionMVS:223384, defaultCollectionMaxLegacyDCPSeqno:223384, scopes.size:3, map.size:9
|
2023-10-11T08:56:13.739208-07:00 CRITICAL *** Fatal error encountered during exception handling ***
|
2023-10-11T08:56:13.748445-07:00 CRITICAL (ITEM) Manifest::update vb:43 scopesToModify:true changeScopeWithDataLimitExists:false VB::Manifest: uid:6, scopeWithDataLimitExists:0, dropInProgress:0, defaultCollectionMVS:225679, defaultCollectionMaxLegacyDCPSeqno:225679, scopes.size:3, map.size:9
|
2023-10-11T08:56:13.748627-07:00 CRITICAL *** Fatal error encountered during exception handling ***
|
2023-10-11T08:56:13.865543-07:00 CRITICAL (NEW_ORDER) Manifest::update vb:43 scopesToModify:true changeScopeWithDataLimitExists:false VB::Manifest: uid:6, scopeWithDataLimitExists:0, dropInProgress:0, defaultCollectionMVS:203388, defaultCollectionMaxLegacyDCPSeqno:203388, scopes.size:3, map.size:9
|
2023-10-11T08:56:13.865640-07:00 CRITICAL *** Fatal error encountered during exception handling ***
|
2023-10-11T08:56:14.952677-07:00 CRITICAL Detected previous crash
|
2023-10-11T08:56:14.952723-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.6.0-1627). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/3137bee1-4910-41c4-544e42b1-4ceb543a.dmp before terminating. Writing dump succeeded: yes
|
2023-10-11T08:56:14.952732-07:00 CRITICAL Stack backtrace of crashed thread:
|
2023-10-11T08:56:14.952733-07:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x850141]
|
2023-10-11T08:56:14.952734-07:00 CRITICAL #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x385) [0x400000+0x86dca5]
|
2023-10-11T08:56:14.952735-07:00 CRITICAL #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x9f) [0x400000+0x86dfef]
|
2023-10-11T08:56:14.952736-07:00 CRITICAL #3 /lib64/libpthread.so.0() [0x7f8c517f8000+0xf5e0]
|
2023-10-11T08:56:14.952737-07:00 CRITICAL #4 /lib64/libc.so.6(gsignal+0x37) [0x7f8c507d8000+0x351f7]
|
2023-10-11T08:56:14.952738-07:00 CRITICAL #5 /lib64/libc.so.6(abort+0x148) [0x7f8c507d8000+0x368e8]
|
2023-10-11T08:56:14.952739-07:00 CRITICAL #6 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8c50e9d000+0xa89ab]
|
2023-10-11T08:56:14.952740-07:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x85fee6]
|
2023-10-11T08:56:14.952741-07:00 CRITICAL #8 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8c50e9d000+0xb82fa]
|
2023-10-11T08:56:14.952742-07:00 CRITICAL #9 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8c50e9d000+0xb8365]
|
2023-10-11T08:56:14.952743-07:00 CRITICAL #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8c50e9d000+0xb85b7]
|
2023-10-11T08:56:14.952743-07:00 CRITICAL #11 /opt/couchbase/bin/memcached() [0x400000+0x9501c5]
|
2023-10-11T08:56:14.952744-07:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0x572e9]
|
2023-10-11T08:56:14.952745-07:00 CRITICAL #13 /opt/couchbase/bin/memcached() [0x400000+0xb3870]
|
2023-10-11T08:56:14.952746-07:00 CRITICAL #14 /opt/couchbase/bin/memcached() [0x400000+0x3d51d6]
|
2023-10-11T08:56:14.952747-07:00 CRITICAL #15 /opt/couchbase/bin/memcached() [0x400000+0x42a7f9]
|
2023-10-11T08:56:14.952748-07:00 CRITICAL #16 /opt/couchbase/bin/memcached() [0x400000+0x3fa363]
|
2023-10-11T08:56:14.952748-07:00 CRITICAL #17 /opt/couchbase/bin/memcached() [0x400000+0x4036c6]
|
2023-10-11T08:56:14.952749-07:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0x7a817a]
|
2023-10-11T08:56:14.952750-07:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0x7a61aa]
|
2023-10-11T08:56:14.952751-07:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0x7a94c3]
|
2023-10-11T08:56:14.952751-07:00 CRITICAL #21 /opt/couchbase/bin/memcached() [0x400000+0x94c1bf]
|
2023-10-11T08:56:14.952752-07:00 CRITICAL #22 /opt/couchbase/bin/memcached() [0x400000+0x934ce7]
|
2023-10-11T08:56:14.952753-07:00 CRITICAL #23 /opt/couchbase/bin/memcached() [0x400000+0x94ed5a]
|
2023-10-11T08:56:14.952754-07:00 CRITICAL #24 /opt/couchbase/bin/memcached() [0x400000+0x7a2590]
|
2023-10-11T08:56:14.952754-07:00 CRITICAL #25 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8c50e9d000+0xe4aa3]
|
2023-10-11T08:56:14.952755-07:00 CRITICAL #26 /lib64/libpthread.so.0() [0x7f8c517f8000+0x7e25]
|
2023-10-11T08:56:14.952756-07:00 CRITICAL #27 /lib64/libc.so.6(clone+0x6d) [0x7f8c507d8000+0xf834d]
|
Had some trouble getting bt full as I saw some errors like below.
Dwarf Error: wrong version in compilation unit header (is 5, should be 2, 3, or 4) [in module /usr/lib/debug/opt/couchbase/bin/memcached-7.6.0-1627.x86_64.debug]
|
Tried few things from https://stackoverflow.com/questions/11671009/dwarf-error-wrong-version-in-compilation-unit-header-is-4-should-be-2 but did not help much.
Also saw some errors like "BFD: warning: /usr/lib/debug/opt/couchbase/lib/libcrypto.so.3-7.6.0-1627.x86_64.debug: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0000002".
bt of 3137bee1-4910-41c4-544e42b1-4ceb543a.dmp on 172.23.96.14
(gdb) bt
|
#0 0x00007f8c5080d1f7 in __GI_____strtod_l_internal (nptr=<optimized out>, endptr=<optimized out>, group=<optimized out>, loc=<optimized out>) at strtod_l.c:1290
|
#1 0x00007f8ae78f33d0 in ?? ()
|
#2 0x00007f8ae58bf800 in ?? ()
|
#3 0x00007f8b2eff29b0 in ?? ()
|
#4 0x0000000000000010 in ?? ()
|
#5 0x00007f8ae78f33d1 in ?? ()
|
#6 0x00000000007d51d6 in VBucket::updateFromManifest(cb::SharedLockRef<internal::VBucketStateLockTag>, Collections::Manifest const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc:1929
|
#7 0x000000000082a7f9 in operator()<Collections::Manifest> (manifest=..., manifest@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, __closure=<optimized out>)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/net/detail/Asm.h:479
|
#8 folly::SynchronizedBase<folly::Synchronized<Collections::Manifest, folly::SharedMutexImpl<false, void, std::atomic, folly::SharedMutexPolicyDefault> >, (folly::detail::SynchronizedMutexLevel)2>::withRLock<Collections::Manager::maybeUpdate(VBucketStateLockRef, VBucket&) const::<lambda(const auto:61&)> > (function=..., this=0x7f8ba81c0180, this@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
|
at /opt/gcc-13.2.0/include/c++/13.2.0/x86_64-pc-linux-gnu/bits/system_error:317
|
#9 Collections::Manager::maybeUpdate(cb::SharedLockRef<internal::VBucketStateLockTag>, VBucket&) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/collections/manager.cc:321
|
#10 0x00000000007fa363 in Warmup::populateVBucketMap(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/system/Function.h:1349
|
#11 0x00000000008036c6 in WarmupPopulateVBucketMap::run (this=0x7f8b50218c30) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/warmup.cc:347
|
#12 0x0000000000ba817a in GlobalTask::execute(std::basic_string_view<char, std::char_traits<char> >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:105
|
#13 0x0000000000ba61aa in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f8b2eff2d40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:314
|
#14 folly::detail::function::FunctionTraits<void ()>::callSmall<FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}>(folly::detail::function::Data&) (p=...)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/gsl-lite/include/gsl/char_traits.h:363
|
#15 0x0000000000ba94c3 in folly::detail::function::FunctionTraits<void ()>::operator()() (this=0x7f8b2eff2d40, this@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42
|
#16 CancellableCPUExecutor::add(GlobalTask*, folly::Function<void ()>)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42
|
#17 0x0000000000d4c1bf in folly::detail::function::FunctionTraits<void ()>::operator()() (this=0x7f8b2eff2ff0) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:98
|
#18 folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) ()
|
at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:98
|
#19 0x0000000000d34ce7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr<folly::ThreadPoolExecutor::Thread>) ()
|
at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:306
|
#20 0x0000000000d4ed5a in std::__invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__f=<optimized out>,
|
__t=<optimized out>, __f=<optimized out>, __t=<optimized out>) at /opt/gcc-10.2.0/include/c++/10.2.0/ext/atomicity.h:100
|
#21 std::__invoke<void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__fn=<optimized out>)
|
at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:95
|
#22 std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) (__args=..., this=<optimized out>) at /opt/gcc-10.2.0/include/c++/10.2.0/functional:416
|
#23 std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)>::operator()<, void>() (this=<optimized out>)
|
at /opt/gcc-10.2.0/include/c++/10.2.0/functional:499
|
#24 folly::detail::function::FunctionTraits<void ()>::callSmall<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:363
|
#25 0x0000000000ba2590 in folly::detail::function::FunctionTraits<void ()>::operator()() (this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49
|
#26 CBRegisteredThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}::operator()() (__closure=0x7f8c4f26d540) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49
|
#27 folly::detail::function::FunctionTraits<void ()>::callBig<CBRegisteredThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/gsl-lite/include/gsl/char_traits.h:377
|
#28 0x00007f8c50f81aa3 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-13.2.0/libstdc++-v3/src/c++11/thread.cc:104
|
#29 0x00007f8c517ffe25 in start_thread (arg=0x0) at pthread_create.c:282
|
#30 0x00007f8c508d034d in putspent (p=0x0, stream=0x7f8b2effd700) at putspent.c:64
|
#31 0x0000000000000000 in ?? ()
|
(gdb)
|
We did not see this on the run we had on 7.6.0-1601. cbcollect_info attached.
Attachments
Issue Links
- relates to
-
-
- Closed
-