Details
-
Bug
-
Resolution: Cannot Reproduce
-
Critical
-
7.6.0
-
7.6.0-1767
-
Untriaged
-
Linux x86_64
-
0
-
Unknown
Description
Steps To Recreate:
- Create a 4 node cluster
- Create a magma bucket(replicas=1, bucket_ram_quota=256MB)
- Create 5000000 docs(doc size=2048 bytes)
- Start new doc ops(read:create)
- Whle doc ops are going on change permission of seqIndex to 000 and then after 5 seconds again change it back to 777(Test keeps doing for some time, chaning permission from 000 to 777 and then again back to 000.
- Observed Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure: 'EncodedSize() == buffer.length()' at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/format.cc:125
Note: Above test is written for magma(a negative test) where we are simulating if we delay writing to seqIndex(by few seconds), how does system behaves). Since it a negative test , Not setting priority to CRITICAL
Below Core was found on node172.23.107.158(29f78ac6-9621-4f4a-54192d95-48445238.dmp)
bt-full:
https://gist.github.com/ankushsharma29/4d089731f00b621f8ab1c8636da320dd
BackTrace:
(gdb) bt
|
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
|
#1 0x00007f085d89a535 in __GI_abort () at abort.c:79
|
#2 0x00007f085dc8a9ab in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-13.2.0/libstdc++-v3/libsupc++/vterminate.cc:95
|
#3 0x0000000000c61dc6 in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:88
|
#4 0x00007f085dc9a2fa in __cxxabiv1::__terminate (handler=<optimized out>) at /tmp/deploy/objdir/../gcc-13.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
|
#5 0x00007f085dc9a365 in std::terminate () at /tmp/deploy/objdir/../gcc-13.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
|
#6 0x00007f085dc9a5b7 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0xe4c9f8 <typeinfo for gsl::fail_fast>, dest=0x5eeeb0 <gsl::fail_fast::~fail_fast()>)
|
at /tmp/deploy/objdir/../gcc-13.2.0/libstdc++-v3/libsupc++/eh_throw.cc:98
|
#7 0x0000000000d45a35 in __cxxabiv1::__cxa_throw (thrownException=0x7f0748000ba0, type=0xe4c9f8 <typeinfo for gsl::fail_fast>, destructor=<optimized out>)
|
at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/experimental/exception_tracer/ExceptionTracerLib.cpp:106
|
#8 0x00000000004630ec in gsl::detail::fail_fast_throw (
|
message=0xe17558 "GSL: Precondition failure: 'EncodedSize() == buffer.length()' at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/format.cc:125") at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/synchronization/Function.h:1769
|
#9 0x00000000004fd902 in magma::ValueFormat::Encode (this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/gsl-lite/include/gsl/basic_string.h:368
|
#10 0x0000000000b75692 in magma::Memtable::CopyBulkLoadIterators (this=0x7f08107d1310)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/gsl-lite/include/gsl/stl_vector.h:162
|
#11 0x0000000000b1eca8 in magma::FlushWork::Execute (this=0x7f07764cf270) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/flush_work.cc:65
|
#12 0x0000000000aafa23 in magma::KVStore::executeFlush (this=0x7f0828250810, kFlush=..., sFlush=..., lFlush=..., batch=...)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:476
|
#13 0x0000000000abafbf in operator() (__closure=0x7f07764cf620) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/common.h:109
|
#14 0x0000000000abc425 in std::__invoke_impl<magma::Status, magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, FlushMode, magma::BlockingMode, bulkLoadArgs)::<lambda()>&> (__f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/directory.h:288
|
#15 std::__invoke_r<void, magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, FlushMode, magma::BlockingMode, bulkLoadArgs)::<lambda()>&> (
|
__fn=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/gsl-lite/include/gsl/stl_iterator.h:111
|
#16 std::_Function_handler<void(), magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, FlushMode, magma::BlockingMode, bulkLoadArgs)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/directory.h:290
|
#17 0x0000000000aef9d8 in std::function<void ()>::operator()() const (this=0x7f08105647d0)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:158
|
#18 magma::TimedTask::Complete (this=<optimized out>, skipExecution=false) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:158
|
#19 0x0000000000af1494 in magma::TaskWorker::loop (w=0x7f08503cd340) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:180
|
#20 0x0000000000cce82c in std::function<void ()>::operator()() const (this=0x7f085039e208)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/googletest.exploded/include/gtest/std_thread.h:247
|
#21 operator() (__closure=0x7f085039e1e8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:117
|
#22 std::__invoke_impl<void, create_thread(std::function<void()>, std::string)::<lambda()> > (__f=...)
|
at /opt/gcc-13.2.0/include/c++/13.2.0/x86_64-pc-linux-gnu/bits/exception:61
|
#23 std::__invoke<create_thread(std::function<void()>, std::string)::<lambda()> > (__fn=...)
|
at /opt/gcc-13.2.0/include/c++/13.2.0/x86_64-pc-linux-gnu/bits/exception:96
|
#24 std::thread::_Invoker<std::tuple<create_thread(std::function<void()>, std::string)::<lambda()> > >::_M_invoke<0> (this=0x7f085039e1e8)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/thread.cc:292
|
#25 std::thread::_Invoker<std::tuple<create_thread(std::function<void()>, std::string)::<lambda()> > >::operator() (this=0x7f085039e1e8)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/thread.cc:299
|
--Type <RET> for more, q to quit, c to continue without paging--
|
#26 std::thread::_State_impl<std::thread::_Invoker<std::tuple<create_thread(std::function<void()>, std::string)::<lambda()> > > >::_M_run(void) (this=0x7f085039e1e0)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/thread.cc:244
|
#27 0x00007f085dcc6aa3 in std::execute_native_thread_routine (__p=0x7f085039e1e0) at /tmp/deploy/objdir/../gcc-13.2.0/libstdc++-v3/src/c++11/thread.cc:104
|
#28 0x00007f085e346fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
|
QE-TEST:
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.124863.ini bucket_storage=magma,rerun=false,bucket_eviction_policy=fullEviction,randomize_value=True,enable_dp=false,GROUP=P0;Majority,get-cbcollect-info=True,autoCompactionDefined=true,get-cbcollect-info=True,infra_log_level=info,log_level=info,bucket_storage=magma,upgrade_version=7.6.0-1767,sirius_url=http://172.23.120.103:4000 -t storage.magma.magma_disk_full.MagmaDiskFull.test_random_seqTree_chmod,nodes_init=4,num_items=5000000,doc_size=2048,sdk_timeout=60,replicas=1,durability=majority,vbuckets=16,GROUP=P0;Majority'
|
Attachments
For Gerrit Dashboard: MB-59607 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
201820,2 | MB-59607 magma: Add additonal info in case on ValueFormat size mismatch | master | magma | Status: MERGED | +2 | +1 |