Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-59607

Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure: 'EncodedSize() == buffer.length()' at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/format.cc:125

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • 7.6.0
    • 7.6.0
    • storage-engine
    • 7.6.0-1767

    Description

      Steps To Recreate:

      1. Create a 4 node cluster
      2. Create a magma bucket(replicas=1, bucket_ram_quota=256MB)
      3. Create 5000000 docs(doc size=2048 bytes)
      4. Start new doc ops(read:create)
      5. Whle doc ops are going on change permission of seqIndex to 000 and then after 5 seconds again change it back to 777(Test keeps doing for some time, chaning permission from 000 to 777 and then again back to 000.
      6. Observed Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure: 'EncodedSize() == buffer.length()' at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/format.cc:125

      Note: Above test is written for magma(a negative test) where we are simulating if we delay writing to seqIndex(by few seconds), how does system behaves). Since it a negative test , Not setting priority to CRITICAL

      Below Core was found on node172.23.107.158(29f78ac6-9621-4f4a-54192d95-48445238.dmp)

      bt-full:
      https://gist.github.com/ankushsharma29/4d089731f00b621f8ab1c8636da320dd

      BackTrace:

      (gdb) bt
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
      #1  0x00007f085d89a535 in __GI_abort () at abort.c:79
      #2  0x00007f085dc8a9ab in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-13.2.0/libstdc++-v3/libsupc++/vterminate.cc:95
      #3  0x0000000000c61dc6 in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:88
      #4  0x00007f085dc9a2fa in __cxxabiv1::__terminate (handler=<optimized out>) at /tmp/deploy/objdir/../gcc-13.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
      #5  0x00007f085dc9a365 in std::terminate () at /tmp/deploy/objdir/../gcc-13.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
      #6  0x00007f085dc9a5b7 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0xe4c9f8 <typeinfo for gsl::fail_fast>, dest=0x5eeeb0 <gsl::fail_fast::~fail_fast()>)
          at /tmp/deploy/objdir/../gcc-13.2.0/libstdc++-v3/libsupc++/eh_throw.cc:98
      #7  0x0000000000d45a35 in __cxxabiv1::__cxa_throw (thrownException=0x7f0748000ba0, type=0xe4c9f8 <typeinfo for gsl::fail_fast>, destructor=<optimized out>)
          at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/experimental/exception_tracer/ExceptionTracerLib.cpp:106
      #8  0x00000000004630ec in gsl::detail::fail_fast_throw (
          message=0xe17558 "GSL: Precondition failure: 'EncodedSize() == buffer.length()' at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/format.cc:125") at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/synchronization/Function.h:1769
      #9  0x00000000004fd902 in magma::ValueFormat::Encode (this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/gsl-lite/include/gsl/basic_string.h:368
      #10 0x0000000000b75692 in magma::Memtable::CopyBulkLoadIterators (this=0x7f08107d1310)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/gsl-lite/include/gsl/stl_vector.h:162
      #11 0x0000000000b1eca8 in magma::FlushWork::Execute (this=0x7f07764cf270) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/flush_work.cc:65
      #12 0x0000000000aafa23 in magma::KVStore::executeFlush (this=0x7f0828250810, kFlush=..., sFlush=..., lFlush=..., batch=...)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:476
      #13 0x0000000000abafbf in operator() (__closure=0x7f07764cf620) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/common.h:109
      #14 0x0000000000abc425 in std::__invoke_impl<magma::Status, magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, FlushMode, magma::BlockingMode, bulkLoadArgs)::<lambda()>&> (__f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/directory.h:288
      #15 std::__invoke_r<void, magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, FlushMode, magma::BlockingMode, bulkLoadArgs)::<lambda()>&> (
          __fn=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/gsl-lite/include/gsl/stl_iterator.h:111
      #16 std::_Function_handler<void(), magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, FlushMode, magma::BlockingMode, bulkLoadArgs)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/directory.h:290
      #17 0x0000000000aef9d8 in std::function<void ()>::operator()() const (this=0x7f08105647d0)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:158
      #18 magma::TimedTask::Complete (this=<optimized out>, skipExecution=false) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:158
      #19 0x0000000000af1494 in magma::TaskWorker::loop (w=0x7f08503cd340) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:180
      #20 0x0000000000cce82c in std::function<void ()>::operator()() const (this=0x7f085039e208)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/googletest.exploded/include/gtest/std_thread.h:247
      #21 operator() (__closure=0x7f085039e1e8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:117
      #22 std::__invoke_impl<void, create_thread(std::function<void()>, std::string)::<lambda()> > (__f=...)
          at /opt/gcc-13.2.0/include/c++/13.2.0/x86_64-pc-linux-gnu/bits/exception:61
      #23 std::__invoke<create_thread(std::function<void()>, std::string)::<lambda()> > (__fn=...)
          at /opt/gcc-13.2.0/include/c++/13.2.0/x86_64-pc-linux-gnu/bits/exception:96
      #24 std::thread::_Invoker<std::tuple<create_thread(std::function<void()>, std::string)::<lambda()> > >::_M_invoke<0> (this=0x7f085039e1e8)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/thread.cc:292
      #25 std::thread::_Invoker<std::tuple<create_thread(std::function<void()>, std::string)::<lambda()> > >::operator() (this=0x7f085039e1e8)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/thread.cc:299
      --Type <RET> for more, q to quit, c to continue without paging--
      #26 std::thread::_State_impl<std::thread::_Invoker<std::tuple<create_thread(std::function<void()>, std::string)::<lambda()> > > >::_M_run(void) (this=0x7f085039e1e0)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/thread.cc:244
      #27 0x00007f085dcc6aa3 in std::execute_native_thread_routine (__p=0x7f085039e1e0) at /tmp/deploy/objdir/../gcc-13.2.0/libstdc++-v3/src/c++11/thread.cc:104
      #28 0x00007f085e346fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
      

      QE-TEST:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.124863.ini bucket_storage=magma,rerun=false,bucket_eviction_policy=fullEviction,randomize_value=True,enable_dp=false,GROUP=P0;Majority,get-cbcollect-info=True,autoCompactionDefined=true,get-cbcollect-info=True,infra_log_level=info,log_level=info,bucket_storage=magma,upgrade_version=7.6.0-1767,sirius_url=http://172.23.120.103:4000 -t storage.magma.magma_disk_full.MagmaDiskFull.test_random_seqTree_chmod,nodes_init=4,num_items=5000000,doc_size=2048,sdk_timeout=60,replicas=1,durability=majority,vbuckets=16,GROUP=P0;Majority'
      

      Attachments

        For Gerrit Dashboard: MB-59607
        # Subject Branch Project Status CR V

        Activity

          People

            ankush.sharma Ankush Sharma
            ankush.sharma Ankush Sharma
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty