Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46811

MagmaKVStore::makeCompactionContext vb:723 failed to read vbstate from disk. Status:NotFound: MagmaKVStore::readLocalDoc vb:723 key:_vbstate not found.

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • No
    • Magma-July-5-2021, Magma-July-12-2021, Magma-July-19-2021

    Description

      Test Job: http://qa.sc.couchbase.com/job/test_suite_executor-TAF/119452/

      Node 172.23.106.122

      cbcollect_info_ns_1@172.23.106.122_20210608-191135/95ac37ba-5db0-4e55-140fb383-34a48bd6.dmp
      2021-06-08T11:50:15.936370-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      2021-06-08T11:50:15.936469-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): MagmaKVStore::makeCompactionContext vb:723 failed to read vbstate from disk. Status:NotFound: MagmaKVStore::readLocalDoc vb:723 key:_vbstate not found.
      2021-06-08T11:50:16.172298-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      2021-06-08T11:50:16.178728-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      2021-06-08T11:50:16.517119-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      2021-06-08T11:50:16.566943-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      2021-06-08T11:50:24.912865-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      2021-06-08T11:50:25.164779-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-5270). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/51e17a53-1199-4f95-020eddba-1cdf597a.dmp before terminating.
      2021-06-08T11:50:25.164790-07:00 CRITICAL Stack backtrace of crashed thread:
      2021-06-08T11:50:25.165471-07:00 CRITICAL     #0  /opt/couchbase/bin/memcached() [0x400000+0x66ea23]
      2021-06-08T11:50:25.165496-07:00 CRITICAL     #1  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x68fa3a]
      2021-06-08T11:50:25.165552-07:00 CRITICAL     #2  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x68fd78]
      2021-06-08T11:50:25.165562-07:00 CRITICAL     #3  /lib64/libpthread.so.0() [0x7fe7959a5000+0xf5f0]
      2021-06-08T11:50:25.165587-07:00 CRITICAL     #4  /lib64/libc.so.6(gsignal+0x37) [0x7fe793413000+0x36337]
      2021-06-08T11:50:25.165613-07:00 CRITICAL     #5  /lib64/libc.so.6(abort+0x148) [0x7fe793413000+0x37a28]
      2021-06-08T11:50:25.165637-07:00 CRITICAL     #6  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe793cfb000+0x9963c]
      2021-06-08T11:50:25.165661-07:00 CRITICAL     #7  /opt/couchbase/bin/memcached() [0x400000+0x67c65f]
      2021-06-08T11:50:25.165696-07:00 CRITICAL     #8  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe793cfb000+0xa48f6]
      2021-06-08T11:50:25.165712-07:00 CRITICAL     #9  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe793cfb000+0xa4961]
      2021-06-08T11:50:25.165728-07:00 CRITICAL     #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe793cfb000+0xa4bf4]
      2021-06-08T11:50:25.165749-07:00 CRITICAL     #11 /opt/couchbase/bin/memcached() [0x400000+0xc6750]
      2021-06-08T11:50:25.165766-07:00 CRITICAL     #12 /opt/couchbase/bin/memcached() [0x400000+0x404993]
      2021-06-08T11:50:25.165782-07:00 CRITICAL     #13 /opt/couchbase/bin/memcached() [0x400000+0x2c8ea4]
      2021-06-08T11:50:25.165795-07:00 CRITICAL     #14 /opt/couchbase/bin/memcached() [0x400000+0x302d10]
      2021-06-08T11:50:25.165808-07:00 CRITICAL     #15 /opt/couchbase/bin/memcached() [0x400000+0x2b356e]
      2021-06-08T11:50:25.165828-07:00 CRITICAL     #16 /opt/couchbase/bin/memcached() [0x400000+0x3edc86]
      2021-06-08T11:50:25.165842-07:00 CRITICAL     #17 /opt/couchbase/bin/memcached() [0x400000+0x255ff8]
      2021-06-08T11:50:25.165981-07:00 CRITICAL     #18 /opt/couchbase/bin/memcached() [0x400000+0x26ee6d]
      2021-06-08T11:50:25.165997-07:00 CRITICAL     #19 /opt/couchbase/bin/memcached() [0x400000+0x1380cc]
      2021-06-08T11:50:25.166016-07:00 CRITICAL     #20 /opt/couchbase/bin/memcached() [0x400000+0x13865f]
      2021-06-08T11:50:25.166039-07:00 CRITICAL     #21 /opt/couchbase/bin/memcached() [0x400000+0x6b4cc3]
      2021-06-08T11:50:25.166065-07:00 CRITICAL     #22 /opt/couchbase/bin/memcached() [0x400000+0x6ab9e9]
      2021-06-08T11:50:25.166072-07:00 CRITICAL     #23 /lib64/libpthread.so.0() [0x7fe7959a5000+0x7e65]
      2021-06-08T11:50:25.166105-07:00 CRITICAL     #24 /lib64/libc.so.6(clone+0x6d) [0x7fe793413000+0xfe88d]
      

      172.23.106.122: Stack Trace of first crash - 95ac37ba-5db0-4e55-140fb383-34a48bd6.dmp

      Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
       #0  0x00007f673cfa0337 in raise () from /lib64/libc.so.6
       #0  0x00007f673cfa0337 in raise () from /lib64/libc.so.6
       No symbol table info available.
       #1  0x00007f673cfa1a28 in abort () from /lib64/libc.so.6
       No symbol table info available.
       #2  0x00007f673d8eb63c in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:95
               terminating = false
               t = <optimized out>
       #3  0x0000000000a7c65f in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:87
       No locals.
       #4  0x00007f673d8f68f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
       No locals.
       #5  0x00007f673d8f6961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
       No locals.
       #6  0x00007f673d8f6bf4 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7f63b4000940, tinfo=0xf676e0 <typeinfo for std::runtime_error>, dest=0x4435c0 <_ZNSt13runtime_errorD1Ev@plt>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95
               globals = <optimized out>
               header = 0x7f63b40008c0
       #7  0x00000000004c7874 in MagmaKVStore::makeCompactionContext (this=0x7f6725e91400, vbid=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:2623
               ss = {<std::basic_iostream<char, std::char_traits<char> >> = {<std::basic_istream<char, std::char_traits<char> >> = {<std::basic_ios<char, std::char_traits<char> >> = {<std::ios_base> = {_vptr.ios_base = 0xf67358 <vtable for std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >+24>, static boolalpha = std::_S_boolalpha, static dec = std::_S_dec, static fixed = std::_S_fixed, static hex = std::_S_hex, static internal = std::_S_internal, static left = std::_S_left, static oct = std::_S_oct, static right = std::_S_right, static scientific = std::_S_scientific, static showbase = std::_S_showbase, static showpoint = std::_S_showpoint, static showpos = std::_S_showpos, static skipws = std::_S_skipws, static unitbuf = std::_S_unitbuf, static uppercase = std::_S_uppercase, static adjustfield = std::_S_adjustfield, static basefield = std::_S_basefield, static floatfield = std::_S_floatfield, static badbit = std::_S_badbit, static eofbit = std::_S_eofbit, static failbit = std::_S_failbit, static goodbit = std::_S_goodbit, static app = std::_S_app, static ate = std::_S_ate, static binary = std::_S_bin, static in = std::_S_in, static out = std::_S_out, static trunc = std::_S_trunc, static beg = std::_S_beg, static cur = std::_S_cur, static end = std::_S_end, _M_precision = 0, _M_width = 16151424, _M_flags = 16150424, _M_exception = std::_S_goodbit, _M_streambuf_state = 635876224, _M_callbacks = 0x7f6725e6b380, _M_word_zero = {_M_pword = 0x7f6725e6b390, _M_iword = 140080994235264}, _M_local_word = {{_M_pword = 0x7f6725e6b417, _M_iword = 140080994235776}, {_M_pword = 0x7f673dc1ddc0 <(anonymous namespace)::c_locale_impl>, _M_iword = 24}, {_M_pword = 0x7f6725e6b380, _M_iword = 16}, {_M_pword = 0x200, _M_iword = 30744794945974132}, {_M_pword = 0xf673a8 <vtable for std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >+104>, _M_iword = 6}, {_M_pword = 0x0, _M_iword = 4098}, {_M_pword = 0x0, _M_iword = 0}, {_M_pword = 0x0, _M_iword = 0}}, _M_word_size = 0, _M_word = 0x0, _M_ios_locale = {static none = 0, static ctype = 1, static numeric = 2, static collate = 4, static time = 8, static monetary = 16, static messages = 32, static all = 63, _M_impl = 0x0, static _S_classic = 0x0, static _S_global = 0x0, static _S_categories = 0x3bd680, static _S_once = 0, static _S_twinned_facets = 0x7f673dc10f00 <std::locale::_S_twinned_facets>}}, _M_tie = 0x0, _M_fill = 0 '\000', _M_fill_init = false, _M_streambuf = 0x0, _M_ctype = 0x0, _M_num_put = 0x0, _M_num_get = 0x0}, _vptr.basic_istream = 0xf67358 <vtable for std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >+24>, _M_gcount = 0}, <std::basic_ostream<char, std::char_traits<char> >> = {_vptr.basic_ostream = 0xf67380 <vtable for std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >+64>}, <No data fields>}, _M_stringbuf = {<std::basic_streambuf<char, std::char_traits<char> >> = {_vptr.basic_streambuf = 0xf66f98 <vtable for std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >+16>, _M_in_beg = 0x7f6725e6b380 <Address 0x7f6725e6b380 out of bounds>, _M_in_cur = 0x7f6725e6b380 <Address 0x7f6725e6b380 out of bounds>, _M_in_end = 0x7f6725e6b390 <Address 0x7f6725e6b390 out of bounds>, _M_out_beg = 0x7f6725e6b380 <Address 0x7f6725e6b380 out of bounds>, _M_out_cur = 0x7f6725e6b417 <Address 0x7f6725e6b417 out of bounds>, _M_out_end = 0x7f6725e6b580 <Address 0x7f6725e6b580 out of bounds>, _M_buf_locale = {static none = 0, static ctype = 1, static numeric = 2, static collate = 4, static time = 8, static monetary = 16, static messages = 32, static all = 63, _M_impl = 0x7f673dc1ddc0 <(anonymous namespace)::c_locale_impl>, static _S_classic = 0x0, static _S_global = 0x0, static _S_categories = 0x3bd680, static _S_once = 0, static _S_twinned_facets = 0x7f673dc10f00 <std::locale::_S_twinned_facets>}}, _M_mode = 24, _M_string = {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x7f6725e6b380 <Address 0x7f6725e6b380 out of bounds>}, _M_string_length = 16, {_M_local_buf = "\000\002\000\000\000\000\000\000tore::m", _M_allocated_capacity = 512}}}}
               config = {purge_before_ts = 1622919015, purge_before_seq = 0, drop_deletes = false, retain_erroneous_tombstones = false}
               ctx = {<std::__shared_ptr<CompactionContext, (__gnu_cxx::_Lock_policy)2>> = {<std::__shared_ptr_access<CompactionContext, (__gnu_cxx::_Lock_policy)2, false, false>> = {<No data fields>}, _M_ptr = 0x7f66dc533f90, _M_refcount = {_M_pi = 0x7f66dc533f80}}, <No data fields>}
               status = <unknown type in /usr/lib/debug/opt/couchbase/bin/memcached.debug, CU 0x52d6910, DIE 0x543ee1a>
               dropped = <unknown type in /usr/lib/debug/opt/couchbase/bin/memcached.debug, CU 0x52d6910, DIE 0x543ee2a>
               readState = {status = {s = {_M_t = {<std::__uniq_ptr_impl<magma::Status::state, std::default_delete<magma::Status::state> >> = {_M_t = {<std::_Tuple_impl<0, magma::Status::state*, std::default_delete<magma::Status::state> >> = {<std::_Tuple_impl<1, std::default_delete<magma::Status::state> >> = {<std::_Head_base<1, std::default_delete<magma::Status::state>, true>> = {<std::default_delete<magma::Status::state>> = {<No data fields>}, <No data fields>}, <No data fields>}, <std::_Head_base<0, magma::Status::state*, false>> = {_M_head_impl = 0x7f672617f1a0}, <No data fields>}, <No data fields>}}, <No data fields>}}}, vbstate = {static CurrentVersion = 4, maxDeletedSeqno = {counter = {_M_elems = "\000\000\000\000\000"}}, highSeqno = 0, purgeSeqno = 0, lastSnapStart = 0, lastSnapEnd = 0, maxCas = 0, hlcCasEpochSeqno = -1, mightContainXattrs = false, supportsNamespaces = true, version = 4, persistedCompletedSeqno = 0, persistedPreparedSeqno = 0, highPreparedSeqno = 0, maxVisibleSeqno = 0, onDiskPrepares = 0, onDiskPrepareBytes = 0, checkpointType = Memory, transition = {failovers = {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x7f669bfec960 ""}, _M_string_length = 0, {_M_local_buf = '\000' <repeats 15 times>, _M_allocated_capacity = 0}}, replicationTopology = {m_type = nlohmann::detail::null, m_value = {object = 0x0, array = 0x0, string = 0x0, boolean = false, number_integer = 0, number_unsigned = 0, number_float = 0}}, state = vbucket_state_dead}}, kvstoreRev = 1}
       #8  0x0000000000859585 in MagmaKVStore::compactionCallBack(MagmaKVStore::MagmaCompactionCB&, magma::Slice const&, magma::Slice const&, magma::Slice const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:278
               vbid = {vbid = 723}
               itemString = @0x7f6634f07418: <error reading variable>
               seqno = <optimized out>
               exptime = <optimized out>
       #9  0x000000000094a6bd in magma::DocSeqGCContext::Transform (this=0x7f66d47c3140, key=..., value=..., buffer=..., outputValue=0x7f669bfecd50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/format.h:44
               drop = false
       #10 0x00000000009cf755 in magma::ItemGCFilter::Process(std::vector<std::pair<magma::Slice, magma::Slice>, std::allocator<std::pair<magma::Slice, magma::Slice> > >&, std::vector<std::pair<magma::Slice, magma::Slice>, std::allocator<std::pair<magma::Slice, magma::Slice> > >*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/mvcc.cc:106
               newValue = {data = 0x0, len = 0}
               deleted = <optimized out>
               oldValue = 0x0
               kv = {<std::__pair_base<magma::Slice, magma::Slice>> = {<No data fields>}, first = {data = 0x7f657f7212c0 <Address 0x7f657f7212c0 out of bounds>, len = 26}, second = {data = 0x7f657f7212da <Address 0x7f657f7212da out of bounds>, len = 51}}
               item = {key = {data = 0x7f657f7212c0 <Address 0x7f657f7212c0 out of bounds>, len = 17}, sn = -8861676691781582848, op = magma::Update}
               gcOffset = 1
               lastSn = <optimized out>
       #11 0x0000000000962bb2 in magma::LSMTree::mergeSSTables(std::vector<std::shared_ptr<magma::Table>, std::allocator<std::shared_ptr<magma::Table> > >&, magma::ItemGCFilter&, unsigned long, double, std::vector<std::shared_ptr<magma::Table>, std::allocator<std::shared_ptr<magma::Table> > >*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:649
               magma::rdBuf = <error reading variable magma::rdBuf (Cannot find thread-local variables on this target)>
               std::__ioinit = {static _S_refcount = 0, static _S_synced_with_stdio = true}
               fmt::v5::internal::basic_data<void>::POWERS_OF_10_32 = {0, 10, 100, 1000, 10000, 100000, 1000000, 10000000, 100000000, 1000000000}
               fmt::v5::internal::basic_data<void>::POWERS_OF_10_64 = {0, 10, 100, 1000, 10000, 100000, 1000000, 10000000, 100000000, 1000000000, 10000000000, 100000000000, 1000000000000, 10000000000000, 100000000000000, 1000000000000000, 10000000000000000, 100000000000000000, 1000000000000000000, 10000000000000000000}
               fmt::v5::internal::basic_data<void>::DIGITS = "00010203040506070809101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899"
               boost::exception_detail::exception_ptr_static_exception_object<boost::exception_detail::bad_exception_>::e = {ptr_ = {px = 0x0, pn = {pi_ = 0x0}}}
               boost::exception_detail::exception_ptr_static_exception_object<boost::exception_detail::bad_alloc_>::e = {ptr_ = {px = 0x0, pn = {pi_ = 0x0}}}
       #12 0x00000000009c919d in magma::LSMTree::compactLevelForInputs(int, int, std::unique_lock<std::mutex>&, std::vector<std::shared_ptr<magma::Table>, std::allocator<std::shared_ptr<magma::Table> > >&, std::vector<std::shared_ptr<magma::Table>, std::allocator<std::shared_ptr<magma::Table> > >&, bool, std::function<std::unique_ptr<magma::GCContext, std::default_delete<magma::GCContext> > (bool)>) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/compaction.cc:168
               status = {s = {_M_t = {<std::__uniq_ptr_impl<magma::Status::state, std::default_delete<magma::Status::state> >> = {_M_t = {<std::_Tuple_impl<0, magma::Status::state*, std::default_delete<magma::Status::state> >> = {<std::_Tuple_impl<1, std::default_delete<magma::Status::state> >> = {<std::_Head_base<1, std::default_delete<magma::Status::state>, true>> = {<std::default_delete<magma::Status::state>> = {<No data fields>}, <No data fields>}, <No data fields>}, <std::_Head_base<0, magma::Status::state*, false>> = {_M_head_impl = 0x0}, <No data fields>}, <No data fields>}}, <No data fields>}}}
               compactID = 2
               _c = {fn = {<std::_Maybe_unary_or_binary_function<void>> = {<No data fields>}, <std::_Function_base> = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x7f65532608e0, _M_const_object = 0x7f65532608e0, _M_function_pointer = 0x7f65532608e0, _M_member_pointer = (void (std::_Undefined_class::*)(std::_Undefined_class * const)) 0x7f65532608e0}, _M_pod_data = "\340\b&Se\177\000\000\000\000\000\000\000\000\000"}, _M_manager = 0x9c5290 <std::_Function_handler<void(), magma::LSMTree::compactLevelForInputs(magma::TreeLevel, magma::TreeLevel, std::unique_lock<std::mutex>&, magma::TableList&, magma::TableList&, bool, magma::GCContextBuilder)::<lambda()> >::_M_manager(std::_Any_data &, const std::_Any_data &, std::_Manager_operation)>}, _M_invoker = 0x9c5a60 <std::_Function_handler<void(), magma::LSMTree::compactLevelForInputs(magma::TreeLevel, magma::TreeLevel, std::unique_lock<std::mutex>&, magma::TableList&, magma::TableList&, bool, magma::GCContextBuilder)::<lambda()> >::_M_invoke(const std::_Any_data &)>}}
               srcLevelSize = 8192
               targetSize = 4194304
               levelSizeRatio = 0
               srcDataSize = <optimized out>
               dstDataSize = <optimized out>
               srcLogicalDataSz = 8192
               dstLogicalDataSz = 147456
               height = <optimized out>
               isHighestLevel = <optimized out>
               writeAmp = 18
               inputList = {<std::_Vector_base<std::shared_ptr<magma::Table>, std::allocator<std::shared_ptr<magma::Table> > >> = {_M_impl = {<std::allocator<std::shared_ptr<magma::Table> >> = {<__gnu_cxx::new_allocator<std::shared_ptr<magma::Table> >> = {<No data fields>}, <No data fields>}, <std::_Vector_base<std::shared_ptr<magma::Table>, std::allocator<std::shared_ptr<magma::Table> > >::_Vector_impl_data> = {_M_start = 0x7f6584bd2080, _M_finish = 0x7f6584bd20a0, _M_end_of_storage = 0x7f6584bd20a0}, <No data fields>}}, <No data fields>}
               outputList = {<std::_Vector_base<std::shared_ptr<magma::Table>, std::allocator<std::shared_ptr<magma::Table> > >> = {_M_impl = {<std::allocator<std::shared_ptr<magma::Table> >> = {<__gnu_cxx::new_allocator<std::shared_ptr<magma::Table> >> = {<No data fields>}, <No data fields>}, <std::_Vector_base<std::shared_ptr<magma::Table>, std::allocator<std::shared_ptr<magma::Table> > >::_Vector_impl_data> = {_M_start = 0x0, _M_finish = 0x0, _M_end_of_storage = 0x0}, <No data fields>}}, <No data fields>}
               needSSTablesMerging = true
               bloomFilterAccuracy = 0.94999998807907104
               levelSize = 2097152
               fileCompaction = true
               gcFilter = {_M_t = {<std::__uniq_ptr_impl<magma::ItemGCFilter, std::default_delete<magma::ItemGCFilter> >> = {_M_t = {<std::_Tuple_impl<0, magma::ItemGCFilter*, std::default_delete<magma::ItemGCFilter> >> = {<std::_Tuple_impl<1, std::default_delete<magma::ItemGCFilter> >> = {<std::_Head_base<1, std::default_delete<magma::ItemGCFilter>, true>> = {<std::default_delete<magma::ItemGCFilter>> = {<No data fields>}, <No data fields>}, <No data fields>}, <std::_Head_base<0, magma::ItemGCFilter*, false>> = {_M_head_impl = 0x7f668de99ac0}, <No data fields>}, <No data fields>}}, <No data fields>}}
               _ = {fn = {<std::_Maybe_unary_or_binary_function<void>> = {<No data fields>}, <std::_Function_base> = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x7f669bfed398, _M_const_object = 0x7f669bfed398, _M_function_pointer = 0x7f669bfed398, _M_member_pointer = (void (std::_Undefined_class::*)(std::_Undefined_class * const)) 0x7f669bfed398, this adjustment 140078680560592}, _M_pod_data = "\230\323\376\233f\177\000\000\320\323\376\233f\177\000"}, _M_manager = 0x9c4f80 <std::_Function_handler<void(), magma::LSMTree::compactLevelForInputs(magma::TreeLevel, magma::TreeLevel, std::unique_lock<std::mutex>&, magma::TableList&, magma::TableList&, bool, magma::GCContextBuilder)::<lambda()> >::_M_manager(std::_Any_data &, const std::_Any_data &, std::_Manager_operation)>}, _M_invoker = 0x9c5320 <std::_Function_handler<void (), magma::LSMTree::compactLevelForInputs(int, int, std::unique_lock<std::mutex>&, std::vector<std::shared_ptr<magma::Table>, std::allocator<std::shared_ptr<magma::Table> > >&, std::vector<std::shared_ptr<magma::Table>, std::allocator<std::shared_ptr<magma::Table> > >&, bool, std::function<std::unique_ptr<magma::GCContext, std::default_delete<magma::GCContext> > (bool)>)::{lambda()#2}>::_M_invoke(std::_Any_data const&)>}}
               resultState = <optimized out>
               outDataSize = <optimized out>
               newSnapshot = {<std::__shared_ptr<magma::TreeSnapshot, (__gnu_cxx::_Lock_policy)2>> = {<std::__shared_ptr_access<magma::TreeSnapshot, (__gnu_cxx::_Lock_policy)2, false, false>> = {<No data fields>}, _M_ptr = 0x7f668d60e210, _M_refcount = {_M_pi = 0x0}}, <No data fields>}
               dur = 391455696
               secs = <optimized out>
               MBps = <optimized out>
               rMBps = 6.920806377997199e-310
               wMBps = 5.0652662371470238e-317
               gcRatio = 819200
               newLevelRatio = 0
       #13 0x00000000009cb5b4 in magma::LSMTree::compactLevel(int, std::unique_lock<std::mutex>&, bool, std::function<std::unique_ptr<magma::GCContext, std::default_delete<magma::GCContext> > (bool)>) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/compaction.cc:73
               srcLevelList = {<std::_Vector_base<std::shared_ptr<magma::Table>, std::allocator<std::shared_ptr<magma::Table> > >> = {_M_impl = {<std::allocator<std::shared_ptr<magma::Table> >> = {<__gnu_cxx::new_allocator<std::shared_ptr<magma::Table> >> = {<No data fields>}, <No data fields>}, <std::_Vector_base<std::shared_ptr<magma::Table>, std::allocator<std::shared_ptr<magma::Table> > >::_Vector_impl_data> = {_M_start = 0x7f66a42dccc0, _M_finish = 0x7f66a42dccd0, _M_end_of_storage = 0x7f66a42dccd0}, <No data fields>}}, <No data fields>}
               dstLevelList = {<std::_Vector_base<std::shared_ptr<magma::Table>, std::allocator<std::shared_ptr<magma::Table> > >> = {_M_impl = {<std::allocator<std::shared_ptr<magma::Table> >> = {<__gnu_cxx::new_allocator<std::shared_ptr<magma::Table> >> = {<No data fields>}, <No data fields>}, <std::_Vector_base<std::shared_ptr<magma::Table>, std::allocator<std::shared_ptr<magma::Table> > >::_Vector_impl_data> = {_M_start = 0x7f6667ed7b80, _M_finish = 0x7f6667ed7b90, _M_end_of_storage = 0x7f6667ed7b90}, <No data fields>}}, <No data fields>}
               inputList = {<std::_Vector_base<std::shared_ptr<magma::Table>, std::allocator<std::shared_ptr<magma::Table> > >> = {_M_impl = {<std::allocator<std::shared_ptr<magma::Table> >> = {<__gnu_cxx::new_allocator<std::shared_ptr<magma::Table> >> = {<No data fields>}, <No data fields>}, <std::_Vector_base<std::shared_ptr<magma::Table>, std::allocator<std::shared_ptr<magma::Table> > >::_Vector_impl_data> = {_M_start = 0x0, _M_finish = 0x0, _M_end_of_storage = 0x0}, <No data fields>}}, <No data fields>}
               outputList = {<std::_Vector_base<std::shared_ptr<magma::Table>, std::allocator<std::shared_ptr<magma::Table> > >> = {_M_impl = {<std::allocator<std::shared_ptr<magma::Table> >> = {<__gnu_cxx::new_allocator<std::shared_ptr<magma::Table> >> = {<No data fields>}, <No data fields>}, <std::_Vector_base<std::shared_ptr<magma::Table>, std::allocator<std::shared_ptr<magma::Table> > >::_Vector_impl_data> = {_M_start = 0x0, _M_finish = 0x0, _M_end_of_storage = 0x0}, <No data fields>}}, <No data fields>}
               status = {s = {_M_t = {<std::__uniq_ptr_impl<magma::Status::state, std::default_delete<magma::Status::state> >> = {_M_t = {<std::_Tuple_impl<0, magma::Status::state*, std::default_delete<magma::Status::state> >> = {<std::_Tuple_impl<1, std::default_delete<magma::Status::state> >> = {<std::_Head_base<1, std::default_delete<magma::Status::state>, true>> = {<std::default_delete<magma::Status::state>> = {<No data fields>}, <No data fields>}, <No data fields>}, <std::_Head_base<0, magma::Status::state*, false>> = {_M_head_impl = 0x0}, <No data fields>}, <No data fields>}}, <No data fields>}}}
               dstLevel = 4
       #14 0x00000000009cb855 in magma::LSMTree::compact(std::unique_lock<std::mutex>&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/compaction.cc:339
               level = <optimized out>
               status = {s = {_M_t = {<std::__uniq_ptr_impl<magma::Status::state, std::default_delete<magma::Status::state> >> = {_M_t = {<std::_Tuple_impl<0, magma::Status::state*, std::default_delete<magma::Status::state> >> = {<std::_Tuple_impl<1, std::default_delete<magma::Status::state> >> = {<std::_Head_base<1, std::default_delete<magma::Status::state>, true>> = {<std::default_delete<magma::Status::state>> = {<No data fields>}, <No data fields>}, <No data fields>}, <std::_Head_base<0, magma::Status::state*, false>> = {_M_head_impl = 0x0}, <No data fields>}, <No data fields>}}, <No data fields>}}}
               ratio = @0x7f667b3778d0: <error reading variable>
               __for_range = @0x7f669bfed6f0: {<std::_Vector_base<std::pair<int, double>, std::allocator<std::pair<int, double> > >> = {_M_impl = {<std::allocator<std::pair<int, double> >> = {<__gnu_cxx::new_allocator<std::pair<int, double> >> = {<No data fields>}, <No data fields>}, <std::_Vector_base<std::pair<int, double>, std::allocator<std::pair<int, double> > >::_Vector_impl_data> = {_M_start = 0x7f667b3778d0, _M_finish = 0x7f667b3778e0, _M_end_of_storage = 0x7f667b3778e0}, <No data fields>}}, <No data fields>}
               compacted = false
               sizeRatios = {<std::_Vector_base<std::pair<int, double>, std::allocator<std::pair<int, double> > >> = {_M_impl = {<std::allocator<std::pair<int, double> >> = {<__gnu_cxx::new_allocator<std::pair<int, double> >> = {<No data fields>}, <No data fields>}, <std::_Vector_base<std::pair<int, double>, std::allocator<std::pair<int, double> > >::_Vector_impl_data> = {_M_start = 0x7f667b3778d0, _M_finish = 0x7f667b3778e0, _M_end_of_storage = 0x7f667b3778e0}, <No data fields>}}, <No data fields>}
       #15 0x0000000000957fc8 in std::_Function_handler<void (), magma::LSMTree::queueCompaction()::{lambda()#1}>::_M_invoke(std::_Any_data const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:297
               magma::rdBuf = <error reading variable magma::rdBuf (Cannot find thread-local variables on this target)>
               std::__ioinit = {static _S_refcount = 0, static _S_synced_with_stdio = true}
               fmt::v5::internal::basic_data<void>::POWERS_OF_10_32 = {0, 10, 100, 1000, 10000, 100000, 1000000, 10000000, 100000000, 1000000000}
               fmt::v5::internal::basic_data<void>::POWERS_OF_10_64 = {0, 10, 100, 1000, 10000, 100000, 1000000, 10000000, 100000000, 1000000000, 10000000000, 100000000000, 1000000000000, 10000000000000, 100000000000000, 1000000000000000, 10000000000000000, 100000000000000000, 1000000000000000000, 10000000000000000000}
               fmt::v5::internal::basic_data<void>::DIGITS = "00010203040506070809101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899"
               boost::exception_detail::exception_ptr_static_exception_object<boost::exception_detail::bad_exception_>::e = {ptr_ = {px = 0x0, pn = {pi_ = 0x0}}}
               boost::exception_detail::exception_ptr_static_exception_object<boost::exception_detail::bad_alloc_>::e = {ptr_ = {px = 0x0, pn = {pi_ = 0x0}}}
       #16 0x000000000099653d in operator() (this=0x7f66873113d8) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248
       No locals.
       #17 magma::TimedTask::Complete(bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:143
               wg = 0x7f6698682d40
               envGuard = {previous = 0x0}
               currTask = {<std::__shared_ptr<magma::Task, (__gnu_cxx::_Lock_policy)2>> = {<std::__shared_ptr_access<magma::Task, (__gnu_cxx::_Lock_policy)2, false, false>> = {<No data fields>}, _M_ptr = 0x7f66873113d0, _M_refcount = {_M_pi = 0x7f66873113c0}}, <No data fields>}
       #18 0x0000000000997d31 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160
               task = {<std::__shared_ptr<magma::TimedTask, (__gnu_cxx::_Lock_policy)2>> = {<std::__shared_ptr_access<magma::TimedTask, (__gnu_cxx::_Lock_policy)2, false, false>> = {<No data fields>}, _M_ptr = 0x7f668ecca660, _M_refcount = {_M_pi = 0x7f668ecca650}}, <No data fields>}
               worker = 0x7f672603f140
       #19 0x0000000000aab9e9 in run (this=0x7f6733cc16b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51
       No locals.
       #20 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64
       No locals.
       #21 0x00007f673f503e65 in start_thread () from /lib64/libpthread.so.0
       No symbol table info available.
       #22 0x00007f673d06888d in clone () from /lib64/libc.so.6
       No symbol table info available.
      

      QE Test:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.26553.ini bucket_storage=magma,rerun=false,bucket_eviction_policy=fullEviction,randomize_value=True,enable_dp=True,log_level=debug,infra_log_level=debug,upgrade_version=7.0.0-5270 -t magma.magma_disk_full.MagmaDiskFull.test_disk_full_reduce_replica,nodes_init=4,num_items=5000000,doc_size=2048,sdk_timeout=60,replicas=3,GROUP=P0'
      

      Attachments

        Issue Links

          Activity

            People

              sarath Sarath Lakshman
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty