Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Elixir
-
7.2.0-1701
-
Untriaged
-
-
1
-
Unknown
Description
Steps:
- create a 3 node 7.1.0 cluster and a magma bucket.
- start the data load.
- graceful failover one node
- upgrade it to 7.2.0 using rpm -U
- do delta recovery for the failed over node. magma crashed.
memcahced logs |
Service 'memcached' exited with status 134. Restarting. Messages:
|
#8 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f91e24b5000+0xa48f6]
|
#9 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f91e24b5000+0xa4961]
|
#10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f91e24b5000+0xa4bf4]
|
#11 /opt/couchbase/bin/../lib/libstdc++.so.6(_ZSt24__throw_invalid_argumentPKc+0x35) [0x7f91e24b5000+0x9be82]
|
#12 /opt/couchbase/bin/memcached(_ZN9__gnu_cxx6__stoaIyycJiEEET0_PFT_PKT1_PPS3_DpT2_EPKcS5_PmS9_+0x8d) [0x400000+0x2bbb8d]
|
#13 /opt/couchbase/bin/memcached() [0x400000+0x64371d]
|
#14 /opt/couchbase/bin/memcached() [0x400000+0x643baf]
|
#15 /opt/couchbase/bin/memcached() [0x400000+0x643f72]
|
#16 /opt/couchbase/bin/memcached() [0x400000+0x65c705]
|
#17 /opt/couchbase/bin/memcached() [0x400000+0x5e4068]
|
#18 /opt/couchbase/bin/memcached() [0x400000+0x5e59e1]
|
#19 /opt/couchbase/bin/memcached() [0x400000+0x77fa4c]
|
#20 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f91e24b5000+0xcdd40]
|
#21 /lib64/libpthread.so.0() [0x7f91e2f7d000+0x7ea5]
|
#22 /lib64/libc.so.6(clone+0x6d) [0x7f91e1bcd000+0xfe8dd]
|
CrashDump |
#0 0x00007fcd26e30387 in raise () from /lib64/libc.so.6
|
#1 0x00007fcd26e31a78 in abort () from /lib64/libc.so.6
|
#2 0x00007fcd277887f2 in __gnu_cxx::__verbose_terminate_handler() () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:50
|
#3 0x0000000000b1dc26 in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:88
|
#4 0x00007fcd277868f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
|
#5 0x00007fcd27786961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
|
#6 0x00007fcd27786bf4 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7fc8f4000940, tinfo=0x10fc908 <typeinfo for std::invalid_argument>,
|
dest=0x4469d0 <_ZNSt16invalid_argumentD1Ev@plt>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95
|
#7 0x00007fcd2777de82 in std::__throw_invalid_argument (__s=__s@entry=0xc9e2cf "stoull")
|
at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/functexcept.cc:74
|
#8 0x00000000006bbb8d in __gnu_cxx::__stoa<unsigned long long, unsigned long long, char, int> (__convf=0x447980 <strtoull@plt>,
|
__name=__name@entry=0xc9e2cf "stoull", __str=0x7fcd1c281000 <Address 0x7fcd1c281000 out of bounds>, __idx=__idx@entry=0x0)
|
at /opt/gcc-10.2.0/include/c++/10.2.0/ext/string_conversions.h:54
|
#9 0x0000000000a4371d in stoull (__base=10, __idx=0x0, __str=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/basic_string.h:6567
|
#10 magma::SSTable::readMetaBlock(magma::SSTableFooter const&, magma::ReaderStats&) ()
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:590
|
#11 0x0000000000a43baf in magma::SSTable::init(magma::ReaderStats&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:472
|
#12 0x0000000000a43f72 in magma::SSTable::Open(magma::ReaderStats&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:203
|
#13 0x0000000000a5c705 in operator() (__closure=0x7fcd1c1624e0) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:779
|
#14 __invoke_impl<void, magma::SSTableLoader::Load()::<lambda()>&> (__f=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:60
|
#15 __invoke_r<void, magma::SSTableLoader::Load()::<lambda()>&> (__fn=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:110
|
#16 std::_Function_handler<void (), magma::SSTableLoader::Load()::{lambda()#1}>::_M_invoke(std::_Any_data const&) ()
|
at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:291
|
#17 0x00000000009e4068 in operator() (this=0x7fcd1c15d940) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248
|
#18 magma::TimedTask::Complete(bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:156
|
#19 0x00000000009e59e1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:178
|
#20 0x0000000000b7fa4c in operator() (this=0x7fcd1c1576e8) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248
|
#21 operator() (__closure=0x7fcd1c1576c8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:117
|
#22 __invoke_impl<void, create_thread(std::function<void()>, std::string)::<lambda()> > (
|
__f=<unknown type in /usr/lib/debug/opt/couchbase/bin/memcached-7.2.0-1701.x86_64.debug, CU 0xb4077c3, DIE 0xb42576a>)
|
at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:60
|
#23 __invoke<create_thread(std::function<void()>, std::string)::<lambda()> > (
|
__fn=<unknown type in /usr/lib/debug/opt/couchbase/bin/memcached-7.2.0-1701.x86_64.debug, CU 0xb4077c3, DIE 0xb425748>)
|
at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:95
|
#24 _M_invoke<0> (this=0x7fcd1c1576c8) at /opt/gcc-10.2.0/include/c++/10.2.0/thread:264
|
#25 operator() (this=0x7fcd1c1576c8) at /opt/gcc-10.2.0/include/c++/10.2.0/thread:271
|
#26 std::thread::_State_impl<std::thread::_Invoker<std::tuple<create_thread(std::function<void()>, std::string)::<lambda()> > > >::_M_run(void) (
|
this=0x7fcd1c1576c0) at /opt/gcc-10.2.0/include/c++/10.2.0/thread:215
|
#27 0x00007fcd277afd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80
|
#28 0x00007fcd281b1ea5 in start_thread () from /lib64/libpthread.so.0
|
#29 0x00007fcd26ef88dd in clone () from /lib64/libc.so.6
|
Attachments
For Gerrit Dashboard: MB-53173 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
178178,5 | MB-53173 magma: Remove keys added to sstable meta for compression | master | magma | Status: MERGED | +2 | +1 |
178287,9 | MB-53173 magma: Add upgrade tests | master | magma | Status: MERGED | +2 | +1 |
186554,3 | [BP] MB-55532 magma: Add upgrade tests | neo | magma | Status: MERGED | +2 | +1 |