Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-53173

magma crashed during upgrade from 7.1.0 to 7.2.0.

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • Elixir
    • Elixir
    • storage-engine
    • 7.2.0-1701

    Description

      Steps:

      1. create a 3 node 7.1.0 cluster and a magma bucket.
      2. start the data load.
      3. graceful failover one node
      4. upgrade it to 7.2.0 using rpm -U
      5. do delta recovery for the failed over node. magma crashed.

      memcahced logs

      Service 'memcached' exited with status 134. Restarting. Messages:
      #8 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f91e24b5000+0xa48f6]
      #9 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f91e24b5000+0xa4961]
      #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f91e24b5000+0xa4bf4]
      #11 /opt/couchbase/bin/../lib/libstdc++.so.6(_ZSt24__throw_invalid_argumentPKc+0x35) [0x7f91e24b5000+0x9be82]
      #12 /opt/couchbase/bin/memcached(_ZN9__gnu_cxx6__stoaIyycJiEEET0_PFT_PKT1_PPS3_DpT2_EPKcS5_PmS9_+0x8d) [0x400000+0x2bbb8d]
      #13 /opt/couchbase/bin/memcached() [0x400000+0x64371d]
      #14 /opt/couchbase/bin/memcached() [0x400000+0x643baf]
      #15 /opt/couchbase/bin/memcached() [0x400000+0x643f72]
      #16 /opt/couchbase/bin/memcached() [0x400000+0x65c705]
      #17 /opt/couchbase/bin/memcached() [0x400000+0x5e4068]
      #18 /opt/couchbase/bin/memcached() [0x400000+0x5e59e1]
      #19 /opt/couchbase/bin/memcached() [0x400000+0x77fa4c]
      #20 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f91e24b5000+0xcdd40]
      #21 /lib64/libpthread.so.0() [0x7f91e2f7d000+0x7ea5]
      #22 /lib64/libc.so.6(clone+0x6d) [0x7f91e1bcd000+0xfe8dd]
      

      CrashDump

      #0  0x00007fcd26e30387 in raise () from /lib64/libc.so.6
      #1  0x00007fcd26e31a78 in abort () from /lib64/libc.so.6
      #2  0x00007fcd277887f2 in __gnu_cxx::__verbose_terminate_handler() () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:50
      #3  0x0000000000b1dc26 in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:88
      #4  0x00007fcd277868f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
      #5  0x00007fcd27786961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
      #6  0x00007fcd27786bf4 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7fc8f4000940, tinfo=0x10fc908 <typeinfo for std::invalid_argument>, 
          dest=0x4469d0 <_ZNSt16invalid_argumentD1Ev@plt>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95
      #7  0x00007fcd2777de82 in std::__throw_invalid_argument (__s=__s@entry=0xc9e2cf "stoull")
          at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/functexcept.cc:74
      #8  0x00000000006bbb8d in __gnu_cxx::__stoa<unsigned long long, unsigned long long, char, int> (__convf=0x447980 <strtoull@plt>, 
          __name=__name@entry=0xc9e2cf "stoull", __str=0x7fcd1c281000 <Address 0x7fcd1c281000 out of bounds>, __idx=__idx@entry=0x0)
          at /opt/gcc-10.2.0/include/c++/10.2.0/ext/string_conversions.h:54
      #9  0x0000000000a4371d in stoull (__base=10, __idx=0x0, __str=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/basic_string.h:6567
      #10 magma::SSTable::readMetaBlock(magma::SSTableFooter const&, magma::ReaderStats&) ()
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:590
      #11 0x0000000000a43baf in magma::SSTable::init(magma::ReaderStats&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:472
      #12 0x0000000000a43f72 in magma::SSTable::Open(magma::ReaderStats&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:203
      #13 0x0000000000a5c705 in operator() (__closure=0x7fcd1c1624e0) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:779
      #14 __invoke_impl<void, magma::SSTableLoader::Load()::<lambda()>&> (__f=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:60
      #15 __invoke_r<void, magma::SSTableLoader::Load()::<lambda()>&> (__fn=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:110
      #16 std::_Function_handler<void (), magma::SSTableLoader::Load()::{lambda()#1}>::_M_invoke(std::_Any_data const&) ()
          at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:291
      #17 0x00000000009e4068 in operator() (this=0x7fcd1c15d940) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248
      #18 magma::TimedTask::Complete(bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:156
      #19 0x00000000009e59e1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:178
      #20 0x0000000000b7fa4c in operator() (this=0x7fcd1c1576e8) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248
      #21 operator() (__closure=0x7fcd1c1576c8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:117
      #22 __invoke_impl<void, create_thread(std::function<void()>, std::string)::<lambda()> > (
          __f=<unknown type in /usr/lib/debug/opt/couchbase/bin/memcached-7.2.0-1701.x86_64.debug, CU 0xb4077c3, DIE 0xb42576a>)
          at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:60
      #23 __invoke<create_thread(std::function<void()>, std::string)::<lambda()> > (
          __fn=<unknown type in /usr/lib/debug/opt/couchbase/bin/memcached-7.2.0-1701.x86_64.debug, CU 0xb4077c3, DIE 0xb425748>)
          at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:95
      #24 _M_invoke<0> (this=0x7fcd1c1576c8) at /opt/gcc-10.2.0/include/c++/10.2.0/thread:264
      #25 operator() (this=0x7fcd1c1576c8) at /opt/gcc-10.2.0/include/c++/10.2.0/thread:271
      #26 std::thread::_State_impl<std::thread::_Invoker<std::tuple<create_thread(std::function<void()>, std::string)::<lambda()> > > >::_M_run(void) (
          this=0x7fcd1c1576c0) at /opt/gcc-10.2.0/include/c++/10.2.0/thread:215
      #27 0x00007fcd277afd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80
      #28 0x00007fcd281b1ea5 in start_thread () from /lib64/libpthread.so.0
      #29 0x00007fcd26ef88dd in clone () from /lib64/libc.so.6
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Build couchbase-server-7.2.0-1733 contains magma commit f9e8c09 with commit message:
          MB-53173 magma: Remove keys added to sstable meta for compression

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-1733 contains magma commit f9e8c09 with commit message: MB-53173 magma: Remove keys added to sstable meta for compression

          Build couchbase-server-8.0.0-1069 contains magma commit f9e8c09 with commit message:
          MB-53173 magma: Remove keys added to sstable meta for compression

          build-team Couchbase Build Team added a comment - Build couchbase-server-8.0.0-1069 contains magma commit f9e8c09 with commit message: MB-53173 magma: Remove keys added to sstable meta for compression
          apaar.gupta Apaar Gupta added a comment -

          Merged the fix, Ankush Sharma could you please verify?

          apaar.gupta Apaar Gupta added a comment - Merged the fix, Ankush Sharma could you please verify?

          Tried few iterations of test on build 7.1.0-1735. Not seeing this anymore.

          ankush.sharma Ankush Sharma added a comment - Tried few iterations of test on build 7.1.0-1735. Not seeing this anymore.

          Build couchbase-server-7.2.0-1785 contains magma commit bc21a8b with commit message:
          MB-53173 magma: Add upgrade tests

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-1785 contains magma commit bc21a8b with commit message: MB-53173 magma: Add upgrade tests

          Build couchbase-server-8.0.0-1076 contains magma commit bc21a8b with commit message:
          MB-53173 magma: Add upgrade tests

          build-team Couchbase Build Team added a comment - Build couchbase-server-8.0.0-1076 contains magma commit bc21a8b with commit message: MB-53173 magma: Add upgrade tests

          People

            ankush.sharma Ankush Sharma
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty