Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-41747

Couchstore: Memcached crashed in Warmup::loadKVPairsforShard(unsigned short) () from /opt/couchbase/bin/../lib/libep.so

    XMLWordPrintable

Details

    Description

      On 172.23.97.40:

      gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/21dab8ad-cce3-49a2-b85d1689-260d42f1.core -ex bt full -ex quit
      		 
      Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
      		 
       #0  0x00007f5989709337 in raise () from /lib64/libc.so.6
      		 
       #0  0x00007f5989709337 in raise () from /lib64/libc.so.6
      		 
       #1  0x00007f598970aa28 in abort () from /lib64/libc.so.6
      		 
       #2  0x00007f598a267195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6
      		 
       #3  0x00000000005a4f22 in backtrace_terminate_handler() ()
      		 
       #4  0x00007f598a264f86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6
      		 
       #5  0x00007f598a264fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6
      		 
       #6  0x00007f598a265213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6
      		 
       #7  0x00007f598a28d867 in std::__throw_out_of_range(char const*) () from /opt/couchbase/bin/../lib/libstdc++.so.6
      		 
       #8  0x00007f598d9830f7 in from_json(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer> const&, vbucket_state&) () from /opt/couchbase/bin/../lib/libep.so
      		 
       #9  0x00007f598d9c8a72 in CouchKVStore::readVBState(_db*, Vbid) () from /opt/couchbase/bin/../lib/libep.so
      		 
       #10 0x00007f598d9cb631 in CouchKVStore::initBySeqnoScanContext(std::unique_ptr<StatusCallback<GetValue>, std::default_delete<StatusCallback<GetValue> > >, std::unique_ptr<StatusCallback<CacheLookup>, std::default_delete<StatusCallback<CacheLookup> > >, Vbid, unsigned long, DocumentFilter, ValueFilter, SnapshotSource) () from /opt/couchbase/bin/../lib/libep.so
      		 
       #11 0x00007f598d990b28 in Warmup::loadKVPairsforShard(unsigned short) () from /opt/couchbase/bin/../lib/libep.so
      		 
       #12 0x00007f598d99c5f0 in WarmupLoadingKVPairs::run() () from /opt/couchbase/bin/../lib/libep.so
      		 
       #13 0x00007f598d910623 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so
      		 
       #14 0x00007f598d8140af in CB3ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so
      		 
       #15 0x00007f598c269947 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0
      		 
       #16 0x00007f5989aa8e65 in start_thread () from /lib64/libpthread.so.0
      		 
       #17 0x00007f59897d188d in clone () from /lib64/libc.so.6

      Memcached lohs from .40:

      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:21.433012-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:21.433053-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): stoll
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.710347-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-3236). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/3e7517e0-3a48-4de5-7dda39b2-66477a9d.dmp before terminating.
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.710416-07:00 CRITICAL Stack backtrace of crashed thread:
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.710651-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x1941dd]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.710683-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x1aa55a]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.710708-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x1aa898]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.710724-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7f8b44fab000+0xf5f0]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.710772-07:00 CRITICAL     /lib64/libc.so.6(gsignal+0x37) [0x7f8b44bdd000+0x36337]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.710815-07:00 CRITICAL     /lib64/libc.so.6(abort+0x148) [0x7f8b44bdd000+0x37a28]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.710862-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7f8b456e0000+0x91195]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.710890-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x1a4f22]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.710926-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8b456e0000+0x8ef86]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.710961-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8b456e0000+0x8efd1]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.710994-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8b456e0000+0x8f213]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.711049-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6(_ZSt20__throw_out_of_rangePKc+0x37) [0x7f8b456e0000+0xb7867]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.711073-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8b48c9b000+0x1f20f7]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.711094-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8b48c9b000+0x237a72]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.711111-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8b48c9b000+0x23dad7]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.711125-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8b48c9b000+0x23e042]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.711137-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8b48c9b000+0x23e185]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.711155-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8b48c9b000+0x191c95]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.711181-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8b48c9b000+0x1ad1fa]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.711196-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8b48c9b000+0x1f639f]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.711210-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8b48c9b000+0x19db4e]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.711228-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8b48c9b000+0x11bd06]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.711244-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8b48c9b000+0x135cc9]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.711258-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8b48c9b000+0x14b03d]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.711283-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x4469c]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.711304-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x44c59]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.711321-07:00 CRITICAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0(_ZN9Couchbase6Thread12thread_entryEv+0xf) [0x7f8b47763000+0x1b5df]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.711333-07:00 CRITICAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f8b47763000+0x10947]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.711348-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7f8b44fab000+0x7e65]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000508.txt:2020-09-29T05:56:22.711409-07:00 CRITICAL     /lib64/libc.so.6(clone+0x6d) [0x7f8b44bdd000+0xfe88d]

      QE test:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/dp_magma_job.ini -t magma.magma_crash_recovery.MagmaCrashTests.test_crash_during_ops,num_items=1000000,infra_log_level=debug,log_level=debug,rerun=False,skip_cleanup=true,rerun=False,doc_size=1024,randomize_value=True,nodes_init=10,num_crashes=20,sdk_timeout=60,bucket_storage=couchstore,replicas=1,vbuckets=1024,graceful=False,doc_ops=create:update:delete:expiry,wait_warmup=True,maxttl=10,get-cbcollect-info=True,multiplier=200,process_concurrency=5,randomize_doc_size=False,stop_server_on_crash=false,durability=PERSIST_TO_MAJORITY -m rest'

      Attachments

        Issue Links

          has to be done before

          Improvement - An improvement or enhancement to an existing feature or task. MB-38428 Reduce persistMajority tail latency - optimize couchstore compaction

          • Major - Major loss of function.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-41736 [System Test] : Memcached crash seen during rebalance of kv nodes

          • Test Blocker - QE-related tests can't proceed
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MB-41612 [Jepsen] Couchbase crash due to stoll exception when parsing vbstate json

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MB-41619 [Collections] - Minidumps seen in system test during rebalance in + collections data load + bucket level durability set in

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-41640 Memcached crashed in CouchKVStore::readVBState(_db*, Vbid) () from /opt/couchbase/bin/../lib/libep.so

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-41669 CouchStore - Memcached crashed in CouchKVStore::readVBState(_db*, Vbid) () -> CouchKVStore::CouchKVStore()

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty