Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40609

memcached crashes after upgrading from 6.5.1 to cheshire-cat

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.0.0
    • Cheshire-Cat
    • memcached
    • None
    • 2 node Vagrant 6.5.1 centos7 cluster
    • Untriaged
    • Centos 64-bit
    • 1
    • Unknown

    Description

      After upgrading from 6.5.1 to cheshire-cat using a "vanilla" cheshire cat latest build from http://latestbuilds.service.couchbase.com like this:

      1. Spin up 2 Vagrant 6.5.1/centos7 nodes.
      2. Using the UI, create a cluster with each node running the data, query and index services.
      3. Load beer-sample
      4. ssh into each node and shutdown the couchbase-server service.
      5. install the cheshire cat rpm using
        sudo rpm -Uvh /vagrant/couchbase-server-enterprise-7.0.0-11048-centos7.x86_64.rpm
        sudo rpm -Uvh /vagrant/couchbase-server-enterprise-7.0.0-2679-centos7.x86_64.rpm

      I see messages like this in the memcached.log files:

      2020-07-23T04:46:17.540073+00:00 CRITICAL *** Fatal error encountered during exception handling ***
      2020-07-23T04:46:17.540073+00:00 CRITICAL Caught unhandled std::exception-derived exception. what(): `unsigned_leb128::decode invalid leb128 of size:0
      2020-07-23T04:46:17.596937+00:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-2673). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/b135dd27-4229-484d-bd66b3a2-02182c23.dmp before terminating.
      2020-07-23T04:46:17.596953+00:00 CRITICAL Stack backtrace of crashed thread:
      2020-07-23T04:46:17.597063+00:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x13dded]
      2020-07-23T04:46:17.597073+00:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x153e1a]
      2020-07-23T04:46:17.597079+00:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x154158]
      2020-07-23T04:46:17.597084+00:00 CRITICAL     /lib64/libpthread.so.0() [0x7f59ea6c3000+0xf130]
      2020-07-23T04:46:17.597102+00:00 CRITICAL     /lib64/libc.so.6(gsignal+0x39) [0x7f59ea302000+0x355c9]
      2020-07-23T04:46:17.597118+00:00 CRITICAL     /lib64/libc.so.6(abort+0x148) [0x7f59ea302000+0x36cd8]
      2020-07-23T04:46:17.597139+00:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7f59eadf8000+0x91195]
      2020-07-23T04:46:17.597146+00:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x14e882]
      2020-07-23T04:46:17.597149+00:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f59eadf8000+0x8ef86]
      2020-07-23T04:46:17.597149+00:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f59eadf8000+0x8efd1]
      2020-07-23T04:46:17.597149+00:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f59eadf8000+0x8f213]
      2020-07-23T04:46:17.597150+00:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f59ee561000+0x2621da]
      2020-07-23T04:46:17.597150+00:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f59ee561000+0x1fdd75]
      2020-07-23T04:46:17.597150+00:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f59ee561000+0x1cca6a]
      2020-07-23T04:46:17.597150+00:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f59ee561000+0x1d90b0]
      2020-07-23T04:46:17.597150+00:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f59ee561000+0x14e123]
      2020-07-23T04:46:17.597150+00:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f59ee561000+0x14430f]
      2020-07-23T04:46:17.597150+00:00 CRITICAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f59ed07d000+0x10777]
      2020-07-23T04:46:17.597150+00:00 CRITICAL     /lib64/libpthread.so.0() [0x7f59ea6c3000+0x7df3]
      2020-07-23T04:46:17.597150+00:00 CRITICAL     /lib64/libc.so.6(clone+0x6d) [0x7f59ea302000+0xf61ad]
      2020-07-23T04:46:17.598225+00:00 INFO ---------- Closing logfile
      

      I'm attaching a tgz of the logs.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sam.cramer Sam Cramer (Inactive)
              sam.cramer Sam Cramer (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty