Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-21659

memcached crash due to uncaught exception in HMAC_SHA512

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 4.6.0
    • 4.5.0
    • memcached
    • None
    • Untriaged
    • Unknown

    Description

      Reported by a forum user - see https://forums.couchbase.com/t/service-memcached-exited-with-status-134-restarting/10759/

      Following exception was thrown:

      memcached<0.76.0>: terminate called after throwing an instance of 'std::runtime_error'
      memcached<0.76.0>:   what():  Couchbase::Crypto::HMAC(SHA512): HMAC failed
      

      I've extracted the backtrace from the minidump file:

      #0  0x00007fdf2d626418 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
      #1  0x00007fdf2d62801a in __GI_abort () at abort.c:89
      #2  0x00007fdf2dc5f84d in __gnu_cxx::__verbose_terminate_handler () at ../../../../src/libstdc++-v3/libsupc++/vterminate.cc:95
      #3  0x0000000000427fcf in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/watson-unix/memcached/utilities/terminate_handler.cc:63
      #4  0x00007fdf2dc5d6b6 in __cxxabiv1::__terminate (handler=<optimised out>) at ../../../../src/libstdc++-v3/libsupc++/eh_terminate.cc:47
      #5  0x00007fdf2dc5d701 in std::terminate () at ../../../../src/libstdc++-v3/libsupc++/eh_terminate.cc:57
      #6  0x00007fdf2dc5d919 in __cxxabiv1::__cxa_throw (obj=0x7fdf2c8b1ea0, tinfo=0x68b110 <typeinfo for std::runtime_error>, dest=0x40c420 <std::runtime_error::~runtime_error()@plt>)
          at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:87
      #7  0x00007fdf2f6dba51 in HMAC_SHA512 (data=<error reading variable: Cannot access memory at address 0x8>, key=std::vector of length 989083944, capacity -140595512139582 = {...})
          at /home/couchbase/jenkins/workspace/watson-unix/memcached/cbsasl/cbcrypto.cc:188
      #8  Couchbase::Crypto::HMAC (algorithm=<optimised out>, key=std::vector of length 0, capacity 0, data=std::vector of length 10, capacity 10 = {...})
          at /home/couchbase/jenkins/workspace/watson-unix/memcached/cbsasl/cbcrypto.cc:311
      #9  0x00007fdf2f6ea1dc in ScramShaBackend::getServerSignature (this=this@entry=0x7fdf0727b620) at /home/couchbase/jenkins/workspace/watson-unix/memcached/cbsasl/scram-sha/scram-sha.cc:227
      #10 0x00007fdf2f6ecd61 in ScramShaServerBackend::step (this=0x7fdf0727b620, conn=0x7fdef0c3c9a0, input=<optimised out>, inputlen=<optimised out>, output=0x7fdef01c6670, outputlen=0x7fdef01c6678)
          at /home/couchbase/jenkins/workspace/watson-unix/memcached/cbsasl/scram-sha/scram-sha.cc:463
      #11 0x000000000044c1a0 in StepSaslAuthTask::execute (this=0x7fdef01c6618) at /home/couchbase/jenkins/workspace/watson-unix/memcached/daemon/sasl_tasks.cc:49
      #12 0x00000000004394f2 in Executor::run (this=0x7fdf2c84f200) at /home/couchbase/jenkins/workspace/watson-unix/memcached/daemon/executor.cc:60
      #13 0x00007fdf2f0c1c8b in Couchbase::Thread::thread_entry (this=0x7fdf2c84f200) at /home/couchbase/jenkins/workspace/watson-unix/platform/src/thread.cc:46
      #14 0x00007fdf2f0bd84a in CouchbaseThread::run (this=0x7fdf2c8ce020) at /home/couchbase/jenkins/workspace/watson-unix/platform/src/cb_pthreads.cc:54
      #15 platform_thread_wrap (arg=0x7fdf2c8ce020) at /home/couchbase/jenkins/workspace/watson-unix/platform/src/cb_pthreads.cc:66
      #16 0x00007fdf2eea26fa in start_thread (arg=0x7fdf21df8700) at pthread_create.c:333
      #17 0x00007fdf2d6f7b5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
      

      From the memcached log the exception is thrown just after a Java SDK client HELLOs:

      2016-11-10T14:50:09.447388-08:00 NOTICE 67: HELO [couchbase-java-client/2.2.6 (git: 2.2.6, core: 1.2.7) (Linux/4.4.0-36-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_91-8u91-b14-3ubuntu1~16.04.1-b14)] TCP NODELAY [ 192.168.2.1
      11:46372 - 192.168.2.109:11210 ]
      2016-11-10T14:50:09.485189-08:00 NOTICE 69: HELO [couchbase-java-client/2.2.6 (git: 2.2.6, core: 1.2.7) (Linux/4.4.0-36-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_91-8u91-b14-3ubuntu1~16.04.1-b14)] TCP NODELAY [ 192.168.2.1
      11:46376 - 192.168.2.109:11210 ]
      2016-11-10T14:50:09.660840-08:00 WARNING Breakpad caught crash in memcached. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/1e4a92b9-e350-b8d2-67b966ba-1b89a355.dmp before terminating.
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            dhaikney David Haikney (Inactive)
            drigby Dave Rigby (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty