Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38017

Memcached is continuously crashing with sigfault.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • None
    • 6.0.2, 6.0.4
    • build, memcached
    • CentOS 7/Red Hat7, gcc 7.3.1, cmake3 3.14.6
    • Untriaged
    • Unknown

    Description

      I have found some problem like described in MB-27087 but with one difference - src/client/linux/handler/minidump_descriptor.cc already uses std::__cxx11::string while kv_engine/utilities/breakpad.cc uses std::basic_string.

      In my case memcached restarting continuously after crash with sigfault.

      Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.Program terminated with signal 11, Segmentation fault.
      #0  0x00007f8bbc0ad466 in __memcpy_ssse3_back () from /lib64/libc.so.6(gdb) bt#0  0x00007f8bbc0ad466 in __memcpy_ssse3_back () from /lib64/libc.so.6
      #1  0x00007f8bbf43321a in copy (__n=7795416, __s2=0x7f8bba0bd798 "/opt/couchbase/var/lib/couchbase/crash", __s1=<optimized out>) at /usr/local/include/c++/7.2.0/bits/char_traits.h:292
      #2  _S_copy (__n=7795416, __s=0x7f8bba0bd798 "/opt/couchbase/var/lib/couchbase/crash", __d=<optimized out>) at /usr/local/include/c++/7.2.0/bits/basic_string.h:324
      #3  _S_copy_chars (__k2=<optimized out>, __k1=0x7f8bba0bd798 "/opt/couchbase/var/lib/couchbase/crash", __p=<optimized out>) at /usr/local/include/c++/7.2.0/bits/basic_string.h:366
      #4  std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*> (this=this@entry=0x7f8bba0d0028, __beg=0x7f8bba0bd798 "/opt/couchbase/var/lib/couchbase/crash", __end=<optimized out>)    at /usr/local/include/c++/7.2.0/bits/basic_string.tcc:225
      #5  0x00007f8bbf433273 in _M_construct_aux<char*> (__end=<optimized out>, __beg=<optimized out>, this=0x7f8bba0d0028) at /usr/local/include/c++/7.2.0/bits/basic_string.h:220
      #6  _M_construct<char*> (__end=<optimized out>, __beg=<optimized out>, this=0x7f8bba0d0028) at /usr/local/include/c++/7.2.0/bits/basic_string.h:239
      #7  basic_string (__str=..., this=0x7f8bba0d0028) at /usr/local/include/c++/7.2.0/bits/basic_string.h:424
      #8  google_breakpad::MinidumpDescriptor::MinidumpDescriptor (this=0x7f8bba0d0020, descriptor=...) at src/client/linux/handler/minidump_descriptor.cc:48
      #9  0x00007f8bbf432b38 in google_breakpad::ExceptionHandler::ExceptionHandler (this=0x7f8bba0d0000, descriptor=..., filter=<optimized out>, callback=<optimized out>, callback_context=<optimized out>, install_handler=<optimized out>,    server_fd=-1) at src/client/linux/handler/exception_handler.cc:234
      #10 0x00007f8bbf41cb53 in create_handler (minidump_dir="/opt/couchbase/var/lib/couchbase/crash") at /home/chupokabra/couchbase/build/couchbase/kv_engine/utilities/breakpad.cc:115
      #11 0x00007f8bbf41cdc9 in cb::breakpad::initialize (settings=...) at /home/chupokabra/couchbase/build/couchbase/kv_engine/utilities/breakpad.cc:127
      #12 0x0000000000451bb0 in Settings::notify_changed (this=this@entry=0x194d140 <settings>, key="breakpad") at /home/chupokabra/couchbase/build/couchbase/kv_engine/daemon/settings.cc:1277
      #13 0x0000000000451eeb in setBreakpadSettings (breakpad=..., this=0x194d140 <settings>) at /home/chupokabra/couchbase/build/couchbase/kv_engine/daemon/settings.h:636
      #14 handle_breakpad (s=..., obj=<optimized out>) at /home/chupokabra/couchbase/build/couchbase/kv_engine/daemon/settings.cc:648Python Exception <class 'gdb.error'> There is no member or method named _M_head_impl.:
      #15 0x0000000000455dc8 in Settings::reconfigure (this=this@entry=0x194d140 <settings>, json=) at /home/chupokabra/couchbase/build/couchbase/kv_engine/daemon/settings.cc:717
      #16 0x0000000000473713 in load_config_file (file=0x7ffd061270d9 "/opt/couchbase/var/lib/couchbase/config/memcached.json", settings=...)    at /home/chupokabra/couchbase/build/couchbase/kv_engine/daemon/config_parse.cc:50
      #17 0x000000000046e8fe in parse_arguments (argc=argc@entry=3, argv=argv@entry=0x7ffd06125d68) at /home/chupokabra/couchbase/build/couchbase/kv_engine/daemon/cmdline.cc:213
      #18 0x00000000004351cf in memcached_main (argc=3, argv=0x7ffd06125d68) at /home/chupokabra/couchbase/build/couchbase/kv_engine/daemon/memcached.cc:2366
      #19 0x00007f8bbbf79545 in __libc_start_main () from /lib64/libc.so.6
      #20 0x000000000042c14e in _start ()
      (gdb) f 8
      #8  google_breakpad::MinidumpDescriptor::MinidumpDescriptor (this=0x7f6c8729a0c0, descriptor=...) at src/client/linux/handler/minidump_descriptor.cc:4848 src/client/linux/handler/minidump_descriptor.cc: No such file or directory.(gdb) ptype /mt descriptortype = const class google_breakpad::MinidumpDescriptor \{  public:    static const google_breakpad::MinidumpDescriptor::MicrodumpOnConsole kMicrodumpOnConsole;  private:    google_breakpad::MinidumpDescriptor::DumpMode mode_;    int fd_;    std::__cxx11::string directory_;    std::__cxx11::string path_;    const char *c_path_;    off_t size_limit_;    google_breakpad::MicrodumpExtraInfo microdump_extra_info_;} &
      (gdb) f 10
      #10 0x00007f6c8c6c6b53 in create_handler (minidump_dir=...) at /home/chupokabra/couchbase/build/couchbase/kv_engine/utilities/breakpad.cc:115115                                       /*server_fd*/ -1));(gdb) ptype /mt minidump_dirtype = const class std::basic_string<char, std::char_traits<char>, std::allocator<char> > [with _CharT = char, _Traits = std::char_traits<char>, _Alloc = std::allocator<char>] \{  public:    static size_type npos;  private:    std::basic_string<_CharT, _Traits, _Alloc>::_Alloc_hider _M_dataplus;} &
      

      Recompiled breakpad can be the potential problem...

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ceej Chris Hillery
              saltos Alexey Saltanov
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty