Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-23525

[5.0.0 Build Sanity] Memcached crashing on Mac with "system_error: mutex lock failed: Invalid argument"

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Test Blocker
    • 5.0.0
    • 5.0.0
    • memcached, phosphor
    • MacOS X Sierra, MacOS X El Capitan
    • Untriaged
    • MacOSX 64-bit
    • Yes

    Description

      Build : 5.0.0-2408

      On Mac OS X Sierra and El Capitan, memcached is crashing. The bucket always remain in Warmup state. Following error is seen in the babysitter log :

      memcached<0.84.0>: 2017-03-23T08:40:03.122346-07:00 WARNING Failed to set the number of file descriptors to 66048 due to system resource restrictions. This may cause the system to misbehave once you reach a high connection count as the system won't be able open new files on the system. The maximum number of file descriptors is currently set to 10240. The system is configured to allow 65000 number of client connections, and in addition to that the overhead of the worker threads is 24. Finally the backed database needs to open files to persist data.
      memcached<0.84.0>: 2017-03-23T08:40:03.139711-07:00 FATAL Failed to create listening socket
      memcached<0.84.0>: *** Fatal error encountered during exception handling ***
      memcached<0.84.0>: Call stack:
      memcached<0.84.0>: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/memcached(_ZL27backtrace_terminate_handlerv+0x89) [0x10ea35349]
      memcached<0.84.0>: /usr/lib/libc++abi.dylib(ZSt11_terminatePFvvE+0x8) [0x7fff9cd8fd69]
      memcached<0.84.0>: /usr/lib/libc++abi.dylib(ZN10_cxxabiv1L22exception_cleanup_funcE19_Unwind_Reason_CodeP17_Unwind_Exception+0) [0x7fff9cd8f7de]
      memcached<0.84.0>: /usr/lib/libc++.1.dylib(ZNSt3120_throw_system_errorEiPKc+0x4d) [0x7fff9cd5d441]
      memcached<0.84.0>: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/lib/libphosphor.dylib(_ZN8phosphor8TraceLog16deregisterThreadEv+0x21) [0x10fe17791]
      memcached<0.84.0>: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/lib/libplatform_so.0.1.0.dylib(_ZL20platform_thread_wrapPv+0xf) [0x10fdee68f]
      memcached<0.84.0>: /usr/lib/system/libsystem_pthread.dylib(_pthread_body+0xb4) [0x7fff9e38dabb]
      memcached<0.84.0>: /usr/lib/system/libsystem_pthread.dylib(_pthread_body+0) [0x7fff9e38da07]
      memcached<0.84.0>: /usr/lib/system/libsystem_pthread.dylib(thread_start+0xd) [0x7fff9e38d231]
      memcached<0.84.0>: libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument

      [ns_server:info,2017-03-23T08:40:03.141-07:00,babysitter_of_ns_1@127.0.0.1:<0.83.0>:supervisor_cushion:handle_info:58]Cushion managed supervisor for memcached failed:

      {abnormal,134}
      [ns_server:info,2017-03-23T08:40:03.141-07:00,babysitter_of_ns_1@127.0.0.1:<0.83.0>:supervisor_cushion:die_slowly:80]Service memcached exited on node 'babysitter_of_ns_1@127.0.0.1' in 0.07s

      [error_logger:error,2017-03-23T08:40:03.141-07:00,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]** Generic server <0.84.0> terminating
      ** Last message in was {#Port<0.3689>,{exit_status,134}}
      ** When Server state == {state,#Port<0.3689>,memcached,
      {["libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument", "\t/usr/lib/system/libsystem_pthread.dylib(thread_start+0xd) [0x7fff9e38d231]", "\t/usr/lib/system/libsystem_pthread.dylib(_pthread_body+0) [0x7fff9e38da07]", "\t/usr/lib/system/libsystem_pthread.dylib(_pthread_body+0xb4) [0x7fff9e38dabb]"], ["\t/Applications/Couchbase Server.app/Contents/Resources/couchbase-core/lib/libplatform_so.0.1.0.dylib(_ZL20platform_thread_wrapPv+0xf) [0x10fdee68f]"]},
      undefined,
      {ok,{1490283603320,#Ref<0.0.0.270>}},
      ["libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument",
      "\t/usr/lib/system/libsystem_pthread.dylib(thread_start+0xd) [0x7fff9e38d231]",
      "\t/usr/lib/system/libsystem_pthread.dylib(_pthread_body+0) [0x7fff9e38da07]",
      "\t/usr/lib/system/libsystem_pthread.dylib(_pthread_body+0xb4) [0x7fff9e38dabb]",
      "\t/Applications/Couchbase Server.app/Contents/Resources/couchbase-core/lib/libplatform_so.0.1.0.dylib(_ZL20platform_thread_wrapPv+0xf) [0x10fdee68f]",
      "\t/Applications/Couchbase Server.app/Contents/Resources/couchbase-core/lib/libphosphor.dylib(_ZN8phosphor8TraceLog16deregisterThreadEv+0x21) [0x10fe17791]",
      "\t/usr/lib/libc++.1.dylib(ZNSt3120_throw_system_errorEiPKc+0x4d) [0x7fff9cd5d441]",
      "\t/usr/lib/libc++abi.dylib(ZN10_cxxabiv1L22exception_cleanup_funcE19_Unwind_Reason_CodeP17_Unwind_Exception+0) [0x7fff9cd8f7de]",
      "\t/usr/lib/libc++abi.dylib(ZSt11_terminatePFvvE+0x8) [0x7fff9cd8fd69]",
      "\t/Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/memcached(_ZL27backtrace_terminate_handlerv+0x89) [0x10ea35349]",
      "Call stack:",
      "*** Fatal error encountered during exception handling ***",
      "2017-03-23T08:40:03.139711-07:00 FATAL Failed to create listening socket",
      "2017-03-23T08:40:03.122346-07:00 WARNING Failed to set the number of file descriptors to 66048 due to system resource restrictions. This may cause the system to misbehave once you reach a high connection count as the system won't be able open new files on the system. The maximum number of file descriptors is currently set to 10240. The system is configured to allow 65000 number of client connections, and in addition to that the overhead of the worker threads is 24. Finally the backed database needs to open files to persist data.",
      "2017-03-23T08:40:03.120842-07:00 WARNING Restarting file logging",
      "Setting SSL minimum protocol to: tlsv1"],
      0}
      ** Reason for termination ==
      ** {abnormal,134}

      [error_logger:error,2017-03-23T08:40:03.146-07:00,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
      =========================CRASH REPORT=========================
      crasher:
      initial call: ns_port_server:init/1
      pid: <0.84.0>
      registered_name: []
      exception exit:

      {abnormal,134}

      in function gen_server:terminate/6 (gen_server.erl, line 744)
      ancestors: [<0.83.0>,<0.82.0>,ns_child_ports_sup,ns_babysitter_sup,
      <0.59.0>]
      messages: [

      {'EXIT',#Port<0.3689>,normal}

      ]
      links: [<0.83.0>]
      dictionary: []
      trap_exit: true
      status: running
      heap_size: 6772
      stack_size: 27
      reductions: 2606
      neighbours:

      On QE Sanity, the last known good build was 5.0.0-2371. After this, the tests are failing consistently on Build Sanity and QE Sanity suites.

      This issue is not seen on other platforms - Linux or Windows.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            trond Trond Norbye
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty