Details
-
Bug
-
Resolution: Cannot Reproduce
-
Test Blocker
-
5.0.0
-
MacOS X Sierra, MacOS X El Capitan
-
Untriaged
-
MacOSX 64-bit
-
Yes
Description
Build : 5.0.0-2408
On Mac OS X Sierra and El Capitan, memcached is crashing. The bucket always remain in Warmup state. Following error is seen in the babysitter log :
memcached<0.84.0>: 2017-03-23T08:40:03.122346-07:00 WARNING Failed to set the number of file descriptors to 66048 due to system resource restrictions. This may cause the system to misbehave once you reach a high connection count as the system won't be able open new files on the system. The maximum number of file descriptors is currently set to 10240. The system is configured to allow 65000 number of client connections, and in addition to that the overhead of the worker threads is 24. Finally the backed database needs to open files to persist data.
memcached<0.84.0>: 2017-03-23T08:40:03.139711-07:00 FATAL Failed to create listening socket
memcached<0.84.0>: *** Fatal error encountered during exception handling ***
memcached<0.84.0>: Call stack:
memcached<0.84.0>: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/memcached(_ZL27backtrace_terminate_handlerv+0x89) [0x10ea35349]
memcached<0.84.0>: /usr/lib/libc++abi.dylib(ZSt11_terminatePFvvE+0x8) [0x7fff9cd8fd69]
memcached<0.84.0>: /usr/lib/libc++abi.dylib(ZN10_cxxabiv1L22exception_cleanup_funcE19_Unwind_Reason_CodeP17_Unwind_Exception+0) [0x7fff9cd8f7de]
memcached<0.84.0>: /usr/lib/libc++.1.dylib(ZNSt3120_throw_system_errorEiPKc+0x4d) [0x7fff9cd5d441]
memcached<0.84.0>: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/lib/libphosphor.dylib(_ZN8phosphor8TraceLog16deregisterThreadEv+0x21) [0x10fe17791]
memcached<0.84.0>: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/lib/libplatform_so.0.1.0.dylib(_ZL20platform_thread_wrapPv+0xf) [0x10fdee68f]
memcached<0.84.0>: /usr/lib/system/libsystem_pthread.dylib(_pthread_body+0xb4) [0x7fff9e38dabb]
memcached<0.84.0>: /usr/lib/system/libsystem_pthread.dylib(_pthread_body+0) [0x7fff9e38da07]
memcached<0.84.0>: /usr/lib/system/libsystem_pthread.dylib(thread_start+0xd) [0x7fff9e38d231]
memcached<0.84.0>: libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument
[ns_server:info,2017-03-23T08:40:03.141-07:00,babysitter_of_ns_1@127.0.0.1:<0.83.0>:supervisor_cushion:handle_info:58]Cushion managed supervisor for memcached failed:
{abnormal,134}[ns_server:info,2017-03-23T08:40:03.141-07:00,babysitter_of_ns_1@127.0.0.1:<0.83.0>:supervisor_cushion:die_slowly:80]Service memcached exited on node 'babysitter_of_ns_1@127.0.0.1' in 0.07s
[error_logger:error,2017-03-23T08:40:03.141-07:00,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]** Generic server <0.84.0> terminating
** Last message in was {#Port<0.3689>,{exit_status,134}}
** When Server state == {state,#Port<0.3689>,memcached,
{["libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument", "\t/usr/lib/system/libsystem_pthread.dylib(thread_start+0xd) [0x7fff9e38d231]", "\t/usr/lib/system/libsystem_pthread.dylib(_pthread_body+0) [0x7fff9e38da07]", "\t/usr/lib/system/libsystem_pthread.dylib(_pthread_body+0xb4) [0x7fff9e38dabb]"], ["\t/Applications/Couchbase Server.app/Contents/Resources/couchbase-core/lib/libplatform_so.0.1.0.dylib(_ZL20platform_thread_wrapPv+0xf) [0x10fdee68f]"]},
undefined,
{ok,{1490283603320,#Ref<0.0.0.270>}},
["libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument",
"\t/usr/lib/system/libsystem_pthread.dylib(thread_start+0xd) [0x7fff9e38d231]",
"\t/usr/lib/system/libsystem_pthread.dylib(_pthread_body+0) [0x7fff9e38da07]",
"\t/usr/lib/system/libsystem_pthread.dylib(_pthread_body+0xb4) [0x7fff9e38dabb]",
"\t/Applications/Couchbase Server.app/Contents/Resources/couchbase-core/lib/libplatform_so.0.1.0.dylib(_ZL20platform_thread_wrapPv+0xf) [0x10fdee68f]",
"\t/Applications/Couchbase Server.app/Contents/Resources/couchbase-core/lib/libphosphor.dylib(_ZN8phosphor8TraceLog16deregisterThreadEv+0x21) [0x10fe17791]",
"\t/usr/lib/libc++.1.dylib(ZNSt3120_throw_system_errorEiPKc+0x4d) [0x7fff9cd5d441]",
"\t/usr/lib/libc++abi.dylib(ZN10_cxxabiv1L22exception_cleanup_funcE19_Unwind_Reason_CodeP17_Unwind_Exception+0) [0x7fff9cd8f7de]",
"\t/usr/lib/libc++abi.dylib(ZSt11_terminatePFvvE+0x8) [0x7fff9cd8fd69]",
"\t/Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/memcached(_ZL27backtrace_terminate_handlerv+0x89) [0x10ea35349]",
"Call stack:",
"*** Fatal error encountered during exception handling ***",
"2017-03-23T08:40:03.139711-07:00 FATAL Failed to create listening socket",
"2017-03-23T08:40:03.122346-07:00 WARNING Failed to set the number of file descriptors to 66048 due to system resource restrictions. This may cause the system to misbehave once you reach a high connection count as the system won't be able open new files on the system. The maximum number of file descriptors is currently set to 10240. The system is configured to allow 65000 number of client connections, and in addition to that the overhead of the worker threads is 24. Finally the backed database needs to open files to persist data.",
"2017-03-23T08:40:03.120842-07:00 WARNING Restarting file logging",
"Setting SSL minimum protocol to: tlsv1"],
0}
** Reason for termination ==
** {abnormal,134}
[error_logger:error,2017-03-23T08:40:03.146-07:00,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:do_log:203]
=========================CRASH REPORT=========================
crasher:
initial call: ns_port_server:init/1
pid: <0.84.0>
registered_name: []
exception exit:
in function gen_server:terminate/6 (gen_server.erl, line 744)
ancestors: [<0.83.0>,<0.82.0>,ns_child_ports_sup,ns_babysitter_sup,
<0.59.0>]
messages: [
]
links: [<0.83.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 6772
stack_size: 27
reductions: 2606
neighbours:
On QE Sanity, the last known good build was 5.0.0-2371. After this, the tests are failing consistently on Build Sanity and QE Sanity suites.
This issue is not seen on other platforms - Linux or Windows.