Details
Description
Using the UI you can set the Audit log location to be any directory which exists (it does not let you set a non-existent directory). However you can set the audit file to be located in a directory which the 'couchbase' user does not have permissions to write to (but can read).
For my test I set the directory to be '/'.
When you do this and an action is taken which triggers an audit event, memcached tries to write the event to the file and then crashes, causing it to restart. This is pretty painful given that users may make this simple mistake and then cause them to suffer downtime (or at best, a failover).
Tracing through the core file I can see that the crashing thread is as follows:
(gdb) bt
|
#0 0x00007fb6cb89fc37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
|
#1 0x00007fb6cb8a3028 in __GI_abort () at abort.c:89
|
#2 0x00007fb6cbea4535 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
|
#3 0x00000000004307af in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/memcached/utilities/terminate_handler.cc:63
|
#4 0x00007fb6cbea26d6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
|
#5 0x00007fb6cbea2703 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
|
#6 0x00007fb6cbea2922 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
|
#7 0x00007fb6cbef41c7 in std::__throw_logic_error(char const*) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
|
#8 0x00007fb6cbeff9f1 in char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
|
#9 0x00007fb6cbeffd88 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
|
#10 0x00007fb6ce8cec26 in Event::process (this=0x7fb695dc3280, audit=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/memcached/auditd/src/event.cc:58
|
#11 0x00007fb6ce8c8cca in consume_events (arg=0x7fb6ca8b1000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/memcached/auditd/src/auditd.cc:66
|
#12 0x00007fb6cd24ce3e in run (this=0x7fb6ca904700) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58
|
#13 platform_thread_wrap (arg=0x7fb6ca904700) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71
|
Tracing this through, we get to the following line in the code where we cannot open the file (https://github.com/couchbase/memcached/blob/109a2d14c20d03f8cd0f602bf72db77638348010/auditd/src/event.cc#L58):
Audit::log_error(AuditErrorCode::OPEN_AUDITFILE_ERROR, NULL);
|
The method signature for Audit::log_error is as follows:
void Audit::log_error(const AuditErrorCode return_code,
|
const std::string& string)
|
The second argument is converting the value at the address of the pointer to a std::string. As we're passing it a null pointer it crashes.
Logs demonstrating the problem:
- https://s3.amazonaws.com/cb-engineering/MB-23269/collectinfo-2017-03-10T151027-ns_1%4010.142.170.101.zip (node that the backtrace is from)
- https://s3.amazonaws.com/cb-engineering/MB-23269/collectinfo-2017-03-10T151027-ns_1%4010.142.170.102.zip
- https://s3.amazonaws.com/cb-engineering/MB-23269/collectinfo-2017-03-10T151027-ns_1%4010.142.170.103.zip
Attachments
Issue Links
- blocks
-
MB-22188 4.6.2 Minor Release
-
- Closed
-