Details
-
Bug
-
Resolution: Fixed
-
Major
-
5.5.0, 5.5.1, 5.5.2, 5.5.3, 6.0.0, 6.5.0
-
Untriaged
-
Unknown
Description
This was seen out in the wild where memcached would crash due to the following assertion:
[ns_server:info,2018-11-23T17:33:04.028-05:00,babysitter_of_ns_1@127.0.0.1:<0.80.0>:ns_port_server:log:223]memcached<0.80.0>: assertion failed [file == NULL] at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/auditd/src/auditfile.cc:94
|
memcached<0.80.0>: Called from:
|
memcached<0.80.0>: /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0(cb_assert_die+0x49) [0x7f57876d6000+0xc179]
|
memcached<0.80.0>: /opt/couchbase/bin/../lib/libauditd.so.0.1.0() [0x7f5789802000+0x23378]
|
memcached<0.80.0>: /opt/couchbase/bin/../lib/libauditd.so.0.1.0() [0x7f5789802000+0x24317]
|
memcached<0.80.0>: /opt/couchbase/bin/../lib/libauditd.so.0.1.0() [0x7f5789802000+0x1e2f3]
|
memcached<0.80.0>: /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f57876d6000+0x8f07]
|
memcached<0.80.0>: /lib64/libpthread.so.0() [0x7f5785849000+0x7e25]
|
memcached<0.80.0>: /lib64/libc.so.6(clone+0x6d) [0x7f5785486000+0xf834d]
|
I haven't been able to reproduce the exact sequence of steps to trigger this bug, but it seems this is due to rotating an empty audit log file.
The assertion comes from http://src.couchbase.org/source/xref/5.5.2/kv_engine/auditd/src/auditfile.cc#94, where we're opening a new file after rotating out the old one (i.e. file should be a nullptr as there is no 'current' file).
The only places the AuditFile::open() method get called are within AuditFile::ensure_open() - http://src.couchbase.org/source/xref/5.5.2/kv_engine/auditd/src/auditfile.h#70.
There are 2 logical paths:
- One checks if file == NULL, if so then it just calls open directly, clearly this cannot cause the assertion.
- The other performs a rotation on the audit file before calling AuditFile::open(), this rotation will set file to NULL, so it's not possible to cause the crash from here. However, there is an edge case where the current audit log is empty and it is also time to rotate the audit log which is covered in http://src.couchbase.org/source/xref/5.5.2/kv_engine/auditd/src/auditfile.h#51.
In this second case, we do not actually close the file handle, we just adjust the last touched time. However, we then return true from the method, so we perform AuditFile::open() anyway without changing the value of the file variable.
As a result of this, the assertion in AuditFile::open() fails and causes the crash we see here.
As I mentioned I haven't got the list of external steps to reproduce this issue, however here's a unit test which reproduces the crash every time (in auditd/tests/auditfile_test.cc):
/**
|
* Test that an empty file is properly rotated using ensure_open()
|
*/
|
TEST_F(AuditFileTest, TestRotateEmptyFile) {
|
AuditConfig defaultvalue;
|
config.set_rotate_interval(defaultvalue.get_min_file_rotation_time());
|
config.set_rotate_size(1024*1024);
|
|
|
AuditFile auditfile;
|
auditfile.reconfigure(config);
|
|
|
auditfile.ensure_open();
|
cb_timeofday_timetravel(defaultvalue.get_min_file_rotation_time() + 1);
|
auditfile.ensure_open();
|
|
|
auditfile.close();
|
}
|
./memcached_auditfile_test
|
Running main() from gtest_main.cc
|
[==========] Running 9 tests from 1 test case.
|
[----------] Global test environment set-up.
|
[----------] 9 tests from AuditFileTest
|
[ RUN ] AuditFileTest.TestFileCreation
|
[ OK ] AuditFileTest.TestFileCreation (3 ms)
|
[ RUN ] AuditFileTest.TestTimeRotate
|
[ OK ] AuditFileTest.TestTimeRotate (7 ms)
|
[ RUN ] AuditFileTest.TestRotateEmptyFile
|
assertion failed [file == NULL] at /Users/matt/couchbase_build/kv_engine/auditd/src/auditfile.cc:94
|
Called from:
|
/Users/matt/couchbase_build/build/platform/libplatform_so.0.1.0.dylib(cb_assert_die+0x4e) [0x10aabb000+0x5f3e]
|
/Users/matt/couchbase_build/build/kv_engine/./memcached_auditfile_test(_ZN9AuditFile4openEv+0x2b6) [0x10a9a6000+0x164c6]
|
/Users/matt/couchbase_build/build/kv_engine/./memcached_auditfile_test(_ZN38AuditFileTest_TestRotateEmptyFile_Test8TestBodyEv+0x4a4) [0x10a9a6000+0x4154]
|
/Users/matt/couchbase_build/build/kv_engine/./memcached_auditfile_test(_ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS_4TestEvEET0_PT_MS4_FS3_vEPKc+0x48) [0x10a9a6000+0x2b0d8]
|
/Users/matt/couchbase_build/build/kv_engine/./memcached_auditfile_test(_ZN7testing4Test3RunEv+0xfd) [0x10a9a6000+0x2b02d]
|
/Users/matt/couchbase_build/build/kv_engine/./memcached_auditfile_test(_ZN7testing8TestInfo3RunEv+0x130) [0x10a9a6000+0x2c220]
|
/Users/matt/couchbase_build/build/kv_engine/./memcached_auditfile_test(_ZN7testing8TestCase3RunEv+0x117) [0x10a9a6000+0x2cb17]
|
/Users/matt/couchbase_build/build/kv_engine/./memcached_auditfile_test(_ZN7testing8internal12UnitTestImpl11RunAllTestsEv+0x4b7) [0x10a9a6000+0x35037]
|
/Users/matt/couchbase_build/build/kv_engine/./memcached_auditfile_test(_ZN7testing8internal35HandleExceptionsInMethodIfSupportedINS0_12UnitTestImplEbEET0_PT_MS4_FS3_vEPKc+0x48) [0x10a9a6000+0x349e8]
|
/Users/matt/couchbase_build/build/kv_engine/./memcached_auditfile_test(_ZN7testing8UnitTest3RunEv+0xaa) [0x10a9a6000+0x3495a]
|
/Users/matt/couchbase_build/build/kv_engine/./memcached_auditfile_test(main+0x31) [0x10a9a6000+0x47901]
|
/usr/lib/system/libdyld.dylib(start+0x1) [0x7fffbad94000+0x5235]
|
[0x1]
|
[1] 36398 abort ./memcached_auditfile_test
|
Attachments
Issue Links
- is a backport of
-
-
- Closed
-