Basic crash dump analysis of a.core. Please send the file to support@couchbase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x. 1 couchbase couchbase 10399688 Jul 9 02:34 /opt/couchbase/bin/memcached 5a4f602e22fb912c85dfa8a3cc0dce78 /opt/couchbase/bin/memcached -rw-r--r--. 1 root root 1810432 Jul 12 05:55 a.core 779c8a90b727199b1fb760aa0976b632 a.core -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/couchbase/bin/memcached...Reading symbols from /opt/couchbase/bin/memcached...(no debugging symbols found)...done. (no debugging symbols found)...done. [New LWP 125477] [New LWP 125183] [New LWP 125184] [New LWP 125192] [New LWP 125193] [New LWP 125194] [New LWP 125198] [New LWP 125200] [New LWP 125220] [New LWP 125221] [New LWP 125222] [New LWP 125223] [New LWP 125224] [New LWP 125225] [New LWP 125226] [New LWP 125227] [New LWP 125228] [New LWP 125229] [New LWP 125230] [New LWP 125231] [New LWP 125232] [New LWP 125233] [New LWP 125234] [New LWP 125235] [New LWP 125236] [New LWP 125237] [New LWP 125238] [New LWP 125239] [New LWP 125240] [New LWP 125241] [New LWP 125242] [New LWP 125243] [New LWP 125244] [New LWP 125245] [New LWP 125467] [New LWP 125469] [New LWP 125470] [New LWP 125471] [New LWP 125472] [New LWP 125473] [New LWP 125474] [New LWP 125475] [New LWP 125476] [New LWP 125478] [New LWP 125479] [New LWP 125480] [New LWP 125481] [New LWP 125482] [New LWP 125483] [New LWP 125484] [New LWP 125485] [New LWP 125486] [New LWP 125487] [New LWP 125488] Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error warning: File "/usr/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py". To enable execution of this file add add-auto-load-safe-path /usr/lib64/libthread_db-1.0.so line to your configuration file "/root/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/root/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error warning: File "/usr/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py". warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. Failed to read a valid object file image from memory. Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007f747af55337 in raise () from /lib64/libc.so.6 warning: File "/opt/couchbase/lib/libstdc++.so.6.0.28-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py". Thread 54 (LWP 125488): #0 0x0000000000b05d88 in cb::JEArenaCoreLocalTracker::memAllocated(unsigned char, unsigned long, std::align_val_t) () #1 0x0000000000b06277 in cb::_JEArenaMalloc::malloc(unsigned long) () #2 0x00000000005aad3f in cb_malloc () #3 0x0000000000afa159 in operator new(unsigned long) () #4 0x00000000007cd34f in DiskDocKey::DiskDocKey(char const*, unsigned long) () #5 0x000000000085ac91 in MagmaKVStore::compactionCallBack(MagmaKVStore::MagmaCompactionCB&, magma::Slice const&, magma::Slice const&, magma::Slice const&) () #6 0x000000000090376d in magma::DocSeqGCContext::Transform(magma::Slice const&, magma::Slice const&, std::__cxx11::basic_string, std::allocator >&, magma::Slice*) () #7 0x000000000098f925 in magma::ItemGCFilter::Process(std::vector, std::allocator > >&, std::vector, std::allocator > >*) () #8 0x000000000091c6d2 in magma::LSMTree::mergeSSTables(std::vector, std::allocator > >&, magma::ItemGCFilter&, unsigned long, double, std::vector, std::allocator > >*) () #9 0x0000000000988771 in magma::LSMTree::compactLevelForInputs(int, int, std::unique_lock&, std::vector, std::allocator > >&, std::vector, std::allocator > >&, bool, std::function > (bool)>) () #10 0x000000000098b1ca in magma::LSMTree::compactLevel(int, std::unique_lock&, bool, std::function > (bool)>) () #11 0x000000000098b45c in magma::LSMTree::compact(std::unique_lock&) () #12 0x0000000000912298 in std::_Function_handler::_M_invoke(std::_Any_data const&) () #13 0x000000000094ec3d in magma::TimedTask::Complete(bool) () #14 0x0000000000950431 in magma::TaskWorker::loop(void*) () #15 0x0000000000af4db9 in platform_thread_wrap(void*) () #16 0x00007f747d6f0e65 in start_thread () from /lib64/libpthread.so.0 #17 0x00007f747b01d88d in clone () from /lib64/libc.so.6 Thread 53 (LWP 125487): #0 0x0000000000730412 in ObjectRegistry::onSwitchThread(EventuallyPersistentEngine*, bool) () #1 0x00000000009540a2 in magma::ExecutionEnvGuard::ExecutionEnvGuard(void*) () #2 0x000000000094a240 in magma::FileManager::GetInstance() () #3 0x00000000009458dc in magma::ManagedFile::ManagedFile(std::function > (std::__cxx11::basic_string, std::allocator > const&)>, std::function, std::allocator > ()>, magma::BlockCache*) () #4 0x000000000096d72d in magma::SSTable::Open() () #5 0x000000000091c930 in magma::LSMTree::mergeSSTables(std::vector, std::allocator > >&, magma::ItemGCFilter&, unsigned long, double, std::vector, std::allocator > >*) () #6 0x0000000000988771 in magma::LSMTree::compactLevelForInputs(int, int, std::unique_lock&, std::vector, std::allocator > >&, std::vector, std::allocator > >&, bool, std::function > (bool)>) () #7 0x000000000098b1ca in magma::LSMTree::compactLevel(int, std::unique_lock&, bool, std::function > (bool)>) () #8 0x000000000098b45c in magma::LSMTree::compact(std::unique_lock&) () #9 0x0000000000912298 in std::_Function_handler::_M_invoke(std::_Any_data const&) () #10 0x000000000094ec3d in magma::TimedTask::Complete(bool) () #11 0x0000000000950431 in magma::TaskWorker::loop(void*) () #12 0x0000000000af4db9 in platform_thread_wrap(void*) () #13 0x00007f747d6f0e65 in start_thread () from /lib64/libpthread.so.0 #14 0x00007f747b01d88d in clone () from /lib64/libc.so.6 Thread 52 (LWP 125486): #0 0x00007f747b014b8d in fdatasync () from /lib64/libc.so.6 #1 0x00000000009404af in magma::BasicFile::Sync() () #2 0x00000000009a89c4 in magma::FileWithStats::Sync() () #3 0x0000000000942c06 in magma::BufferedFileWriter::flush(bool) () #4 0x0000000000942d9d in magma::BufferedFileWriter::Write(char const*, unsigned long, bool) () #5 0x0000000000972348 in magma::SSTableWriter::writeFooter(magma::BlockPointer const&, magma::BlockPointer const&, magma::BlockPointer const&) () #6 0x0000000000974d6f in magma::SSTableWriter::GenerateTable() () #7 0x000000000091c809 in magma::LSMTree::mergeSSTables(std::vector, std::allocator > >&, magma::ItemGCFilter&, unsigned long, double, std::vector, std::allocator > >*) () #8 0x0000000000988771 in magma::LSMTree::compactLevelForInputs(int, int, std::unique_lock&, std::vector, std::allocator > >&, std::vector, std::allocator > >&, bool, std::function > (bool)>) () #9 0x000000000098b1ca in magma::LSMTree::compactLevel(int, std::unique_lock&, bool, std::function > (bool)>) () #10 0x000000000098b45c in magma::LSMTree::compact(std::unique_lock&) () #11 0x0000000000912298 in std::_Function_handler::_M_invoke(std::_Any_data const&) () #12 0x000000000094ec3d in magma::TimedTask::Complete(bool) () #13 0x0000000000950431 in magma::TaskWorker::loop(void*) () #14 0x0000000000af4db9 in platform_thread_wrap(void*) () #15 0x00007f747d6f0e65 in start_thread () from /lib64/libpthread.so.0 #16 0x00007f747b01d88d in clone () from /lib64/libc.so.6 Thread 51 (LWP 125485): #0 0x00007f747d6f7f83 in pread64 () from /lib64/libpthread.so.0 #1 0x000000000093fa3b in magma::BasicFile::Read(long, unsigned long, char*, unsigned long*) () #2 0x00000000009a87da in magma::FileWithStats::Read(long, unsigned long, char*, unsigned long*) () #3 0x00000000009431b7 in magma::BufferedFileReader::Read(magma::File*, long, unsigned long, magma::Slice&, unsigned long&) () #4 0x000000000094e3f5 in magma::ReadAndTransform(magma::File*, long, unsigned long, std::function >&, unsigned long&)>, std::unique_ptr >&, unsigned long&, unsigned long&, magma::BufferedFileReader*) () /tmp/gdb-commands.56625:1: Error in sourced command file: Cannot access memory at address 0x7f73f07dd4c8 Missing separate debuginfos, use: debuginfo-install couchbase-server-7.1.0-1061.x86_64 [?1034h(gdb)