Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-59966

Fix Magma memory tracking issues exposed by arena debug checks

    XMLWordPrintable

Details

    • 0

    Description

      As highlighted by the changes made for MB-58644, Magma has at least one instance (and potentially many) where memory is deallocated from a different context (bucket / domain) than where it was originally allocated.

      For example, when running ep_testsuite under Magma with debug checks enabled (see https://review.couchbase.org/c/kv_engine/+/199673):

          Start 336: ep_testsuite.value_eviction.magma
      		 
          ...
      		 
          Running [4/156]: expiration on compaction...===ERROR===: JeArenaMalloc deallocation mismatch
      		 
      	Memory freed by client:0 domain:1 which is assigned arena:3, but memory was previously allocated from arena:2.
      		 
      	Allocation address:0x107000960 size:16
      		 
      	#0  /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(cb::verifyMemDeallocatedByCorrectClient(cb::JEArenaMallocBase::CurrentClient const&, void*, unsigned long)+0x194) [0x100000000+0xf54d8]
      		 
      	#1  /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(cb::_JEArenaMalloc<cb::JEArenaSimpleTracker>::sized_free(void*, unsigned long)+0x50) [0x100000000+0xf5628]
      		 
      	#2  /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(cb_sized_free+0x34) [0x100000000+0x33a74]
      		 
      	#3  /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(MagmaKVStore::MagmaCompactionCB::MagmaCompactionCB(MagmaKVStore&, Vbid, std::__1::shared_ptr<CompactionContext>, std::__1::optional<CollectionID>)+0x108) [0x100000000+0x1973d0]
      		 
      	#4  /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(std::__1::__function::__func<MagmaKVStore::compactDBInternal(std::__1::unique_lock<std::__1::mutex>&, std::__1::shared_ptr<CompactionContext>)::$_49, std::__1::allocator<MagmaKVStore::compactDBInternal(std::__1::
      		 
      	#5  /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(std::__1::__function::__func<magma::KVStore::makeGCContext(std::__1::function<std::__1::unique_ptr<magma::Magma::CompactionCallback, std::__1::default_delete<magma::Magma::CompactionCallback>> (unsigned short)> c
      		 
      	#6  /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(magma::DocSeqGCContext::DocSeqGCContext(magma::DocSequenceCache&, bool, std::__1::function<unsigned long long (magma::Slice const&)>, std::__1::function<std::__1::unique_ptr<magma::Magma::CompactionCallback, std:
      		 
      	#7  /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(std::__1::__unique_if<magma::DocSeqGCContext>::__unique_single std::__1::make_unique[abi:v160006]<magma::DocSeqGCContext, magma::DocSequenceCache&, bool&, std::__1::function<unsigned long long (magma::Slice const
      		 
      	#8  /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(std::__1::__function::__func<magma::KVStore::makeGCContext(std::__1::function<std::__1::unique_ptr<magma::Magma::CompactionCallback, std::__1::default_delete<magma::Magma::CompactionCallback>> (unsigned short)> c
      		 
      	#9  /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(magma::LSMTree::newGCFilter(std::__1::function<std::__1::unique_ptr<magma::GCContext, std::__1::default_delete<magma::GCContext>> (bool)>, bool, long long, bool)+0x44) [0x100000000+0x44d4a4]
      		 
      	#10 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(magma::LSMTree::compactLevelForInputs(int, int, std::__1::unique_lock<std::__1::mutex>&, magma::LSMTree::CompactionTrigger, std::__1::vector<std::__1::shared_ptr<magma::Table>, std::__1::allocator<std::__1::share
      		 
      	#11 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(magma::LSMTree::compactLevel(int, std::__1::unique_lock<std::__1::mutex>&, magma::LSMTree::CompactionTrigger, std::__1::function<std::__1::unique_ptr<magma::GCContext, std::__1::default_delete<magma::GCContext>>
      		 
      	#12 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(magma::LSMTree::compactAll(int, std::__1::function<std::__1::unique_ptr<magma::GCContext, std::__1::default_delete<magma::GCContext>> (bool)>)+0xf8) [0x100000000+0x4146f8]
      		 
      	#13 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(magma::LSMTree::CompactAll(std::__1::function<std::__1::unique_ptr<magma::GCContext, std::__1::default_delete<magma::GCContext>> (bool)>)+0xa4) [0x100000000+0x4148f4]
      		 
      	#14 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(magma::KVStore::CompactAll(magma::Magma::StoreType, std::__1::function<std::__1::unique_ptr<magma::Magma::CompactionCallback, std::__1::default_delete<magma::Magma::CompactionCallback>> (unsigned short)>)+0x170)
      		 
      	#15 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(magma::Magma::Impl::CompactKVStore(unsigned short, magma::Magma::StoreType, std::__1::function<std::__1::unique_ptr<magma::Magma::CompactionCallback, std::__1::default_delete<magma::Magma::CompactionCallback>> (u
      		 
      	#16 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(magma::Magma::Impl::CompactKVStore(unsigned short, magma::Magma::StoreType, std::__1::function<std::__1::unique_ptr<magma::Magma::CompactionCallback, std::__1::default_delete<magma::Magma::CompactionCallback>> (u
      		 
      	#17 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(magma::Magma::Impl::CompactKVStore(unsigned short, magma::Slice const&, magma::Slice const&, std::__1::function<std::__1::unique_ptr<magma::Magma::CompactionCallback, std::__1::default_delete<magma::Magma::Compac
      		 
      	#18 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(magma::Magma::CompactKVStore(unsigned short, magma::Slice const&, magma::Slice const&, std::__1::function<std::__1::unique_ptr<magma::Magma::CompactionCallback, std::__1::default_delete<magma::Magma::CompactionCa
      		 
      	#19 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(MagmaMemoryTrackingProxy::CompactKVStore(unsigned short, magma::Slice const&, magma::Slice const&, std::__1::function<std::__1::unique_ptr<magma::Magma::CompactionCallback, std::__1::default_delete<magma::Magma::
      		 
      	#20 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(MagmaKVStore::compactDBInternal(std::__1::unique_lock<std::__1::mutex>&, std::__1::shared_ptr<CompactionContext>)+0xb2c) [0x100000000+0x1a512c]
      		 
      	#21 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(MagmaKVStore::compactDB(std::__1::unique_lock<std::__1::mutex>&, std::__1::shared_ptr<CompactionContext>)+0x50) [0x100000000+0x1a4590]
      		 
      	#22 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(EPBucket::compactInternal(LockedVBucketPtr&, CompactionConfig&)+0x80) [0x100000000+0x2ab1c8]
      		 
      	#23 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(EPBucket::doCompact(Vbid, CompactionConfig&, std::__1::vector<CookieIface*, std::__1::allocator<CookieIface*>>&)+0x5c) [0x100000000+0x2ab5bc]
      		 
      	#24 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(CompactTask::runInner()+0x10c) [0x100000000+0x1feca0]
      		 
      	#25 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(GlobalTask::execute(std::__1::basic_string_view<char, std::__1::char_traits<char>>)+0x12c) [0x100000000+0xcd34c]
      		 
      	#26 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(EpTask::execute(std::__1::basic_string_view<char, std::__1::char_traits<char>>)+0x3c) [0x100000000+0x295ba8]
      		 
      	#27 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::'lambda0'()::operator()() const+0xe0) [0x100000000+0xd00e8]
      		 
      	#28 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(void folly::detail::function::FunctionTraits<void ()>::callSmall<CancellableCPUExecutor::add(GlobalTask*, folly::Function<void ()>)::$_0>(folly::detail::function::Data&)+0x2f8) [0x100000000+0xd72a0]
      		 
      	#29 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(folly::ThreadPoolExecutor::runTask(std::__1::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&)+0x94) [0x100000000+0x50e9c4]
      		 
      	#30 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(folly::CPUThreadPoolExecutor::threadRun(std::__1::shared_ptr<folly::ThreadPoolExecutor::Thread>)+0x184) [0x100000000+0x4fdf28]
      		 
      	#31 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(void folly::detail::function::FunctionTraits<void ()>::callSmall<std::__1::__bind<void (folly::ThreadPoolExecutor::*)(std::__1::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*, std::__1
      		 
      	#32 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(CBRegisteredThreadFactory::newThread(folly::Function<void ()>&&)::'lambda'()::operator()()+0x74) [0x100000000+0xce5bc]
      		 
      	#33 /Users/dave/repos/couchbase/server/source/build-debug-arm64/kv_engine/ep_testsuite(void* std::__1::__thread_proxy[abi:v160006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, folly::NamedThreadFactory::newThread(folly::Functi
      		 
      	#34 /usr/lib/system/libsystem_pthread.dylib(_pthread_start+0x94) [0x19ca51000+0x6fa8]
      		 
      	#35 /usr/lib/system/libsystem_pthread.dylib(thread_start+0x8) [0x19ca51000+0x1da0]

      Steps to Reproduce

      Build with linked patch (https://review.couchbase.org/c/kv_engine/+/199673), and then run the ep_testsuite tests with a Magma variant - similar backtraces to the above should be seen.

      it can also be helpful to run under a debugger, then you'll see the debugger stop in the abort().

      Attachments

        Issue Links

          relates to

          Task - A task that needs to be done. MB-58644 Improve checks in per-bucket memory tracking in Debug builds

          • Major - Major loss of function.
          • Resolved
          For Gerrit Dashboard: MB-59966
          # Subject Branch Project Status CR V
          199673,6 MB-59966: Enable memory checking for Magma ep_testsuite tests master kv_engine Status: NEW 0 -1
          202274,1 WIP MB-59966 Temp patch for kv team master magma Status: NEW 0 0

          Activity

            People

              owend Daniel Owen
              drigby Dave Rigby (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are 2 open Gerrit changes

                  PagerDuty