Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37400

cv failure: test vbucket compact no purge ASAN failure with Magma

    XMLWordPrintable

Details

    Description

      As shown in
      http://cv.jenkins.couchbase.com/job/kv_engine.ASan-UBSan/job/master/8804/console

      There are repeated failures when executing the "test vbucket compact no purge" test

      20:03:36  Running [0147/0154]: test vbucket compact no purge...=================================================================
      20:03:36  ==14500==ERROR: AddressSanitizer: heap-use-after-free on address 0x6070003a2a89 at pc 0x7f216e722d32 bp 0x7f2146947070 sp 0x7f2146946820
      20:03:36  READ of size 3 at 0x6070003a2a89 thread T4661 (mc:reader_0)
      20:03:36      #0 0x7f216e722d31  (/home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/install/lib/libasan.so.4+0x75d31)
      20:03:36      #1 0x5c5857 in std::char_traits<char>::copy(char*, char const*, unsigned long) /usr/local/include/c++/7.3.0/bits/char_traits.h:350
      20:03:36      #2 0x5c5857 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_S_copy(char*, char const*, unsigned long) /usr/local/include/c++/7.3.0/bits/basic_string.h:340
      20:03:36      #3 0x5c5857 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_S_copy_chars(char*, char const*, char const*) /usr/local/include/c++/7.3.0/bits/basic_string.h:387
      20:03:36      #4 0x5c5857 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) /usr/local/include/c++/7.3.0/bits/basic_string.tcc:225
      20:03:36      #5 0x7f21690e5d32 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct_aux<char const*>(char const*, char const*, std::__false_type) /usr/local/include/c++/7.3.0/bits/basic_string.h:236
      20:03:36      #6 0x7f21690e5d32 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*) /usr/local/include/c++/7.3.0/bits/basic_string.h:255
      20:03:36      #7 0x7f21690e5d32 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, unsigned long, std::allocator<char> const&) /usr/local/include/c++/7.3.0/bits/basic_string.h:502
      20:03:36      #8 0x7f21690e5d32 in DiskDocKey::DiskDocKey(char const*, unsigned long) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/engines/ep/src/diskdockey.cc:48
      20:03:36      #9 0x7f21699d319e in makeDiskDocKey /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:374
      20:03:36      #10 0x7f21699fe6ec in MagmaKVStore::scan(ScanContext*) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:1634
      20:03:36      #11 0x7f21697adab4 in Warmup::keyDumpforShard(unsigned short) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/engines/ep/src/warmup.cc:1298
      20:03:36      #12 0x7f21697f893f in WarmupKeyDump::run() /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/engines/ep/src/warmup.cc:369
      20:03:36      #13 0x7f216943f0ba in GlobalTask::execute() /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/engines/ep/src/globaltask.cc:73
      20:03:36      #14 0x7f2169407cbd in ExecutorThread::run() /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/engines/ep/src/executorthread.cc:188
      20:03:36      #15 0x7f216940a403 in launch_executor_thread /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/engines/ep/src/executorthread.cc:36
      20:03:36      #16 0x7f21620c7708 in CouchbaseThread::run() /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/platform/src/cb_pthreads.cc:58
      20:03:36      #17 0x7f21620c7708 in platform_thread_wrap /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/platform/src/cb_pthreads.cc:71
      20:03:36      #18 0x7f21602ab6b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
      20:03:36      #19 0x7f215ffe141c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10741c)
      20:03:36  
      20:03:36  0x6070003a2a89 is located 25 bytes inside of 76-byte region [0x6070003a2a70,0x6070003a2abc)
      20:03:36  freed by thread T4661 (mc:reader_0) here:
      20:03:36      #0 0x7f216e7886b0 in operator delete(void*) (/home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/install/lib/libasan.so.4+0xdb6b0)
      20:03:36      #1 0x7f216181b14e in __gnu_cxx::new_allocator<char>::deallocate(char*, unsigned long) /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:125
      20:03:36      #2 0x7f216181b14e in std::allocator_traits<std::allocator<char> >::deallocate(std::allocator<char>&, char*, unsigned long) /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/alloc_traits.h:462
      20:03:36      #3 0x7f216181b14e in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_destroy(unsigned long) /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:226
      20:03:36      #4 0x7f216181b14e in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_dispose() /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.h:221
      20:03:36      #5 0x7f216181b14e in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:327
      20:03:36  
      20:03:36  previously allocated by thread T4661 (mc:reader_0) here:
      20:03:36      #0 0x7f216e7879b0 in operator new(unsigned long) (/home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/install/lib/libasan.so.4+0xda9b0)
      20:03:36      #1 0x7f216181b0c9 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:317
      20:03:36  
      20:03:36  Thread T4661 (mc:reader_0) created by T0 here:
      20:03:36      #0 0x7f216e6e3a00 in __interceptor_pthread_create (/home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/install/lib/libasan.so.4+0x36a00)
      20:03:36      #1 0x7f21620c71fa in cb_create_named_thread(unsigned long*, void (*)(void*), void*, int, char const*) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/platform/src/cb_pthreads.cc:109
      20:03:36      #2 0x7f2169403afb in ExecutorThread::start() /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/engines/ep/src/executorthread.cc:53
      20:03:36      #3 0x7f21693cd5b5 in ExecutorPool::_adjustWorkers(task_type_t, unsigned long) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/engines/ep/src/executorpool.cc:500
      20:03:36      #4 0x7f21693cf865 in ExecutorPool::_startWorkers() /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/engines/ep/src/executorpool.cc:563
      20:03:36      #5 0x7f21693cb263 in ExecutorPool::_registerTaskable(Taskable&) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/engines/ep/src/executorpool.cc:456
      20:03:36      #6 0x7f21693cb58f in ExecutorPool::registerTaskable(Taskable&) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/engines/ep/src/executorpool.cc:461
      20:03:36      #7 0x7f2169508e6c in KVBucket::KVBucket(EventuallyPersistentEngine&) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/engines/ep/src/kv_bucket.cc:306
      20:03:36      #8 0x7f21691806e9 in EPBucket::EPBucket(EventuallyPersistentEngine&) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/engines/ep/src/ep_bucket.cc:252
      20:03:36      #9 0x7f216924d39e in std::_MakeUniq<EPBucket>::__single_object std::make_unique<EPBucket, EventuallyPersistentEngine&>(EventuallyPersistentEngine&) /usr/local/include/c++/7.3.0/bits/unique_ptr.h:825
      20:03:36      #10 0x7f216924d39e in EventuallyPersistentEngine::makeBucket(Configuration&) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/engines/ep/src/ep_engine.cc:6546
      20:03:36      #11 0x7f216929483c in EventuallyPersistentEngine::initialize(char const*) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/engines/ep/src/ep_engine.cc:2137
      20:03:36      #12 0x710aa9 in MockTestHarness::create_bucket(bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/programs/engine_testapp/engine_testapp.cc:217
      20:03:36      #13 0x70ba2b in MockTestHarness::reload_engine(EngineIface**, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool) /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/programs/engine_testapp/engine_testapp.cc:241
      20:03:36      #14 0x4bac13 in test_vbucket_compact_no_purge /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/engines/ep/tests/ep_testsuite.cc:7906
      20:03:36      #15 0x70678d in execute_test /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/programs/engine_testapp/engine_testapp.cc:356
      20:03:36      #16 0x709fb5 in main /home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/kv_engine/programs/engine_testapp/engine_testapp.cc:609
      20:03:36      #17 0x7f215fefa82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
      20:03:36  
      [2020-01-04T20:03:36.731Z] SUMMARY: AddressSanitizer: heap-use-after-free (/home/couchbase/jenkins/workspace/kv_engine.ASan-UBSan_master/install/lib/libasan.so.4+0x75d31) 
      20:03:36  Shadow bytes around the buggy address:
      20:03:36    0x0c0e8006c500: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
      20:03:36    0x0c0e8006c510: fd fd fa fa fa fa fd fd fd fd fd fd fd fd fd fd
      20:03:36    0x0c0e8006c520: fa fa fa fa 00 00 00 00 00 00 00 00 00 fa fa fa
      20:03:36    0x0c0e8006c530: fa fa 00 00 00 00 00 00 00 00 00 00 fa fa fa fa
      20:03:36    0x0c0e8006c540: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fd fd
      20:03:36  =>0x0c0e8006c550: fd[fd]fd fd fd fd fd fd fa fa fa fa fd fd fd fd
      20:03:36    0x0c0e8006c560: fd fd fd fd fd fa fa fa fa fa 00 00 00 00 00 00
      20:03:36    0x0c0e8006c570: 00 00 00 00 fa fa fa fa fd fd fd fd fd fd fd fd
      20:03:36    0x0c0e8006c580: fd fa fa fa fa fa 00 00 00 00 00 00 00 00 00 00
      20:03:36    0x0c0e8006c590: fa fa fa fa 00 00 00 00 00 00 00 00 00 03 fa fa
      20:03:36    0x0c0e8006c5a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      20:03:36  Shadow byte legend (one shadow byte represents 8 application bytes):
      20:03:36    Addressable:           00
      20:03:36    Partially addressable: 01 02 03 04 05 06 07 
      20:03:36    Heap left redzone:       fa
      20:03:36    Freed heap region:       fd
      20:03:36    Stack left redzone:      f1
      20:03:36    Stack mid redzone:       f2
      20:03:36    Stack right redzone:     f3
      20:03:36    Stack after return:      f5
      20:03:36    Stack use after scope:   f8
      20:03:36    Global redzone:          f9
      20:03:36    Global init order:       f6
      20:03:36    Poisoned by user:        f7
      20:03:36    Container overflow:      fc
      20:03:36    Array cookie:            ac
      20:03:36    Intra object redzone:    bb
      20:03:36    ASan internal:           fe
      20:03:36    Left alloca redzone:     ca
      20:03:36    Right alloca redzone:    cb
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          jwalker Jim Walker added a comment -

          Test is disabled for magma

          jwalker Jim Walker added a comment - Test is disabled for magma

          Build couchbase-server-7.0.0-1171 contains kv_engine commit 45efa82 with commit message:
          MB-37400: Disable test vbucket compact no purge under Magma

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1171 contains kv_engine commit 45efa82 with commit message: MB-37400 : Disable test vbucket compact no purge under Magma

          Build couchbase-server-7.0.0-1810 contains kv_engine commit 026e116 with commit message:
          MB-37400: Enable test vbucket compact no purge for magma

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1810 contains kv_engine commit 026e116 with commit message: MB-37400 : Enable test vbucket compact no purge for magma

          People

            ben.huddleston Ben Huddleston
            owend Daniel Owen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty