Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48708

[Magma] - Caught unhandled std::exception-derived exception. what(): std::bad_alloc

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Critical
    • 7.1.0
    • 7.1.0
    • qe
    • 7.1.0-1380-enterprise
    • Untriaged
    • 1
    • No

    Description

      Script to Repro

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.40477.ini GROUP=failover_with_rebalance_out_P0_set0,rerun=False,upgrade_version=7.1.0-1386 -t test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=1,compaction=True,bucket_spec=magma_dgm.10_percent_dgm.5_node_2_replica_magma_512,doc_size=512,randomize_value=True,data_load_stage=during,skip_validations=False,GROUP=graceful_failover_and_rebalance_out_P0_set0
      

      Steps to Repro
      1. Create a 5 node cluster

      2021-09-30 11:29:49,002 | test  | INFO    | MainProcess | pool-12-thread-6 | [table_view:display:72] Rebalance Overview
      +----------------+----------+-----------------------+---------------+--------------+
      | Nodes          | Services | Version               | CPU           | Status       |
      +----------------+----------+-----------------------+---------------+--------------+
      | 172.23.105.164 | kv       | 7.1.0-1380-enterprise | 2.05926670015 | Cluster node |
      | 172.23.105.206 | None     |                       |               | <--- IN ---  |
      | 172.23.105.33  | None     |                       |               | <--- IN ---  |
      | 172.23.105.36  | None     |                       |               | <--- IN ---  |
      | 172.23.106.177 | None     |                       |               | <--- IN ---  |
      +----------------+----------+-----------------------+---------------+--------------+
      

      2. Create buckets/scopes/collections/data

      2021-09-30 11:58:10,828 | test  | INFO    | MainProcess | MainThread | [table_view:display:72] Bucket statistics
      +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+
      | Bucket  | Type      | Storage Backend | Replicas | Durability | TTL | Items    | RAM Quota | RAM Used   | Disk Used  | ARR           |
      +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+
      | bucket1 | couchbase | couchstore      | 2        | none       | 0   | 100000   | 9.77 GiB  | 337.54 MiB | 335.69 MiB | 100           |
      | bucket2 | couchbase | magma           | 2        | none       | 0   | 50000    | 4.88 GiB  | 505.19 MiB | 523.73 MiB | 100           |
      | default | couchbase | magma           | 2        | none       | 0   | 16287500 | 2.50 GiB  | 1.87 GiB   | 24.07 GiB  | 10.1939401381 |
      +---------+-----------+-----------------+----------+------------+-----+----------+-----------+------------+------------+---------------+
      

      3. Graceful failover 2 nodes.

      2021-09-30 11:58:16,898 | test  | INFO    | MainProcess | MainThread | [collections_rebalance:rebalance_operation:570] failing over nodes [ip:172.23.105.36 port:8091 ssh_username:root, ip:172.23.106.177 port:8091 ssh_username:root]
      

      4. Start compaction on all the buckets

      2021-09-30 12:09:33,635 | test  | INFO    | MainProcess | MainThread | [collections_rebalance:compact_all_buckets:237] Starting compaction for each bucket
      

      5. Wait for graceful failover to complete

      2021-09-30 12:09:33,729 | test  | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1846] 172.23.105.36 - Node not part of cluster inactiveFailed
      2021-09-30 12:09:33,729 | test  | WARNING | MainProcess | MainThread | [rest_client:get_nodes:1846] 172.23.106.177 - Node not part of cluster inactiveFailed
      

      6. Wait for compaction to complete

      2021-09-30 12:10:20,905 | test  | INFO    | MainProcess | pool-12-thread-12 | [task:call:5708] bucket1 compaction done: 100%
      2021-09-30 12:13:16,407 | test  | INFO    | MainProcess | pool-12-thread-11 | [task:call:5708] bucket2 compaction done: 100%
      

      7. Rebalance out the failed over nodes. Rebalance completes successfully.
      2021-09-30 12:26:58,538 | test | INFO | MainProcess | pool-12-thread-3 | [table_view:display:72] Rebalance Overview
      -----------------------------------------------------------------------

      Nodes Services Version CPU Status

      -----------------------------------------------------------------------

      172.23.105.36 kv 7.1.0-1380-enterprise 0.701578551741 — OUT --->
      172.23.105.33 kv 7.1.0-1380-enterprise 26.9687580687 Cluster node
      172.23.105.206 kv 7.1.0-1380-enterprise 23.9712857326 Cluster node
      172.23.106.177 kv 7.1.0-1380-enterprise 0.788091068301 — OUT --->
      172.23.105.164 kv 7.1.0-1380-enterprise 21.3883917157 Cluster node

      -----------------------------------------------------------------------

      At this point we notice minidump 6cfb217e-30d5-490b-d1d0cc95-d2089cde.dmp on 172.23.105.206.

      Grep CRITICAL on 172.23.105.206

      [root@s81706 cbcollect_info_ns_1@172.23.105.206_20210930-195606]# grep CRITICAL memcached.log 
      2021-09-30T12:18:44.659622-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      2021-09-30T12:18:44.659677-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): std::bad_alloc
      2021-09-30T12:18:45.484074-07:00 CRITICAL Detected previous crash
      2021-09-30T12:18:45.484131-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.1.0-1380). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/c before terminating.
      2021-09-30T12:18:45.484136-07:00 CRITICAL Stack backtrace of crashed thread:
      2021-09-30T12:18:45.484137-07:00 CRITICAL    #0  /opt/couchbase/bin/memcached() [0x400000+0x69d858]
      2021-09-30T12:18:45.484139-07:00 CRITICAL    #1  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x6ed8ca]
      2021-09-30T12:18:45.484140-07:00 CRITICAL    #2  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x6edc08]
      2021-09-30T12:18:45.484142-07:00 CRITICAL    #3  /lib64/libpthread.so.0() [0x7fe434251000+0xf630]
      2021-09-30T12:18:45.484144-07:00 CRITICAL    #4  /lib64/libc.so.6(gsignal+0x37) [0x7fe431a7f000+0x36387]
      2021-09-30T12:18:45.484145-07:00 CRITICAL    #5  /lib64/libc.so.6(abort+0x148) [0x7fe431a7f000+0x37a78]
      2021-09-30T12:18:45.484147-07:00 CRITICAL    #6  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe432367000+0x9963c]
      2021-09-30T12:18:45.484148-07:00 CRITICAL    #7  /opt/couchbase/bin/memcached() [0x400000+0x6aa91b]
      2021-09-30T12:18:45.484152-07:00 CRITICAL    #8  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe432367000+0xa48f6]
      2021-09-30T12:18:45.484154-07:00 CRITICAL    #9  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe432367000+0xa4961]
      2021-09-30T12:18:45.484155-07:00 CRITICAL    #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe432367000+0xa4bf4]
      2021-09-30T12:18:45.484171-07:00 CRITICAL    #11 /opt/couchbase/bin/memcached() [0x400000+0x129667]
      2021-09-30T12:18:45.484175-07:00 CRITICAL    #12 /opt/couchbase/bin/memcached() [0x400000+0x59e484]
      2021-09-30T12:18:45.484176-07:00 CRITICAL    #13 /opt/couchbase/bin/memcached() [0x400000+0x536484]
      2021-09-30T12:18:45.484209-07:00 CRITICAL    #14 /opt/couchbase/bin/memcached() [0x400000+0x52f790]
      2021-09-30T12:18:45.484210-07:00 CRITICAL    #15 /opt/couchbase/bin/memcached() [0x400000+0x5308c4]
      2021-09-30T12:18:45.484214-07:00 CRITICAL    #16 /opt/couchbase/bin/memcached() [0x400000+0x531236]
      2021-09-30T12:18:45.484216-07:00 CRITICAL    #17 /opt/couchbase/bin/memcached() [0x400000+0x531a0d]
      2021-09-30T12:18:45.484240-07:00 CRITICAL    #18 /opt/couchbase/bin/memcached() [0x400000+0x531f48]
      2021-09-30T12:18:45.484242-07:00 CRITICAL    #19 /opt/couchbase/bin/memcached(_ZN5magma5Magma4Impl11syncKVStoreEtb+0x215) [0x400000+0x4dd5c5]
      2021-09-30T12:18:45.484250-07:00 CRITICAL    #20 /opt/couchbase/bin/memcached(_ZN5magma5Magma4Impl14CompactKVStoreEtNS0_9StoreTypeESt8functionIFSt10unique_ptrINS0_18CompactionCallbackESt14default_deleteIS5_EEtEE+0x27e) [0x400000+0x4ddb5e]
      2021-09-30T12:18:45.484252-07:00 CRITICAL    #21 /opt/couchbase/bin/memcached(_ZN5magma5Magma4Impl14CompactKVStoreEtNS0_9StoreTypeESt8functionIFSt10unique_ptrINS0_18CompactionCallbackESt14default_deleteIS5_EEtEE+0x218) [0x400000+0x4ddaf8]
      2021-09-30T12:18:45.484254-07:00 CRITICAL    #22 /opt/couchbase/bin/memcached(_ZN5magma5Magma4Impl14CompactKVStoreEtRKNS_5SliceES4_St8functionIFSt10unique_ptrINS0_18CompactionCallbackESt14default_deleteIS7_EEtEE+0x65) [0x400000+0x4dddd5]
      2021-09-30T12:18:45.484258-07:00 CRITICAL    #23 /opt/couchbase/bin/memcached(_ZN5magma5Magma14CompactKVStoreEtRKNS_5SliceES3_St8functionIFSt10unique_ptrINS0_18CompactionCallbackESt14default_deleteIS6_EEtEE+0x6d) [0x400000+0x4de0dd]
      2021-09-30T12:18:45.484324-07:00 CRITICAL    #24 /opt/couchbase/bin/memcached() [0x400000+0x45629e]
      2021-09-30T12:18:45.484326-07:00 CRITICAL    #25 /opt/couchbase/bin/memcached() [0x400000+0x44d46d]
      2021-09-30T12:18:45.484351-07:00 CRITICAL    #26 /opt/couchbase/bin/memcached() [0x400000+0x44da06]
      2021-09-30T12:18:45.484353-07:00 CRITICAL    #27 /opt/couchbase/bin/memcached() [0x400000+0x3e9ada]
      2021-09-30T12:18:45.484354-07:00 CRITICAL    #28 /opt/couchbase/bin/memcached() [0x400000+0x3eb0f1]
      2021-09-30T12:18:45.484356-07:00 CRITICAL    #29 /opt/couchbase/bin/memcached() [0x400000+0x3089c6]
      2021-09-30T12:18:45.484357-07:00 CRITICAL    #30 /opt/couchbase/bin/memcached() [0x400000+0x61cf32]
      2021-09-30T12:18:45.484359-07:00 CRITICAL    #31 /opt/couchbase/bin/memcached() [0x400000+0x61a055]
      2021-09-30T12:18:45.484361-07:00 CRITICAL    #32 /opt/couchbase/bin/memcached() [0x400000+0x7649a0]
      2021-09-30T12:18:45.484362-07:00 CRITICAL    #33 /opt/couchbase/bin/memcached() [0x400000+0x74c75a]
      2021-09-30T12:18:45.484364-07:00 CRITICAL    #34 /opt/couchbase/bin/memcached() [0x400000+0x767959]
      2021-09-30T12:18:45.484365-07:00 CRITICAL    #35 /opt/couchbase/bin/memcached() [0x400000+0x619ce4]
      2021-09-30T12:18:45.484367-07:00 CRITICAL    #36 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe432367000+0xcdd40]
      2021-09-30T12:18:45.484368-07:00 CRITICAL    #37 /lib64/libpthread.so.0() [0x7fe434251000+0x7ea5]
      2021-09-30T12:18:45.484370-07:00 CRITICAL    #38 /lib64/libc.so.6(clone+0x6d) [0x7fe431a7f000+0xfe8dd]
      [root@s81706 cbcollect_info_ns_1@172.23.105.206_20210930-195606]# 
      

      bt

      (gdb) bt
      #0  0x00007fe431ab5387 in __bsd_signal (sig=18163, handler=0x5c95) at ../sysdeps/posix/signal.c:50
      #1  0x00007fe431ab6a78 in __GI_abort () at abort.c:79
      #2  0x00007fe43240063c in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:95
      #3  0x0000000000aaa91b in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:88
      #4  0x00007fe43240b8f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
      #5  0x00007fe43240b961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
      #6  0x00007fe43240bbf4 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0xfcdee0 <typeinfo for std::bad_alloc>, dest=0x441600 <_ZNSt9bad_allocD1Ev@plt>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95
      #7  0x0000000000529667 in operator new (count=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/global_new_replacement.cc:65
      #8  0x000000000099e484 in _M_construct<char*> (__end=0xffc6eaf2c680 <Address 0xffc6eaf2c680 out of bounds>, __beg=0x7fe375796340 <Address 0x7fe375796340 out of bounds>, this=0x7fe3a3fec410)
          at /opt/gcc-10.2.0/include/c++/10.2.0/bits/char_traits.h:389
      #9  _M_construct_aux<char*> (__end=0xffc6eaf2c680 <Address 0xffc6eaf2c680 out of bounds>, __beg=0x7fe375796340 <Address 0x7fe375796340 out of bounds>, this=0x7fe3a3fec410) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/basic_string.h:247
      #10 _M_construct<char*> (__end=0xffc6eaf2c680 <Address 0xffc6eaf2c680 out of bounds>, __beg=0x7fe375796340 <Address 0x7fe375796340 out of bounds>, this=0x7fe3a3fec410) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/basic_string.h:266
      #11 basic_string (__str=..., this=0x7fe3a3fec410) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/basic_string.h:451
      #12 GetMeta (this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/tree_snapshot.h:113
      #13 GetMeta (this=0x7fe3a3fec420) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/checkpoint.cc:454
      #14 magma::Checkpoint::String[abi:cxx11]() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/checkpoint.cc:564
      #15 0x0000000000936484 in magma::KVStoreCheckpoint::String[abi:cxx11]() (this=0x7fe3775d1cc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.h:446
      #16 0x000000000092f790 in magma::KVStore::getAllCheckpoints(std::vector<magma::KVStoreCheckpoint, std::allocator<magma::KVStoreCheckpoint> >&) ()
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:1408
      #17 0x00000000009308c4 in magma::KVStore::verifyCheckpoints() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:1469
      #18 0x0000000000931236 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode)::{lambda()#2}::operator()() ()
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:503
      #19 0x0000000000931a0d in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:514
      #20 0x0000000000931f48 in magma::KVStore::FlushMemTables (this=<optimized out>, wal=<optimized out>, flushMode=<optimized out>, blockMode=<optimized out>)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:336
      #21 0x00000000008dd5c5 in magma::Magma::Impl::syncKVStore(unsigned short, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:1133
      #22 0x00000000008ddb5e in magma::Magma::Impl::CompactKVStore(unsigned short, magma::Magma::StoreType, std::function<std::unique_ptr<magma::Magma::CompactionCallback, std::default_delete<magma::Magma::CompactionCallback> > (unsigned short)>) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:746
      #23 0x00000000008ddaf8 in magma::Magma::Impl::CompactKVStore(unsigned short, magma::Magma::StoreType, std::function<std::unique_ptr<magma::Magma::CompactionCallback, std::default_delete<magma::Magma::CompactionCallback> > (unsigned short)>) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:725
      #24 0x00000000008dddd5 in magma::Magma::Impl::CompactKVStore(unsigned short, magma::Slice const&, magma::Slice const&, std::function<std::unique_ptr<magma::Magma::CompactionCallback, std::default_delete<magma::Magma::CompactionCallback> > (unsigned short)>) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:773
      #25 0x00000000008de0dd in magma::Magma::CompactKVStore(unsigned short, magma::Slice const&, magma::Slice const&, std::function<std::unique_ptr<magma::Magma::CompactionCallback, std::default_delete<magma::Magma::CompactionCallback> > (unsigned short)>) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248
      #26 0x000000000085629e in MagmaMemoryTrackingProxy::CompactKVStore(unsigned short, magma::Slice const&, magma::Slice const&, std::function<std::unique_ptr<magma::Magma::CompactionCallback, std::default_delete<magma::Magma::CompactionCallback> > (unsigned short)>) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248
      #27 0x000000000084d46d in MagmaKVStore::compactDBInternal(std::unique_lock<std::mutex>&, std::shared_ptr<CompactionContext>) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62
      #28 0x000000000084da06 in MagmaKVStore::compactDB(std::unique_lock<std::mutex>&, std::shared_ptr<CompactionContext>) ()
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:2032
      #29 0x00000000007e9ada in EPBucket::compactInternal(LockedVBucketPtr&, CompactionConfig&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.h:2599
      #30 0x00000000007eb0f1 in EPBucket::doCompact(Vbid, CompactionConfig&, std::vector<CookieIface const*, std::allocator<CookieIface const*> >&) ()
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:1359
      #31 0x00000000007089c6 in CompactTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/tasks.cc:73
      #32 0x0000000000a1cf32 in GlobalTask::execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:68
      #33 0x0000000000a1a055 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7fe3a3fed840)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:189
      #34 0x0000000000b649a0 in operator() (this=0x7fe3a3fed840) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416
      #35 folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) (this=0x7fe4309e4400, thread=..., 
          task=<unknown type in /usr/lib/debug/opt/couchbase/bin/memcached.debug, CU 0x65a24fb, DIE 0x65e7bd8>)
          at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97
      #36 0x0000000000b4c75a in folly::CPUThreadPoolExecutor::threadRun (this=0x7fe4309e4400, thread=...)
          at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265
      #37 0x0000000000b67959 in __invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (
          __t=<optimized out>, __f=<optimized out>) at /usr/local/include/c++/7.3.0/bits/invoke.h:73
      #38 __invoke<void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__fn=<optimized out>)
          at /usr/local/include/c++/7.3.0/bits/invoke.h:95
      #39 __call<void, 0, 1> (__args=<optimized out>, this=<optimized out>) at /usr/local/include/c++/7.3.0/functional:467
      #40 operator()<> (this=<optimized out>) at /usr/local/include/c++/7.3.0/functional:551
      #41 folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401
      #42 0x0000000000a19ce4 in operator() (this=0x7fe424025b80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47
      ---Type <return> to continue, or q <return> to quit---
      #43 operator() (__closure=0x7fe424025b80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47
      #44 folly::detail::function::FunctionTraits<void ()>::callBig<CBRegisteredThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401
      #45 0x00007fe432434d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80
      #46 0x00007fe434258ea5 in start_thread (arg=0x7fe3a3fff700) at pthread_create.c:307
      #47 0x00007fe431b7d8dd in ioperm () at ../sysdeps/unix/syscall-template.S:81
      #48 0x0000000000000000 in ?? ()
      (gdb) 
      

      cbcollect_info attached. This is a new test being run. So, we don't have a baseline for this test yet.

      Attachments

        1. bt_full.txt
          31 kB
        2. info_threads.txt
          38 kB
        3. Screenshot 2021-10-06 at 08.01.02.png
          Screenshot 2021-10-06 at 08.01.02.png
          40 kB
        4. Screenshot 2021-10-06 at 08.03.52.png
          Screenshot 2021-10-06 at 08.03.52.png
          45 kB
        5. thread_apply_all_bt.txt
          37 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            Balakumaran.Gopal Balakumaran Gopal
            Balakumaran.Gopal Balakumaran Gopal
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty