Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
None
-
7.1.0
-
7.1.0-1049
-
Untriaged
-
-
1
-
Unknown
Description
|
QE Test |
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P "args=-i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction -t volumetests.Magma.volume.SystemTestMagma,nodes_init=2,replicas=2,skip_cleanup=True,num_items=500000000,doc_size=256,bucket_type=membase,compression_mode=off,iterations=1,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=info,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=True,randomize_value=True,assert_crashes_on_load=True,maxttl=60,num_collections=20,doc_ops=expiry,durability=Majority,fragmentation=20,pc=2,crashes=10,enable_dp=true,sdk_client_pool=True -m rest"
|
Steps:
1. Create a 2 node cluster
2. Start an expiry load and wait for 300s
3. Rebalance in 1 node and crash memcached at 20% rebalance progress
4. Resume rebalance. Rebalance failed due to memcahced crashed at 16%.
Error in memcahced logs:
|
172.23.110.64 |
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt:2021-07-01T21:18:59.070241-07:00 CRITICAL *** Fatal error encountered during exception handling ***
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt:2021-07-01T21:18:59.070285-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): ThrowExceptionUnderflowPolicy current:0 arg:-4914
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt:2021-07-01T21:18:59.070289-07:00 CRITICAL Exception thrown from:
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt:2021-07-01T21:18:59.070322-07:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x2e30ae]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt:2021-07-01T21:18:59.070338-07:00 CRITICAL #1 /opt/couchbase/bin/memcached() [0x400000+0xdb0fa]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt:2021-07-01T21:18:59.070356-07:00 CRITICAL #2 /opt/couchbase/bin/memcached() [0x400000+0x47ea82]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt:2021-07-01T21:18:59.070373-07:00 CRITICAL #3 /opt/couchbase/bin/memcached() [0x400000+0x488142]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt:2021-07-01T21:18:59.070387-07:00 CRITICAL #4 /opt/couchbase/bin/memcached() [0x400000+0x489e23]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt:2021-07-01T21:18:59.070400-07:00 CRITICAL #5 /opt/couchbase/bin/memcached() [0x400000+0x48b9a1]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt:2021-07-01T21:18:59.070412-07:00 CRITICAL #6 /opt/couchbase/bin/memcached() [0x400000+0x48bf84]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt:2021-07-01T21:18:59.070433-07:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x600be2]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt:2021-07-01T21:18:59.070454-07:00 CRITICAL #8 /opt/couchbase/bin/memcached() [0x400000+0x5fdd05]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt:2021-07-01T21:18:59.070478-07:00 CRITICAL #9 /opt/couchbase/bin/memcached() [0x400000+0x759a90]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt:2021-07-01T21:18:59.070501-07:00 CRITICAL #10 /opt/couchbase/bin/memcached() [0x400000+0x74184a]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt:2021-07-01T21:18:59.070522-07:00 CRITICAL #11 /opt/couchbase/bin/memcached() [0x400000+0x75ca49]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt:2021-07-01T21:18:59.070544-07:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0x5fd884]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt:2021-07-01T21:18:59.070582-07:00 CRITICAL #13 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe3a27e0000+0xcdd40]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt:2021-07-01T21:18:59.070589-07:00 CRITICAL #14 /lib64/libpthread.so.0() [0x7fe3a46c2000+0x7ea5]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000037.txt:2021-07-01T21:18:59.070620-07:00 CRITICAL #15 /lib64/libc.so.6(clone+0x6d) [0x7fe3a1ef8000+0xfe8dd]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391605-07:00 CRITICAL Detected previous crash
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391641-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.1.0-1049). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/afe1d42b-2fa0-4496-1cc2f68e-5d759237.dmp before terminating.
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391642-07:00 CRITICAL Stack backtrace of crashed thread:
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391643-07:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x688ab8]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391643-07:00 CRITICAL #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x6e100a]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391644-07:00 CRITICAL #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x6e1348]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391644-07:00 CRITICAL #3 /lib64/libpthread.so.0() [0x7fe3a46c2000+0xf630]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391645-07:00 CRITICAL #4 /lib64/libc.so.6(gsignal+0x37) [0x7fe3a1ef8000+0x36387]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391645-07:00 CRITICAL #5 /lib64/libc.so.6(abort+0x148) [0x7fe3a1ef8000+0x37a78]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391646-07:00 CRITICAL #6 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe3a27e0000+0x9963c]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391646-07:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x6990db]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391646-07:00 CRITICAL #8 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe3a27e0000+0xa48f6]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391647-07:00 CRITICAL #9 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe3a27e0000+0xa4961]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391647-07:00 CRITICAL #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe3a27e0000+0xa4bf4]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391647-07:00 CRITICAL #11 /opt/couchbase/bin/memcached() [0x400000+0x2e31c2]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391647-07:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0xdb0fa]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391648-07:00 CRITICAL #13 /opt/couchbase/bin/memcached() [0x400000+0x47ea82]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391648-07:00 CRITICAL #14 /opt/couchbase/bin/memcached() [0x400000+0x488142]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391651-07:00 CRITICAL #15 /opt/couchbase/bin/memcached() [0x400000+0x489e23]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391652-07:00 CRITICAL #16 /opt/couchbase/bin/memcached() [0x400000+0x48b9a1]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391652-07:00 CRITICAL #17 /opt/couchbase/bin/memcached() [0x400000+0x48bf84]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391652-07:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0x600be2]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391652-07:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0x5fdd05]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391653-07:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0x759a90]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391653-07:00 CRITICAL #21 /opt/couchbase/bin/memcached() [0x400000+0x74184a]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391653-07:00 CRITICAL #22 /opt/couchbase/bin/memcached() [0x400000+0x75ca49]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391653-07:00 CRITICAL #23 /opt/couchbase/bin/memcached() [0x400000+0x5fd884]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391654-07:00 CRITICAL #24 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe3a27e0000+0xcdd40]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391654-07:00 CRITICAL #25 /lib64/libpthread.so.0() [0x7fe3a46c2000+0x7ea5]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000038.txt:2021-07-01T21:19:01.391654-07:00 CRITICAL #26 /lib64/libc.so.6(clone+0x6d) [0x7fe3a1ef8000+0xfe8dd]
|
Stack Trace from afe1d42b-2fa0-4496-1cc2f68e-5d759237.dmp
gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/afe1d42b-2fa0-4496-1cc2f68e-5d759237.core -ex bt full -ex quit |
|
|
|
|
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
|
#0 0x00007fe3a1f2e387 in raise () from /lib64/libc.so.6 |
warning: File "/opt/couchbase/lib/libstdc++.so.6.0.28-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py". |
#0 0x00007fe3a1f2e387 in raise () from /lib64/libc.so.6 |
#1 0x00007fe3a1f2fa78 in abort () from /lib64/libc.so.6 |
#2 0x00007fe3a287963c in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:95 |
#3 0x0000000000a990db in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:88 |
#4 0x00007fe3a28848f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48 |
#5 0x00007fe3a2884961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58 |
#6 0x00007fe3a2884bf4 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7fe2e8000940, tinfo=tinfo@entry=0xf9df80 <typeinfo for boost::exception_detail::error_info_injector<std::underflow_error>>, dest=dest@entry=0x6df910 <boost::exception_detail::error_info_injector<std::underflow_error>::~error_info_injector()>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95 |
#7 0x00000000006e31c2 in cb::throwWithTrace<std::underflow_error> (exception=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/boost.exploded/include/boost/exception/info.hpp:129 |
#8 0x00000000004db0fa in cb::ThrowExceptionUnderflowPolicy<unsigned long>::underflow (current=0, arg=-4914, desired=<optimized out>, this=0x7fe1762d1c78) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:779 |
#9 0x000000000087ea82 in store (desired=18446744073709546702, this=0x7fe1762d1c78) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_mutex.h:104 |
#10 NonNegativeCounter (initial=18446744073709546702, this=0x7fe1762d1c78) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/non_negative_counter.h:81 |
#11 _M_construct<unsigned long&> (this=0x7fe1762d1c78) at /opt/gcc-10.2.0/include/c++/10.2.0/optional:419 |
#12 operator=<long unsigned int&> (__u=<synthetic pointer>, __u=<synthetic pointer>, this=0x7fe1762d1c78) at /opt/gcc-10.2.0/include/c++/10.2.0/optional:794 |
#13 setBackfillRemaining_UNLOCKED (value=18446744073709546702, this=0x7fe1762d1b10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.cc:642 |
#14 ActiveStream::setBackfillRemaining(unsigned long) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.cc:638 |
#15 0x0000000000888142 in DCPBackfillBySeqnoDisk::create() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_by_seqno_disk.cc:152 |
#16 0x0000000000889e23 in DCPBackfillDisk::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_disk.cc:131 |
#17 0x000000000088b9a1 in BackfillManager::backfill() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:333 |
#18 0x000000000088bf84 in BackfillManagerTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:71 |
#19 0x0000000000a00be2 in GlobalTask::execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:68 |
#20 0x00000000009fdd05 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7fe2fefeb540) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:189 |
#21 0x0000000000b59a90 in operator() (this=0x7fe2fefeb540) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 |
#22 folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) (this=0x7fe3827a4000, thread=..., task=<unknown type in /usr/lib/debug/opt/couchbase/bin/memcached.debug, CU 0x64a4243, DIE 0x64e9ab1>) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 |
#23 0x0000000000b4184a in folly::CPUThreadPoolExecutor::threadRun (this=0x7fe3827a4000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 |
#24 0x0000000000b5ca49 in __invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__t=<optimized out>, __f=<optimized out>) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 |
#25 __invoke<void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__fn=<optimized out>) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 |
#26 __call<void, 0, 1> (__args=<optimized out>, this=<optimized out>) at /usr/local/include/c++/7.3.0/functional:467 |
#27 operator()<> (this=<optimized out>) at /usr/local/include/c++/7.3.0/functional:551 |
#28 folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 |
#29 0x00000000009fd884 in operator() (this=0x7fe39eb6b000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 |
#30 operator() (__closure=0x7fe39eb6b000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 |
#31 folly::detail::function::FunctionTraits<void ()>::callBig<CBRegisteredThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 |
#32 0x00007fe3a28add40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 |
#33 0x00007fe3a46c9ea5 in start_thread () from /lib64/libpthread.so.0 |
#34 0x00007fe3a1ff68dd in clone () from /lib64/libc.so.6 |
Attachments
Issue Links
- duplicates
-
-
- Closed
-