Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
7.0.0-4200
-
Triaged
-
-
1
-
Unknown
-
KV-Engine 2021-Jan
Description
Steps:
- Create a 15 node cluster
- Create required buckets and collections.
- Create 1000000 items sequentially
- Create 1000000 random keys
- Update 2000000 random keys to create 50 percent fragmentation
- Rebalance in with Loading of docs
- Rebalance Out with Loading of docs
- Rebalance In_Out with Loading of docs
- Swap with Loading of docs
- Failover a node and RebalanceOut that node with loading in parallel
- Failover a node and FullRecovery that node with loading in parallel - Failover -> recovery -> rebalance successful. But memcached is crashed after that during data load.
172.23.120.170:
Stack Trace of ac3ed14a-caeb-4025-63f4bfab-6583d59c.dmp
rm -rf /opt/couchbase/var/lib/couchbase/crash/ac3ed14a-caeb-4025-63f4bfab-6583d59c.core |
//opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/ac3ed14a-caeb-4025-63f4bfab-6583d59c.dmp > /opt/couchbase/var/lib/couchbase/crash/ac3ed14a-caeb-4025-63f4bfab-6583d59c.core
|
gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/ac3ed14a-caeb-4025-63f4bfab-6583d59c.core -ex bt full -ex quit |
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
|
#0 0x00007fc7c9845387 in raise () from /lib64/libc.so.6 |
#0 0x00007fc7c9845387 in raise () from /lib64/libc.so.6 |
#1 0x00007fc7c9846a78 in abort () from /lib64/libc.so.6 |
#2 0x00007fc7ca3a3195 in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/vterminate.cc:95 |
#3 0x0000000000555472 in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:86 |
#4 0x00007fc7ca3a0f86 in __cxxabiv1::__terminate (handler=<optimized out>) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:47 |
#5 0x00007fc7ca39ff49 in __cxa_call_terminate (ue_header=ue_header@entry=0x7fc720000920) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_call.cc:54 |
#6 0x00007fc7ca3a08b8 in __cxxabiv1::__gxx_personality_v0 (version=<optimized out>, actions=6, exception_class=5138137972254386944, ue_header=0x7fc720000920, context=<optimized out>) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/eh_personality.cc:676 |
#7 0x00007fc7c9e08eb3 in _Unwind_RaiseException_Phase2 (exc=exc@entry=0x7fc720000920, context=context@entry=0x7fc7337e49b0) at /tmp/deploy/objdir/../gcc-7.3.0/libgcc/unwind.inc:62 |
#8 0x00007fc7c9e096de in _Unwind_Resume (exc=exc@entry=0x7fc720000920) at /tmp/deploy/objdir/../gcc-7.3.0/libgcc/unwind.inc:230 |
#9 0x00007fc7cddd4df5 in ~__shared_count (this=<optimized out>, __in_chrg=<optimized out>) at /usr/local/include/c++/7.3.0/bits/shared_ptr_base.h:684 |
#10 ~__shared_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/local/include/c++/7.3.0/bits/shared_ptr_base.h:1123 |
#11 ~shared_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/local/include/c++/7.3.0/bits/shared_ptr.h:93 |
#12 KVBucket::deleteExpiredItem (this=<optimized out>, it=..., startTime=1610543460, source=Pager) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:594 |
#13 0x00007fc7cddc907f in KVBucket::deleteExpiredItems (this=0x7fc7bc29a000, itms=..., source=Pager) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:622 |
#14 0x00007fc7cddeb8bd in PagingVisitor::update (this=this@entry=0x7fc7703da4c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/paging_visitor.cc:292 |
#15 0x00007fc7cddeba93 in PagingVisitor::visitBucket (this=0x7fc7703da4c0, vb=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/paging_visitor.cc:214 |
#16 0x00007fc7cddcb299 in VBCBAdaptor::run (this=0x7fc7b03a0f90) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:2386 |
#17 0x00007fc7cddb0343 in GlobalTask::execute (this=0x7fc7b03a0f90) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:73 |
#18 0x00007fc7cddaa432 in operator() (__closure=0x7fc7337e50c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:167 |
#19 folly::detail::function::FunctionTraits<void ()>::callSmall<FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:387 |
#20 0x00007fc7cdf27b86 in operator() (this=0x7fc7337e50c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 |
#21 folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) (this=0x7fc7c6036400, thread=..., task=<unknown type in /usr/lib/debug/opt/couchbase/lib/libep.so.debug, CU 0x30665d8, DIE 0x30aca29>) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:98 |
#22 0x00007fc7cdf1007a in folly::CPUThreadPoolExecutor::threadRun (this=0x7fc7c6036400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 |
#23 0x00007fc7cdf2aca9 in __invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__t=<optimized out>, __f=<optimized out>) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 |
#24 __invoke<void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__fn=<optimized out>) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 |
#25 __call<void, 0, 1> (__args=<optimized out>, this=<optimized out>) at /usr/local/include/c++/7.3.0/functional:467 |
#26 operator()<> (this=<optimized out>) at /usr/local/include/c++/7.3.0/functional:551 |
#27 folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 |
#28 0x00007fc7cdda8873 in operator() (this=0x7fc7c5d46f00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 |
#29 operator() (__closure=0x7fc7c5d46ef0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 |
#30 folly::detail::function::FunctionTraits<void ()>::callBig<folly::PriorityThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 |
#31 0x00007fc7ca3cbdcf in std::execute_native_thread_routine (__p=0x7fc7c85780f0) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 |
#32 0x00007fc7c9be4ea5 in start_thread () from /lib64/libpthread.so.0 |
#33 0x00007fc7c990d8dd in clone () from /lib64/libc.so.6 |
memcahced:
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt:2021-01-13T05:11:01.443383-08:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc:3654
|