Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.0
-
7.1.0-2075
-
Untriaged
-
-
1
-
Unknown
-
KV 2022-Feb
Description
Steps:
1. Take a backup of 5TB data cluster. Data in 10 collections, 55M items in each collection.
|
CLI |
/opt/couchbase/bin/cbbackupmgr config -a /data/bk -r bk --include-data GleamBookUsers0._default.VolumeCollection0,GleamBookUsers0._default.VolumeCollection1,GleamBookUsers0._default.VolumeCollection2,GleamBookUsers0._default.VolumeCollection3,GleamBookUsers0._default.VolumeCollection4,GleamBookUsers0._default.VolumeCollection5,GleamBookUsers0._default.VolumeCollection6,GleamBookUsers0._default.VolumeCollection7,GleamBookUsers0._default.VolumeCollection8,GleamBookUsers0._default.VolumeCollection9
|
|
|
/opt/couchbase/bin/cbbackupmgr backup -a /data/bk -r bk --resume --cluster couchbase://172.23.110.69 -u Administrator -p password
|
|
Backup Result |
Copied all data in 11h50m33s (Avg. 110.12MiB/Sec) 5500000000 items / 4.48TiB
|
[====================================================================================================================================================================================================================] 100.00%
|
|
|
| Transfer
|
| --------
|
| Status | Avg Transfer Rate | Started At | Finished At | Duration |
|
| Succeeded | 110.12MiB/s | Wed, 26 Jan 2022 07:19:01 -0800 | Wed, 26 Jan 2022 19:09:34 -0800 | 11h50m33s |
|
|
|
| Bucket
|
| ------
|
| Name | Status | Transferred | Avg Transfer Rate | Started At | Finished At | Duration |
|
| GleamBookUsers0 | Succeeded | 4.48TiB | 110.13MiB/s | Wed, 26 Jan 2022 07:19:02 -0800 | Wed, 26 Jan 2022 19:09:34 -0800 | 11h50m32s |
|
|
|
| Mutations | Deletions | Expirations |
|
| --------- | --------- | ----------- |
|
| Received | Errored | Skipped | Received | Errored | Skipped | Received | Errored | Skipped |
|
| 5500000000 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
2. Flush the bucket and restore data:
|
CLI |
/opt/couchbase/bin/cbbackupmgr restore -a /data/bk -r bk --threads 48 --purge --cluster couchbase://172.23.110.69 -u Administrator -p password
|
|
Restore Result |
[========================================================================================================================================================================================= ] 87.31%
|
Error restoring cluster: EOF
|
|
|
| Transfer
|
| --------
|
| Status | Avg Transfer Rate | Started At | Finished At | Duration |
|
| Failed | 209.87MiB/s | Wed, 26 Jan 2022 23:09:37 -0800 | Thu, 27 Jan 2022 04:44:09 -0800 | 5h34m32s |
|
|
|
| Bucket
|
| ------
|
| Name | Status | Transferred | Avg Transfer Rate | Started At | Finished At | Duration |
|
| GleamBookUsers0 | Failed | 4.02TiB | 209.92MiB/s | Wed, 26 Jan 2022 23:09:40 -0800 | Thu, 27 Jan 2022 04:44:09 -0800 | 5h34m28s |
|
|
|
| Mutations | Deletions | Expirations |
|
| --------- | --------- | ----------- |
|
| Received | Errored | Skipped | Received | Errored | Skipped | Received | Errored | Skipped |
|
| 4932194128 | 487 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
3. Observe that during restore memcahced crashed on 172.23.110.127
Logs from memcached on 172.23.110.127
[root@cen-sb21 ~]# grep CRITICAL /opt/couchbase/var/lib/couchbase/logs/memcached.log.0001*
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:15.732630-08:00 CRITICAL *** Fatal error encountered during exception handling ***
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:15.733585-08:00 CRITICAL Caught unhandled std::exception-derived exception. what(): AtomicQueue::initialize: exceeded maximum allowed threads
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103154-08:00 CRITICAL Detected previous crash
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103241-08:00 CRITICAL Breakpad caught a crash (Couchbase version 7.1.0-2075). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/57154c42-2e29-4027-3b0b69ac-1e047fdd.dmp before terminating.
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103248-08:00 CRITICAL Stack backtrace of crashed thread:
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103249-08:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x728ee8]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103251-08:00 CRITICAL #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x77a6ba]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103252-08:00 CRITICAL #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x77a9f8]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103253-08:00 CRITICAL #3 /lib64/libpthread.so.0() [0x7f566e7b4000+0xf630]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103254-08:00 CRITICAL #4 /lib64/libc.so.6(gsignal+0x37) [0x7f566bffe000+0x36387]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103255-08:00 CRITICAL #5 /lib64/libc.so.6(abort+0x148) [0x7f566bffe000+0x37a78]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103256-08:00 CRITICAL #6 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f566c8e6000+0x9963c]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103261-08:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x7333bb]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103262-08:00 CRITICAL #8 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f566c8e6000+0xa48f6]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103263-08:00 CRITICAL #9 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f566c8e6000+0xa4961]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103263-08:00 CRITICAL #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f566c8e6000+0xa4bf4]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103264-08:00 CRITICAL #11 /opt/couchbase/bin/memcached() [0x400000+0xbfdd8]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103265-08:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0x4d7044]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103266-08:00 CRITICAL #13 /opt/couchbase/bin/memcached() [0x400000+0x4e145c]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103266-08:00 CRITICAL #14 /opt/couchbase/bin/memcached() [0x400000+0x4d7fe2]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103267-08:00 CRITICAL #15 /opt/couchbase/bin/memcached() [0x400000+0x3de06b]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103267-08:00 CRITICAL #16 /opt/couchbase/bin/memcached() [0x400000+0x3251d4]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103268-08:00 CRITICAL #17 /opt/couchbase/bin/memcached() [0x400000+0x41163e]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103269-08:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0x32786d]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103269-08:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0x427d48]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103275-08:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0x4286ff]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103276-08:00 CRITICAL #21 /opt/couchbase/bin/memcached() [0x400000+0x2da910]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103277-08:00 CRITICAL #22 /opt/couchbase/bin/memcached() [0x400000+0x2db260]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103277-08:00 CRITICAL #23 /opt/couchbase/bin/memcached() [0x400000+0x6a6662]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103278-08:00 CRITICAL #24 /opt/couchbase/bin/memcached() [0x400000+0x6a3875]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103279-08:00 CRITICAL #25 /opt/couchbase/bin/memcached() [0x400000+0x7fa660]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103279-08:00 CRITICAL #26 /opt/couchbase/bin/memcached() [0x400000+0x7e241a]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103283-08:00 CRITICAL #27 /opt/couchbase/bin/memcached() [0x400000+0x7fd619]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103284-08:00 CRITICAL #28 /opt/couchbase/bin/memcached() [0x400000+0x6a3504]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103285-08:00 CRITICAL #29 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f566c8e6000+0xcdd40]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103286-08:00 CRITICAL #30 /lib64/libpthread.so.0() [0x7f566e7b4000+0x7ea5]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000145.txt:2022-01-27T04:43:37.103287-08:00 CRITICAL #31 /lib64/libc.so.6(clone+0x6d) [0x7f566bffe000+0xfe8dd]
|
Bt full on 172.23.110.127
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
|
#0 0x00007f566c034387 in raise () from /lib64/libc.so.6
|
warning: File "/opt/couchbase/lib/libstdc++.so.6.0.28-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".
|
#0 0x00007f566c034387 in raise () from /lib64/libc.so.6
|
#1 0x00007f566c035a78 in abort () from /lib64/libc.so.6
|
#2 0x00007f566c97f63c in __gnu_cxx::__verbose_terminate_handler() [clone .cold] () from /opt/couchbase/bin/../lib/libstdc++.so.6
|
#3 0x0000000000b333bb in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:88
|
#4 0x00007f566c98a8f6 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6
|
#5 0x00007f566c98a961 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6
|
#6 0x00007f566c98abf4 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6
|
#7 0x00000000004bfdd8 in initialize (this=0x7f566408e038) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:779
|
#8 swapQueue (newQueue=0x0, this=0x7f566408e038) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/atomicqueue.h:125
|
#9 push (value=<optimized out>, this=0x7f566408e038) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/atomicqueue.h:51
|
#10 ConnMap::addConnectionToPending (this=0x7f566408e000, conn=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/connmap.cc:134
|
#11 0x00000000008d7044 in DcpConsumer::scheduleNotify() () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/shared_ptr_base.h:254
|
#12 0x00000000008e145c in PassiveStream::notifyStreamReady() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/passive_stream.cc:1237
|
#13 0x00000000008d7fe2 in DcpConsumer::seqnoAckStream(Vbid, long) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/consumer.cc:1173
|
#14 0x00000000007de06b in DcpConnMap::seqnoAckVBPassiveStream(Vbid, long) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/dcpconnmap.cc:430
|
#15 0x00000000007251d4 in operator() (__args#1=<optimized out>, __args#0=..., this=<optimized out>) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248
|
#16 VBucket::sendSeqnoAck(long) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc:1051
|
#17 0x000000000081163e in sendSeqnoAck (this=0x7f545e968220) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/durability/passive_durability_monitor.cc:283
|
#18 PassiveDurabilityMonitor::notifyLocalPersistence() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/durability/passive_durability_monitor.cc:267
|
#19 0x000000000072786d in VBucket::notifyPersistenceToDurabilityMonitor() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc:4061
|
#20 0x0000000000827d48 in EPBucket::flushVBucket_UNLOCKED(LockedVBucketPtr) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.h:2641
|
#21 0x00000000008286ff in EPBucket::flushVBucket(Vbid) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:374
|
#22 0x00000000006da910 in Flusher::flushVB (this=0x7f55d07bf900) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:285
|
#23 0x00000000006db260 in Flusher::step(GlobalTask*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:200
|
#24 0x0000000000aa6662 in GlobalTask::execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:68
|
#25 0x0000000000aa3875 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f5617fed840) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:189
|
#26 0x0000000000bfa660 in operator() (this=0x7f5617fed840) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416
|
#27 folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f5668d33800, thread=..., task=task@entry=<unknown type in /usr/lib/debug/opt/couchbase/bin/memcached-7.1.0-2075.x86_64.debug, CU 0xa37b680, DIE 0xa3ff5b5>) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97
|
#28 0x0000000000be241a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f5668d33800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265
|
#29 0x0000000000bfd619 in __invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__t=<optimized out>, __f=<optimized out>) at /usr/local/include/c++/7.3.0/bits/invoke.h:73
|
#30 __invoke<void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__fn=<optimized out>) at /usr/local/include/c++/7.3.0/bits/invoke.h:95
|
#31 __call<void, 0, 1> (__args=<optimized out>, this=<optimized out>) at /usr/local/include/c++/7.3.0/functional:467
|
#32 operator()<> (this=<optimized out>) at /usr/local/include/c++/7.3.0/functional:551
|
#33 folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401
|
#34 0x0000000000aa3504 in operator() (this=0x7f5668d9a5c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47
|
#35 operator() (__closure=0x7f5668d9a5c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47
|
#36 folly::detail::function::FunctionTraits<void ()>::callBig<CBRegisteredThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401
|
#37 0x00007f566c9b3d40 in execute_native_thread_routine () from /opt/couchbase/bin/../lib/libstdc++.so.6
|
#38 0x00007f566e7bbea5 in start_thread () from /lib64/libpthread.so.0
|
#39 0x00007f566c0fc8dd in clone () from /lib64/libc.so.6
|
