Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
None
-
6.6.2
-
None
-
6.6.2-9588
-
Untriaged
-
Centos 64-bit
-
1
-
Unknown
Description
Our goal was to repro MB-53183 on 6.6.2-9588 and then upgrade to 6.6.5 MP5 using swap rebalance. Pavan PB was executing following combination of steps on 6.6.2-9588 to repro the issue when we saw a minidump.
Steps to reproduce:
- Create a 6 node cluster 3KV-2index-1query.
- Create about 4 buckets with 2 indexes on each bucket.
- Load the docs until all of the buckets have data greater than memory.
- Kill memcached (either by OOM kill or by kill -9) on two of the nodes. Failover one of the nodes.
- Do a delta/full recovery and rebalance.
172.23.100.15
[root@vm-100-15 logs]# grep CRITICAL memcached.log.0000*
|
memcached.log.000011.txt:2022-08-04T23:47:25.393616-07:00 CRITICAL Breakpad caught a crash (Couchbase version 6.6.2-9588). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/5b55c6a0-18a2-4575-7233cbcf-43a6d962.dmp before terminating.
|
memcached.log.000011.txt:2022-08-04T23:47:25.393641-07:00 CRITICAL Stack backtrace of crashed thread:
|
memcached.log.000011.txt:2022-08-04T23:47:25.394163-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x136c1d]
|
memcached.log.000011.txt:2022-08-04T23:47:25.394174-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x14f74e]
|
memcached.log.000011.txt:2022-08-04T23:47:25.394183-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x14fa64]
|
memcached.log.000011.txt:2022-08-04T23:47:25.394201-07:00 CRITICAL /lib64/libpthread.so.0() [0x7f911e468000+0xf5f0]
|
memcached.log.000011.txt:2022-08-04T23:47:25.394225-07:00 CRITICAL /lib64/libc.so.6(gsignal+0x37) [0x7f911e09a000+0x36337]
|
memcached.log.000011.txt:2022-08-04T23:47:25.394244-07:00 CRITICAL /lib64/libc.so.6(abort+0x148) [0x7f911e09a000+0x37a28]
|
memcached.log.000011.txt:2022-08-04T23:47:25.394249-07:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f9120c4c000+0xb786]
|
memcached.log.000011.txt:2022-08-04T23:47:25.394258-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x99434]
|
memcached.log.000011.txt:2022-08-04T23:47:25.394265-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x6d0cc]
|
memcached.log.000011.txt:2022-08-04T23:47:25.394270-07:00 CRITICAL /opt/couchbase/bin/../lib/libevent_core.so.2.1.8() [0x7f911fd14000+0x17107]
|
memcached.log.000011.txt:2022-08-04T23:47:25.394275-07:00 CRITICAL /opt/couchbase/bin/../lib/libevent_core.so.2.1.8(event_base_loop+0x39f) [0x7f911fd14000+0x1767f]
|
memcached.log.000011.txt:2022-08-04T23:47:25.394282-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x6b209]
|
memcached.log.000011.txt:2022-08-04T23:47:25.394288-07:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f9120c4c000+0x8f17]
|
memcached.log.000011.txt:2022-08-04T23:47:25.394293-07:00 CRITICAL /lib64/libpthread.so.0() [0x7f911e468000+0x7e65]
|
memcached.log.000011.txt:2022-08-04T23:47:25.394322-07:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7f911e09a000+0xfe88d]
|
[root@vm-100-15 logs]#
|
|
Backtrace of 5b55c6a0-18a2-4575-7233cbcf-43a6d962.dmp on 172.23.100.15
(gdb) bt
|
#0 0x00007f911e0d0337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
|
#1 0x00007f911e0d1a28 in __GI_abort () at abort.c:90
|
#2 0x00007f9120c57786 in cb_assert_die (expression=expression@entry=0x58ea20 "iter != connections.conns.end()", file=file@entry=0x58e9c8 "/home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connections.cc", line=line@entry=281)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cbassert.c:49
|
#3 0x0000000000499434 in release_connection (c=0x7f90d3a35000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connections.cc:281
|
#4 run_event_loop (c=0x7f90d3a35000, which=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connections.cc:157
|
#5 0x000000000046d0cc in thread_libevent_process (fd=<optimized out>, arg=0x7f911cc00028) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:338
|
#6 0x00007f911fd2b107 in event_persist_closure (ev=<optimized out>, base=0x2454860) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1580
|
#7 event_process_active_single_queue (base=base@entry=0x2454860, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0, activeq=<optimized out>)
|
at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1639
|
#8 0x00007f911fd2b67f in event_process_active (base=0x2454860) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1738
|
#9 event_base_loop (base=0x2454860, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1961
|
#10 0x000000000046b209 in worker_libevent (arg=0x7f911cc00028) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:218
|
#11 0x00007f9120c54f17 in run (this=0x7f911cc819d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58
|
#12 platform_thread_wrap (arg=0x7f911cc819d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71
|
#13 0x00007f911e46fe65 in start_thread (arg=0x7f9117c41700) at pthread_create.c:307
|
#14 0x00007f911e19888d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
|
(gdb)
|
cbcollect_info and other useful gdb o/p s attached.