Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
None
-
7.0.2
-
Untriaged
-
1
-
Unknown
Description
7.0.2-6522
Test:
-test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
Scale 3
1st iteration
Following crash observed:
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155358-07:00 CRITICAL *** Fatal error encountered during exception handling ***
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155431-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): Monotonic<m> (ActiveStream(vb:769)::lastReadSeqno) invariant failed: new value (25644) breaks invariant on current value (25644)
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155444-07:00 CRITICAL Exception thrown from:
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155495-07:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0xb2b90]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155536-07:00 CRITICAL #1 /opt/couchbase/bin/memcached() [0x400000+0xce10c]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155594-07:00 CRITICAL #2 /opt/couchbase/bin/memcached() [0x400000+0x481279]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155616-07:00 CRITICAL #3 /opt/couchbase/bin/memcached() [0x400000+0x484207]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155637-07:00 CRITICAL #4 /opt/couchbase/bin/memcached() [0x400000+0x4849cc]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155678-07:00 CRITICAL #5 /opt/couchbase/bin/memcached() [0x400000+0x484ae4]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155358-07:00 CRITICAL *** Fatal error encountered during exception handling ***
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155431-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): Monotonic<m> (ActiveStream(vb:769)::lastReadSeqno) invariant failed: new value (25644) breaks invariant on current value (25644)
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155444-07:00 CRITICAL Exception thrown from:
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155495-07:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0xb2b90]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155536-07:00 CRITICAL #1 /opt/couchbase/bin/memcached() [0x400000+0xce10c]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155594-07:00 CRITICAL #2 /opt/couchbase/bin/memcached() [0x400000+0x481279]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155616-07:00 CRITICAL #3 /opt/couchbase/bin/memcached() [0x400000+0x484207]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155637-07:00 CRITICAL #4 /opt/couchbase/bin/memcached() [0x400000+0x4849cc]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155678-07:00 CRITICAL #5 /opt/couchbase/bin/memcached() [0x400000+0x484ae4]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155693-07:00 CRITICAL #6 /opt/couchbase/bin/memcached() [0x400000+0x490219]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.156331-07:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x2a6e57]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.156377-07:00 CRITICAL #8 /opt/couchbase/bin/memcached() [0x400000+0x2a1fa0]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.156405-07:00 CRITICAL #9 /opt/couchbase/bin/memcached() [0x400000+0x711f30]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.156433-07:00 CRITICAL #10 /opt/couchbase/bin/memcached() [0x400000+0x6f9cea]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.156467-07:00 CRITICAL #11 /opt/couchbase/bin/memcached() [0x400000+0x714ee9]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.156487-07:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0x2a067d]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.156603-07:00 CRITICAL #13 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f467f349000+0xcdd40]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.156612-07:00 CRITICAL #14 /lib64/libpthread.so.0() [0x7f4680ff3000+0x7e65]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.156655-07:00 CRITICAL #15 /lib64/libc.so.6(clone+0x6d) [0x7f467ea61000+0xfe88d]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.341814-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.2-6522). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/99b3493f-b170-4ea4-4d65f9be-9a4ca054.dmp before terminating.
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.341858-07:00 CRITICAL Stack backtrace of crashed thread:
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342083-07:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x676753]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342139-07:00 CRITICAL #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x69776a]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342183-07:00 CRITICAL #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x697aa8]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342194-07:00 CRITICAL #3 /lib64/libpthread.so.0() [0x7f4680ff3000+0xf5f0]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342242-07:00 CRITICAL #4 /lib64/libc.so.6(gsignal+0x37) [0x7f467ea61000+0x36337]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342286-07:00 CRITICAL #5 /lib64/libc.so.6(abort+0x148) [0x7f467ea61000+0x37a28]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342358-07:00 CRITICAL #6 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f467f349000+0x9963c]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342396-07:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x68438f]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342416-07:00 CRITICAL #8 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f467f349000+0xa48f6]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342449-07:00 CRITICAL #9 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f467f349000+0xa4961]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342483-07:00 CRITICAL #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f467f349000+0xa4bf4]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342499-07:00 CRITICAL #11 /opt/couchbase/bin/memcached() [0x400000+0xb2c95]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342528-07:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0xce10c]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342620-07:00 CRITICAL #13 /opt/couchbase/bin/memcached() [0x400000+0x481279]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342676-07:00 CRITICAL #14 /opt/couchbase/bin/memcached() [0x400000+0x484207]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342773-07:00 CRITICAL #15 /opt/couchbase/bin/memcached() [0x400000+0x4849cc]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342804-07:00 CRITICAL #16 /opt/couchbase/bin/memcached() [0x400000+0x484ae4]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342837-07:00 CRITICAL #17 /opt/couchbase/bin/memcached() [0x400000+0x490219]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342876-07:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0x2a6e57]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342949-07:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0x2a1fa0]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342978-07:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0x711f30]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.343005-07:00 CRITICAL #21 /opt/couchbase/bin/memcached() [0x400000+0x6f9cea]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.343028-07:00 CRITICAL #22 /opt/couchbase/bin/memcached() [0x400000+0x714ee9]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.343046-07:00 CRITICAL #23 /opt/couchbase/bin/memcached() [0x400000+0x2a067d]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.343087-07:00 CRITICAL #24 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f467f349000+0xcdd40]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.343100-07:00 CRITICAL #25 /lib64/libpthread.so.0() [0x7f4680ff3000+0x7e65]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.343630-07:00 CRITICAL #26 /lib64/libc.so.6(clone+0x6d) [0x7f467ea61000+0xfe88d]
|
At the same time, the following service exit is observed as well:
172.23.120.73 : crash
|
/opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-08-17T20:49:09.734-07:00,ns_1@172.23.120.73:<0.21752.0>:ns_log:crash_consumption_loop:63]Service 'memcached' exited with status 134. Restarting. Messages:
|
|
172.23.120.77 : crash
|
/opt/couchbase/var/lib/couchbase/logs/info.log.4.gz:[user:info,2021-08-17T20:48:58.097-07:00,ns_1@172.23.120.77:<0.13228.0>:ns_log:crash_consumption_loop:63]Service 'memcached' exited with status 134. Restarting. Messages:
|
Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.106.134.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.106.136.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.58.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.73.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.74.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.75.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.81.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.86.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.121.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.24.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.25.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.26.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.31.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.32.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.33.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.14.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.243.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.254.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.48.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.105.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.110.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.112.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.149.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.150.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.241.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.74.zip