Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.6.2
-
Enterprise Edition 7.6.2 build 3721
-
Untriaged
-
Linux x86_64
-
0
-
Yes
Description
Script to Repro
./sequoia -client 172.23.110.181:2375 -provider file:debian_pine.yml -test tests/integration/7.6/test_7.6.yml -scope tests/integration/7.6/scope_7.6_magma.yml -scale 2 -repeat 0 -log_level 0 -version 7.6.2-3721 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=1209600 -show_topology=true
|
Saw the following panic after 7 days of system test run.
172.23.96.111: memcached
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.541987-07:00 CRITICAL [(bucket7) magma_7/kvstore-615/rev-000000001/seqIndex]TableID: 457 out of order
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.542005-07:00 CRITICAL [(bucket7) magma_7/kvstore-615/rev-000000001/seqIndex]LSMTree
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.543697-07:00 CRITICAL [(bucket7) magma_7/kvstore-615/rev-000000001/seqIndex]Tree Before Compaction : TreeSnapshot : (bucket7) magma_7/kvstore-615/rev-000000001/seqIndex
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.545107-07:00 CRITICAL [(bucket7) magma_7/kvstore-615/rev-000000001/seqIndex]Tree After Compaction : TreeSnapshot : (bucket7) magma_7/kvstore-615/rev-000000001/seqIndex
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.545312-07:00 CRITICAL [(bucket7) magma_7/kvstore-615/rev-000000001/seqIndex]Source Level List : | ID:/data/couchbase/bucket7/magma.7/kvstore-615/rev-000000001/seqIndex/sstable.449.data Range:({"Seqno":22753,"op":"delta","sn":24202}, {"Seqno":24541,"op":"delta","sn":27943}) Size:32768 LogicalSize:1505517 DataBlocksSize:35777 DataBlocksCompressedSize:24067 IndexBlocksCompressedSize:219 AvgDataBlockSize:2674.11 CompressionRatio:1.48656 FileSize:32768 ItemCount:1636 State:Deleted HistoryLowTimestamp:0 HistoryHighTimestamp:0 | | ID:/data/couchbase/bucket7/magma.7/kvstore-615/rev-000000001/seqIndex/sstable.451.data Range:({"Seqno":24544,"op":"delta","sn":27966}, {"Seqno":24545,"op":"delta","sn":27969}) Size:8192 LogicalSize:644 DataBlocksSize:63 DataBlocksCompressedSize:63 IndexBlocksCompressedSize:42 AvgDataBlockSize:63 CompressionRatio:1 FileSize:8192 ItemCount:2 State:Deleted HistoryLowTimestamp:0 HistoryHighTimestamp:0 |
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.545489-07:00 CRITICAL [(bucket7) magma_7/kvstore-615/rev-000000001/seqIndex]Destination Level List :
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.556003-07:00 CRITICAL *** Fatal error encountered during exception handling ***
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.556974-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): LSMTree verification failed
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563834-07:00 CRITICAL Detected previous crash
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563891-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.6.2-3721). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/400baaee-3174-42b7-2fbbe693-b43079cf.dmp before terminating. Writing dump succeeded: yes
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563901-07:00 CRITICAL Stack backtrace of crashed thread:
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563902-07:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x836331]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563903-07:00 CRITICAL #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x385) [0x400000+0x8543b5]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563904-07:00 CRITICAL #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x9f) [0x400000+0x8546ff]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563905-07:00 CRITICAL #3 /lib/x86_64-linux-gnu/libc.so.6() [0x7f2fa8e1f000+0x3c050]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563905-07:00 CRITICAL #4 /lib/x86_64-linux-gnu/libc.so.6() [0x7f2fa8e1f000+0x8ae2c]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563906-07:00 CRITICAL #5 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12) [0x7f2fa8e1f000+0x3bfb2]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563907-07:00 CRITICAL #6 /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3) [0x7f2fa8e1f000+0x26472]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563908-07:00 CRITICAL #7 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f2fa6200000+0xa89ab]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563909-07:00 CRITICAL #8 /opt/couchbase/bin/memcached() [0x400000+0x8464d2]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563909-07:00 CRITICAL #9 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f2fa6200000+0xb82fa]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563910-07:00 CRITICAL #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f2fa6200000+0xb8365]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563911-07:00 CRITICAL #11 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f2fa6200000+0xb85b7]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563912-07:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0x92d535]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563913-07:00 CRITICAL #13 /opt/couchbase/bin/memcached() [0x400000+0x108a96]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563913-07:00 CRITICAL #14 /opt/couchbase/bin/memcached() [0x400000+0x6f4508]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563914-07:00 CRITICAL #15 /opt/couchbase/bin/memcached() [0x400000+0x6f5836]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563915-07:00 CRITICAL #16 /opt/couchbase/bin/memcached() [0x400000+0x6f6d03]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563916-07:00 CRITICAL #17 /opt/couchbase/bin/memcached() [0x400000+0x723eca]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563917-07:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0x6d8548]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563949-07:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0x6da004]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563951-07:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0x8b317c]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563952-07:00 CRITICAL #21 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f2fa6200000+0xe4aa3]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.564087-07:00 CRITICAL #22 /lib/x86_64-linux-gnu/libc.so.6() [0x7f2fa8e1f000+0x89134]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.564088-07:00 CRITICAL #23 /lib/x86_64-linux-gnu/libc.so.6() [0x7f2fa8e1f000+0x1097dc]
|
cbcollect_info attached. This is the first time we are seeing this panic in 7.6.2. This was RC7 build.
Attachments
Issue Links
- is a backport of
-
MB-62750 [Magma] :- Caught unhandled std::exception-derived exception. what(): LSMTree verification failed
-
- Resolved
-