[Magma] :- Caught unhandled std::exception-derived exception. what(): LSMTree verification failed

./sequoia -client 172.23.110.181:2375 -provider file:debian_pine.yml -test tests/integration/7.6/test_7.6.yml -scope tests/integration/7.6/scope_7.6_magma.yml -scale 2 -repeat 0 -log_level 0 -version 7.6.2-3721 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=1209600 -show_topology=true

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.541987-07:00 CRITICAL [(bucket7) magma_7/kvstore-615/rev-000000001/seqIndex]TableID: 457 out of order

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.542005-07:00 CRITICAL [(bucket7) magma_7/kvstore-615/rev-000000001/seqIndex]LSMTree

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.543697-07:00 CRITICAL [(bucket7) magma_7/kvstore-615/rev-000000001/seqIndex]Tree Before Compaction : TreeSnapshot : (bucket7) magma_7/kvstore-615/rev-000000001/seqIndex

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.545107-07:00 CRITICAL [(bucket7) magma_7/kvstore-615/rev-000000001/seqIndex]Tree After Compaction : TreeSnapshot : (bucket7) magma_7/kvstore-615/rev-000000001/seqIndex

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.545312-07:00 CRITICAL [(bucket7) magma_7/kvstore-615/rev-000000001/seqIndex]Source Level List :  | ID:/data/couchbase/bucket7/magma.7/kvstore-615/rev-000000001/seqIndex/sstable.449.data Range:({"Seqno":22753,"op":"delta","sn":24202}, {"Seqno":24541,"op":"delta","sn":27943}) Size:32768 LogicalSize:1505517 DataBlocksSize:35777 DataBlocksCompressedSize:24067 IndexBlocksCompressedSize:219 AvgDataBlockSize:2674.11 CompressionRatio:1.48656 FileSize:32768 ItemCount:1636 State:Deleted HistoryLowTimestamp:0 HistoryHighTimestamp:0 |  | ID:/data/couchbase/bucket7/magma.7/kvstore-615/rev-000000001/seqIndex/sstable.451.data Range:({"Seqno":24544,"op":"delta","sn":27966}, {"Seqno":24545,"op":"delta","sn":27969}) Size:8192 LogicalSize:644 DataBlocksSize:63 DataBlocksCompressedSize:63 IndexBlocksCompressedSize:42 AvgDataBlockSize:63 CompressionRatio:1 FileSize:8192 ItemCount:2 State:Deleted HistoryLowTimestamp:0 HistoryHighTimestamp:0 | 

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.545489-07:00 CRITICAL [(bucket7) magma_7/kvstore-615/rev-000000001/seqIndex]Destination Level List : 

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.556003-07:00 CRITICAL *** Fatal error encountered during exception handling ***

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.556974-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): LSMTree verification failed

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563834-07:00 CRITICAL Detected previous crash

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563891-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.6.2-3721). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/400baaee-3174-42b7-2fbbe693-b43079cf.dmp before terminating. Writing dump succeeded: yes

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563901-07:00 CRITICAL Stack backtrace of crashed thread:

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563902-07:00 CRITICAL    #0  /opt/couchbase/bin/memcached() [0x400000+0x836331]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563903-07:00 CRITICAL    #1  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x385) [0x400000+0x8543b5]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563904-07:00 CRITICAL    #2  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x9f) [0x400000+0x8546ff]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563905-07:00 CRITICAL    #3  /lib/x86_64-linux-gnu/libc.so.6() [0x7f2fa8e1f000+0x3c050]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563905-07:00 CRITICAL    #4  /lib/x86_64-linux-gnu/libc.so.6() [0x7f2fa8e1f000+0x8ae2c]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563906-07:00 CRITICAL    #5  /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12) [0x7f2fa8e1f000+0x3bfb2]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563907-07:00 CRITICAL    #6  /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3) [0x7f2fa8e1f000+0x26472]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563908-07:00 CRITICAL    #7  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f2fa6200000+0xa89ab]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563909-07:00 CRITICAL    #8  /opt/couchbase/bin/memcached() [0x400000+0x8464d2]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563909-07:00 CRITICAL    #9  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f2fa6200000+0xb82fa]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563910-07:00 CRITICAL    #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f2fa6200000+0xb8365]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563911-07:00 CRITICAL    #11 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f2fa6200000+0xb85b7]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563912-07:00 CRITICAL    #12 /opt/couchbase/bin/memcached() [0x400000+0x92d535]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563913-07:00 CRITICAL    #13 /opt/couchbase/bin/memcached() [0x400000+0x108a96]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563913-07:00 CRITICAL    #14 /opt/couchbase/bin/memcached() [0x400000+0x6f4508]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563914-07:00 CRITICAL    #15 /opt/couchbase/bin/memcached() [0x400000+0x6f5836]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563915-07:00 CRITICAL    #16 /opt/couchbase/bin/memcached() [0x400000+0x6f6d03]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563916-07:00 CRITICAL    #17 /opt/couchbase/bin/memcached() [0x400000+0x723eca]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563917-07:00 CRITICAL    #18 /opt/couchbase/bin/memcached() [0x400000+0x6d8548]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563949-07:00 CRITICAL    #19 /opt/couchbase/bin/memcached() [0x400000+0x6da004]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563951-07:00 CRITICAL    #20 /opt/couchbase/bin/memcached() [0x400000+0x8b317c]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563952-07:00 CRITICAL    #21 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f2fa6200000+0xe4aa3]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.564087-07:00 CRITICAL    #22 /lib/x86_64-linux-gnu/libc.so.6() [0x7f2fa8e1f000+0x89134]

/opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.564088-07:00 CRITICAL    #23 /lib/x86_64-linux-gnu/libc.so.6() [0x7f2fa8e1f000+0x1097dc]