Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62794

[BP 7.6.3] :- Caught unhandled std::exception-derived exception. what(): LSMTree verification failed

    XMLWordPrintable

Details

    • Untriaged
    • Linux x86_64
    • 0
    • Yes

    Description

      Script to Repro

      ./sequoia -client 172.23.110.181:2375 -provider file:debian_pine.yml -test tests/integration/7.6/test_7.6.yml -scope tests/integration/7.6/scope_7.6_magma.yml -scale 2 -repeat 0 -log_level 0 -version 7.6.2-3721 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=1209600 -show_topology=true

      Saw the following panic after 7 days of system test run.

      172.23.96.111: memcached

      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.541987-07:00 CRITICAL [(bucket7) magma_7/kvstore-615/rev-000000001/seqIndex]TableID: 457 out of order
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.542005-07:00 CRITICAL [(bucket7) magma_7/kvstore-615/rev-000000001/seqIndex]LSMTree
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.543697-07:00 CRITICAL [(bucket7) magma_7/kvstore-615/rev-000000001/seqIndex]Tree Before Compaction : TreeSnapshot : (bucket7) magma_7/kvstore-615/rev-000000001/seqIndex
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.545107-07:00 CRITICAL [(bucket7) magma_7/kvstore-615/rev-000000001/seqIndex]Tree After Compaction : TreeSnapshot : (bucket7) magma_7/kvstore-615/rev-000000001/seqIndex
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.545312-07:00 CRITICAL [(bucket7) magma_7/kvstore-615/rev-000000001/seqIndex]Source Level List :  | ID:/data/couchbase/bucket7/magma.7/kvstore-615/rev-000000001/seqIndex/sstable.449.data Range:({"Seqno":22753,"op":"delta","sn":24202}, {"Seqno":24541,"op":"delta","sn":27943}) Size:32768 LogicalSize:1505517 DataBlocksSize:35777 DataBlocksCompressedSize:24067 IndexBlocksCompressedSize:219 AvgDataBlockSize:2674.11 CompressionRatio:1.48656 FileSize:32768 ItemCount:1636 State:Deleted HistoryLowTimestamp:0 HistoryHighTimestamp:0 |  | ID:/data/couchbase/bucket7/magma.7/kvstore-615/rev-000000001/seqIndex/sstable.451.data Range:({"Seqno":24544,"op":"delta","sn":27966}, {"Seqno":24545,"op":"delta","sn":27969}) Size:8192 LogicalSize:644 DataBlocksSize:63 DataBlocksCompressedSize:63 IndexBlocksCompressedSize:42 AvgDataBlockSize:63 CompressionRatio:1 FileSize:8192 ItemCount:2 State:Deleted HistoryLowTimestamp:0 HistoryHighTimestamp:0 | 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.545489-07:00 CRITICAL [(bucket7) magma_7/kvstore-615/rev-000000001/seqIndex]Destination Level List : 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.556003-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:23:31.556974-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): LSMTree verification failed
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563834-07:00 CRITICAL Detected previous crash
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563891-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.6.2-3721). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/400baaee-3174-42b7-2fbbe693-b43079cf.dmp before terminating. Writing dump succeeded: yes
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563901-07:00 CRITICAL Stack backtrace of crashed thread:
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563902-07:00 CRITICAL    #0  /opt/couchbase/bin/memcached() [0x400000+0x836331]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563903-07:00 CRITICAL    #1  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x385) [0x400000+0x8543b5]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563904-07:00 CRITICAL    #2  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x9f) [0x400000+0x8546ff]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563905-07:00 CRITICAL    #3  /lib/x86_64-linux-gnu/libc.so.6() [0x7f2fa8e1f000+0x3c050]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563905-07:00 CRITICAL    #4  /lib/x86_64-linux-gnu/libc.so.6() [0x7f2fa8e1f000+0x8ae2c]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563906-07:00 CRITICAL    #5  /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12) [0x7f2fa8e1f000+0x3bfb2]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563907-07:00 CRITICAL    #6  /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3) [0x7f2fa8e1f000+0x26472]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563908-07:00 CRITICAL    #7  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f2fa6200000+0xa89ab]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563909-07:00 CRITICAL    #8  /opt/couchbase/bin/memcached() [0x400000+0x8464d2]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563909-07:00 CRITICAL    #9  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f2fa6200000+0xb82fa]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563910-07:00 CRITICAL    #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f2fa6200000+0xb8365]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563911-07:00 CRITICAL    #11 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f2fa6200000+0xb85b7]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563912-07:00 CRITICAL    #12 /opt/couchbase/bin/memcached() [0x400000+0x92d535]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563913-07:00 CRITICAL    #13 /opt/couchbase/bin/memcached() [0x400000+0x108a96]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563913-07:00 CRITICAL    #14 /opt/couchbase/bin/memcached() [0x400000+0x6f4508]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563914-07:00 CRITICAL    #15 /opt/couchbase/bin/memcached() [0x400000+0x6f5836]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563915-07:00 CRITICAL    #16 /opt/couchbase/bin/memcached() [0x400000+0x6f6d03]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563916-07:00 CRITICAL    #17 /opt/couchbase/bin/memcached() [0x400000+0x723eca]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563917-07:00 CRITICAL    #18 /opt/couchbase/bin/memcached() [0x400000+0x6d8548]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563949-07:00 CRITICAL    #19 /opt/couchbase/bin/memcached() [0x400000+0x6da004]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563951-07:00 CRITICAL    #20 /opt/couchbase/bin/memcached() [0x400000+0x8b317c]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.563952-07:00 CRITICAL    #21 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f2fa6200000+0xe4aa3]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.564087-07:00 CRITICAL    #22 /lib/x86_64-linux-gnu/libc.so.6() [0x7f2fa8e1f000+0x89134]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000987.txt:2024-07-16T05:25:10.564088-07:00 CRITICAL    #23 /lib/x86_64-linux-gnu/libc.so.6() [0x7f2fa8e1f000+0x1097dc]

      cbcollect_info attached. This is the first time we are seeing this panic in 7.6.2. This was RC7 build.

      Attachments

        Issue Links

          is a backport of

          Bug - A problem which impairs or prevents the functions of the product. MB-62750 [Magma] :- Caught unhandled std::exception-derived exception. what(): LSMTree verification failed

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          For Gerrit Dashboard: MB-62794
          # Subject Branch Project Status CR V
          212986,1 [BP] MB-62794 lsm: Remove overlap check in compactTable master magma Status: ABANDONED -1 -1
          212987,1 [BP] MB-62794 lsm: Fix fetching overlapping tables for L0 master magma Status: ABANDONED -1 -1
          212988,4 [BP] MB-62794 lsm: Remove overlap check in compactTable trinity magma Status: MERGED +2 +1
          212989,3 [BP] MB-62794 lsm: Fix fetching overlapping tables for L0 trinity magma Status: MERGED +2 +1
          213036,3 [BP] MB-62794 lsm: Release compaction range locks in error scenarios trinity magma Status: MERGED +2 +1

          Activity

            People

              rohan.suri Rohan Suri
              srinath.duvuru Srinath Duvuru
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  PagerDuty