Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-52046

[System Test] Caught unhandled std::exception-derived exception. what(): kvstore-194/rev-000000001: found keyIndex seqno < seqIndex seqno (2071646 < 2073699)

    XMLWordPrintable

Details

    Description

      QE TEST

      -test tests/integration/neo/test_neo_magma_milestone4.yml -scope tests/integration/neo/scope_neo_magma.yml
      

      Day - 1
      Cycle - 1
      Scale - 3

      TEST STEP
      Perform failover and rebalance out pair of KV nodes.

      [2022-05-02T17:31:05-07:00, sequoiatools/couchbase-cli:7.1:7e0802] failover -c 172.23.108.139:8091 --server-failover 172.23.108.138:8091 -u Administrator -p password
      [2022-05-02T17:32:28-07:00, sequoiatools/couchbase-cli:7.1:c01f33] failover -c 172.23.108.139:8091 --server-failover 172.23.108.143:8091 -u Administrator -p password --hard
      [2022-05-02T17:32:37-07:00, sequoiatools/couchbase-cli:7.1:dd033d] rebalance -c 172.23.108.139:8091 -u Administrator -p password
       
      Error occurred on container - sequoiatools/couchbase-cli:7.1:[rebalance -c 172.23.108.139:8091 -u Administrator -p password]
       
      docker logs dd033d
      docker start dd033d
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      

      OBSERVATION
      Rebalance failed due to memcached/magma crash.

      grep CRITICAL memcached.log on 172.23.107.25

      2022-05-03T16:51:19.307675-07:00 CRITICAL (default) WarmupBackfillTask::run(): caught exception while running backfill - aborting warmup: kvstore-194/rev-000000001: found keyIndex seqno < seqIndex seqno (2071646 < 2073699)
      2022-05-03T16:57:27.143860-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      2022-05-03T16:57:27.143927-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): kvstore-194/rev-000000001: found keyIndex seqno < seqIndex seqno (2071646 < 2073699)
      2022-05-03T16:57:29.724023-07:00 CRITICAL Detected previous crash
      2022-05-03T16:57:29.724112-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.1.1-3030). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/3ea660d2-ba86-4918-5eb49695-3f215dcf.dmp before terminating.
      2022-05-03T16:57:29.724116-07:00 CRITICAL Stack backtrace of crashed thread:
      2022-05-03T16:57:29.724118-07:00 CRITICAL    #0  /opt/couchbase/bin/memcached() [0x400000+0x7280e8]
      2022-05-03T16:57:29.724120-07:00 CRITICAL    #1  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x7798ba]
      2022-05-03T16:57:29.724122-07:00 CRITICAL    #2  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x779bf8]
      2022-05-03T16:57:29.724123-07:00 CRITICAL    #3  /lib64/libpthread.so.0() [0x7fc3bba2b000+0xf630]
      2022-05-03T16:57:29.724125-07:00 CRITICAL    #4  /lib64/libc.so.6(gsignal+0x37) [0x7fc3b9275000+0x36387]
      2022-05-03T16:57:29.724127-07:00 CRITICAL    #5  /lib64/libc.so.6(abort+0x148) [0x7fc3b9275000+0x37a78]
      2022-05-03T16:57:29.724128-07:00 CRITICAL    #6  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fc3b9b5d000+0x9963c]
      2022-05-03T16:57:29.724130-07:00 CRITICAL    #7  /opt/couchbase/bin/memcached() [0x400000+0x7325bb]
      2022-05-03T16:57:29.724131-07:00 CRITICAL    #8  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fc3b9b5d000+0xa48f6]
      2022-05-03T16:57:29.724131-07:00 CRITICAL    #9  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fc3b9b5d000+0xa4961]
      2022-05-03T16:57:29.724133-07:00 CRITICAL    #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fc3b9b5d000+0xa4bf4]
      2022-05-03T16:57:29.724134-07:00 CRITICAL    #11 /opt/couchbase/bin/memcached() [0x400000+0x10e709]
      2022-05-03T16:57:29.724135-07:00 CRITICAL    #12 /opt/couchbase/bin/memcached() [0x400000+0x5d563f]
      2022-05-03T16:57:29.724136-07:00 CRITICAL    #13 /opt/couchbase/bin/memcached() [0x400000+0x487863]
      2022-05-03T16:57:29.724137-07:00 CRITICAL    #14 /opt/couchbase/bin/memcached() [0x400000+0x469c61]
      2022-05-03T16:57:29.724138-07:00 CRITICAL    #15 /opt/couchbase/bin/memcached() [0x400000+0x4be0ed]
      2022-05-03T16:57:29.724139-07:00 CRITICAL    #16 /opt/couchbase/bin/memcached() [0x400000+0x4c1a02]
      2022-05-03T16:57:29.724140-07:00 CRITICAL    #17 /opt/couchbase/bin/memcached() [0x400000+0x4c3471]
      2022-05-03T16:57:29.724141-07:00 CRITICAL    #18 /opt/couchbase/bin/memcached() [0x400000+0x4c3c14]
      2022-05-03T16:57:29.724142-07:00 CRITICAL    #19 /opt/couchbase/bin/memcached() [0x400000+0x6a1049]
      2022-05-03T16:57:29.724143-07:00 CRITICAL    #20 /opt/couchbase/bin/memcached() [0x400000+0x69a70a]
      2022-05-03T16:57:29.724144-07:00 CRITICAL    #21 /opt/couchbase/bin/memcached() [0x400000+0x6a23ee]
      2022-05-03T16:57:29.724145-07:00 CRITICAL    #22 /opt/couchbase/bin/memcached() [0x400000+0x7f9af0]
      2022-05-03T16:57:29.724146-07:00 CRITICAL    #23 /opt/couchbase/bin/memcached() [0x400000+0x7e458a]
      2022-05-03T16:57:29.724148-07:00 CRITICAL    #24 /opt/couchbase/bin/memcached() [0x400000+0x7fcaa9]
      2022-05-03T16:57:29.724150-07:00 CRITICAL    #25 /opt/couchbase/bin/memcached() [0x400000+0x69a404]
      2022-05-03T16:57:29.724151-07:00 CRITICAL    #26 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fc3b9b5d000+0xcdd40]
      2022-05-03T16:57:29.724152-07:00 CRITICAL    #27 /lib64/libpthread.so.0() [0x7fc3bba2b000+0x7ea5]
      2022-05-03T16:57:29.724154-07:00 CRITICAL    #28 /lib64/libc.so.6(clone+0x6d) [0x7fc3b9275000+0xfeb0d]
      

      Attachments

        For Gerrit Dashboard: MB-52046
        # Subject Branch Project Status CR V

        Activity

          Wayne Siu Lynn Straus  Please approve for 7.1.1.  This is a corruption issue.

          srinath.duvuru Srinath Duvuru added a comment - Wayne Siu Lynn Straus   Please approve for 7.1.1.  This is a corruption issue.

          Build couchbase-server-7.1.1-3085 contains magma commit 4e3381b with commit message:
          MB-52046 [BP] magma: Fix target level selection for L0 compactions

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.1-3085 contains magma commit 4e3381b with commit message: MB-52046 [BP] magma: Fix target level selection for L0 compactions

          Build couchbase-server-7.2.0-1260 contains magma commit b805c84 with commit message:
          MB-52046 test: Assert ValueFormat state after decoding empty Slice

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-1260 contains magma commit b805c84 with commit message: MB-52046 test: Assert ValueFormat state after decoding empty Slice

          Build couchbase-server-7.1.1-3087 contains magma commit a5e1835 with commit message:
          MB-52046 [BP] magma: Handle empty value in ValueFormat

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.1-3087 contains magma commit a5e1835 with commit message: MB-52046 [BP] magma: Handle empty value in ValueFormat

          Build couchbase-server-8.0.0-1006 contains magma commit b805c84 with commit message:
          MB-52046 test: Assert ValueFormat state after decoding empty Slice

          build-team Couchbase Build Team added a comment - Build couchbase-server-8.0.0-1006 contains magma commit b805c84 with commit message: MB-52046 test: Assert ValueFormat state after decoding empty Slice

          People

            rohan.suri Rohan Suri
            sujay.gad Sujay Gad
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty