Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-52170

queuedSeqnoAck: invariant failed

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 1
    • Yes
    • KV June 2022

    Description

       

      Build:  7.1.1-3056

      Scenario:

      • 3 kv node cluster
      • Couchbase bucket with replicas=2
      • Delta failover (172.23.108.97) and rebalance out the node

      CB failure log:

      Rebalance exited with reason {{badmatch, {error, failed_nodes,[\'ns_1@172.23.108.98\']}}},
      [{ns_janitor,cleanup_apply_config_body,4,[{file,"src/ns_janitor.erl"},{line,295}]},
      {ns_janitor,\'-cleanup_apply_config/4-fun-0-\',4,[{file,"src/ns_janitor.erl"},{line,215}]},
      {async,\'-async_init/4-fun-1-\',3, [{file,"src/async.erl"},{line,191}]}]}.
      Rebalance Operation Id = 4d82719bd30c809e7ebc3b32e1f55882'}

      TAF test:

       

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.77890.ini GROUP=P0;durability,upgrade_version=7.1.1-3056 -t failover.failovertests.FailoverTests.test_failover_then_add_back_and_rebalance_in,nodes_init=3,replicas=2,num_failed_nodes=1,num_items=300000,recoveryType=delta:full,graceful=True,durability=MAJORITY,GROUP=P0;durability;windows;luks'

       

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            owend Daniel Owen added a comment -

            Many thanks for identifying last working build - 3045.

            We believe it is due to patches that were submitted to address MB-51606.
            The set of patches went in-between 3046 and 3064.

            The patches are currently in the process of being reverted - see https://review.couchbase.org/c/kv_engine/+/175268

            Once the above patch is in - we should retest

            owend Daniel Owen added a comment - Many thanks for identifying last working build - 3045. We believe it is due to patches that were submitted to address MB-51606 . The set of patches went in-between 3046 and 3064. The patches are currently in the process of being reverted - see https://review.couchbase.org/c/kv_engine/+/175268 Once the above patch is in - we should retest
            owend Daniel Owen added a comment -

            Hey, Ashwin Govindarajulu, The revert patches are now in. Therefore please could you retest to check the issue has been addressed?

            many thanks

            owend Daniel Owen added a comment - Hey, Ashwin Govindarajulu , The revert patches are now in. Therefore please could you retest to check the issue has been addressed? many thanks

            Hello Daniel Owen, Not seeing this issue on last week's regression run on 7.1.1-3166{}

            Ran the targeted test individually to validate the same.

            ashwin.govindarajulu Ashwin Govindarajulu added a comment - Hello Daniel Owen , Not seeing this issue on last week's regression run on 7.1.1-3166 { } Ran the targeted test individually to validate the same.
            drigby Dave Rigby added a comment -

            Ashwin Govindarajulu Can this issue be resolved then now?

            drigby Dave Rigby added a comment - Ashwin Govindarajulu Can this issue be resolved then now?

            Yes Dave Rigby, we can mark this as resolved.

            ashwin.govindarajulu Ashwin Govindarajulu added a comment - Yes Dave Rigby , we can mark this as resolved.

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty