Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-52170

queuedSeqnoAck: invariant failed

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 1
    • Yes
    • KV June 2022

    Description

       

      Build:  7.1.1-3056

      Scenario:

      • 3 kv node cluster
      • Couchbase bucket with replicas=2
      • Delta failover (172.23.108.97) and rebalance out the node

      CB failure log:

      Rebalance exited with reason {{badmatch, {error, failed_nodes,[\'ns_1@172.23.108.98\']}}},
      [{ns_janitor,cleanup_apply_config_body,4,[{file,"src/ns_janitor.erl"},{line,295}]},
      {ns_janitor,\'-cleanup_apply_config/4-fun-0-\',4,[{file,"src/ns_janitor.erl"},{line,215}]},
      {async,\'-async_init/4-fun-1-\',3, [{file,"src/async.erl"},{line,191}]}]}.
      Rebalance Operation Id = 4d82719bd30c809e7ebc3b32e1f55882'}

      TAF test:

       

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.77890.ini GROUP=P0;durability,upgrade_version=7.1.1-3056 -t failover.failovertests.FailoverTests.test_failover_then_add_back_and_rebalance_in,nodes_init=3,replicas=2,num_failed_nodes=1,num_items=300000,recoveryType=delta:full,graceful=True,durability=MAJORITY,GROUP=P0;durability;windows;luks'

       

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Yes Dave Rigby, we can mark this as resolved.

            ashwin.govindarajulu Ashwin Govindarajulu added a comment - Yes Dave Rigby , we can mark this as resolved.
            drigby Dave Rigby added a comment -

            Ashwin Govindarajulu Can this issue be resolved then now?

            drigby Dave Rigby added a comment - Ashwin Govindarajulu Can this issue be resolved then now?

            Hello Daniel Owen, Not seeing this issue on last week's regression run on 7.1.1-3166{}

            Ran the targeted test individually to validate the same.

            ashwin.govindarajulu Ashwin Govindarajulu added a comment - Hello Daniel Owen , Not seeing this issue on last week's regression run on 7.1.1-3166 { } Ran the targeted test individually to validate the same.
            owend Daniel Owen added a comment -

            Hey, Ashwin Govindarajulu, The revert patches are now in. Therefore please could you retest to check the issue has been addressed?

            many thanks

            owend Daniel Owen added a comment - Hey, Ashwin Govindarajulu , The revert patches are now in. Therefore please could you retest to check the issue has been addressed? many thanks
            owend Daniel Owen added a comment -

            Many thanks for identifying last working build - 3045.

            We believe it is due to patches that were submitted to address MB-51606.
            The set of patches went in-between 3046 and 3064.

            The patches are currently in the process of being reverted - see https://review.couchbase.org/c/kv_engine/+/175268

            Once the above patch is in - we should retest

            owend Daniel Owen added a comment - Many thanks for identifying last working build - 3045. We believe it is due to patches that were submitted to address MB-51606 . The set of patches went in-between 3046 and 3064. The patches are currently in the process of being reverted - see https://review.couchbase.org/c/kv_engine/+/175268 Once the above patch is in - we should retest

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty