Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-42256

"seqno_acknowledge failed because this vbucket doesn't exist" disconnects replication

    XMLWordPrintable

    Details

    • Triage:
      Untriaged
    • Story Points:
      1
    • Is this a Regression?:
      No

      Description

      If the timing of a "seqno-ack" from consumer to producer is just right, it can find that the active vbucket has been deleted, the producer will respond to the consumer with "not-my-vbucket". The consumer will receive the response packet and error as follows:

      WARNING 121: Unsupported response packet received with opcode: 0x61 (DCP_SEQNO_ACKNOWLEDGED)

      The error happens because "mcbp" has no response handler configured for this opcode. This forces a disconnect of replication, which will fail a rebalance.

      Clearly from the code, a seqno-ack can generate a response and we should gracefully handle them to avoid unexpected disconnects and rebalance failure.

        Attachments

        For Gerrit Dashboard: MB-42256
        # Subject Branch Project Status CR V
        139100,2 MB-42256: Handle seqno ack responses master kv_engine Status: ABANDONED 0 0
        139108,6 MB-42256: Handle seqno ack responses mad-hatter kv_engine Status: MERGED +2 +1
        140025,3 Merge branch mad-hatter into master master kv_engine Status: MERGED +2 +1

          Activity

          Ascending order - Click to sort in descending order
          jwalker Jim Walker created issue -
          jwalker Jim Walker made changes -
          Field Original Value New Value
          Link This issue causes CBSE-9110 [ CBSE-9110 ]
          owend Daniel Owen made changes -
          Fix Version/s 6.6.1 [ 17002 ]
          owend Daniel Owen made changes -
          Is this a Regression? Unknown [ 10452 ] No [ 10451 ]
          owend Daniel Owen made changes -
          Assignee Daniel Owen [ owend ] Dave Rigby [ drigby ]
          ben.huddleston Ben Huddleston made changes -
          Assignee Dave Rigby [ drigby ] Ben Huddleston [ ben.huddleston ]
          wayne Wayne Siu made changes -
          Link This issue blocks MB-40528 [ MB-40528 ]
          wayne Wayne Siu made changes -
          Labels approved-for-6.6.1
          Hide
          Permalink
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-6.6.1-9159 contains kv_engine commit 30a1c9e with commit message:
          MB-42256: Handle seqno ack responses

          Show
          build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9159 contains kv_engine commit 30a1c9e with commit message: MB-42256 : Handle seqno ack responses
          ben.huddleston Ben Huddleston made changes -
          Assignee Ben Huddleston [ ben.huddleston ] Jim Walker [ jwalker ]
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]
          Hide
          Permalink
          ritam.sharma Ritam Sharma added a comment -

          Jim Walker - Please help with step to validate the issue.

          Show
          ritam.sharma Ritam Sharma added a comment - Jim Walker - Please help with step to validate the issue.
          Hide
          Permalink
          jwalker Jim Walker added a comment -

          This one has unreliable steps to validate, we never actually reproduced this issue, just fixed it from logs and code inspection.

          To reproduce would require cyles of rebalance whilst sync-writes are performed and hope that a vbucket is deleted before a seqno-ack.

          Show
          jwalker Jim Walker added a comment - This one has unreliable steps to validate, we never actually reproduced this issue, just fixed it from logs and code inspection. To reproduce would require cyles of rebalance whilst sync-writes are performed and hope that a vbucket is deleted before a seqno-ack.
          jwalker Jim Walker made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          ashwin.govindarajulu Ashwin Govindarajulu made changes -
          Labels approved-for-6.6.1 approved-for-6.6.1 request-dev-verify
          Hide
          Permalink
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-7.0.0-3717 contains kv_engine commit 30a1c9e with commit message:
          MB-42256: Handle seqno ack responses

          Show
          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3717 contains kv_engine commit 30a1c9e with commit message: MB-42256 : Handle seqno ack responses

            People

            Assignee:
            jwalker Jim Walker
            Reporter:
            jwalker Jim Walker
            Votes:
            1 Vote for this issue
            Watchers:
            12 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty