Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51805

Graceful shutdown mid-snapshot on replica node can prevent node from acking until next prepare received

    XMLWordPrintable

Details

    • Triaged
    • 1
    • No
    • KV May 22

    Description

      Found while working on MB-51689. This issue is similar to MB-51606 in terms of impact/workaround.

      This issue is only applicable to memory snapshots as the consumer will ack the snapshot end of fully persisted disk snapshots.

      Consider a replica receiving a memory snapshot as follows:

      [1:Prepare(keyA), 2:Mutation(keyB)]
      

      This snapshot is dealt with by the replica by three DCP messages:

      1. SnapshotMarker 1-2 with flag Memory (and perhaps Checkpoint but that's orthogonal to this issue)
      2. DcpPrepare for keyA with seqno1
      3. DcpMutation for keyB with seqno2

      After processing 2 or 3 the flusher may run. In this scenario the flusher runs after processing 2 so the Checkpoint state is as follows:

      [1:Prepare(keyA) (HPS = 1, SnapshotStart = 1, SnapshotEnd = 2)]
      

      The flusher sees that this item is not the end of a snapshot range and so it does not attempt to persist the HPS value of the Checkpoint. The node then restarts. The HPS value loaded into memory is 0, which is correct as the replica has not yet received a full snapshot. The prepare is loaded into memory so the PDM has the correct state. On stream resumption the active now sends the following:

      1. SnapshotMarker 1-2 with flag Memory (and perhaps Checkpoint but that's orthogonal to this issue)
      2. DcpMutation for keyB with seqno2

      At this point the PassiveStream would normally notify the PDM of a snapshot end as it has the final item and the snapshot contained a prepare. This is an in-memory variable though so post-restart the value is reset and a PassiveSteam processing these messages would not notify the PDM of the snapshot end as it has not seen a prepare. Disk snapshots always notify the PDM so they are not an issue here. As the PDM is not notified, the prepare at seqno 1 is not acked back to the Producer and this node cannot contribute towards it's commit. A subsequent prepare should fix this issue as the PassiveStream should notify the PDM of the snapshot in which it is contained.

      Workaround
      Perform another durable write to the affected vBucket and the original one will be "unstuck".

      Solution

      First thoughts here are that we should notify the PDM for all snapshots, but we didn't do that originally, perhaps to avoid some performance issue, so we could experiment with such a fix but may need to do something else.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Build couchbase-server-7.2.0-1353 contains kv_engine commit d84e86e with commit message:
          MB-51805, MB-51806: Inherit HPS from previous Checkpoint

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-1353 contains kv_engine commit d84e86e with commit message: MB-51805 , MB-51806: Inherit HPS from previous Checkpoint

          Build couchbase-server-8.0.0-1021 contains kv_engine commit d84e86e with commit message:
          MB-51805, MB-51806: Inherit HPS from previous Checkpoint

          build-team Couchbase Build Team added a comment - Build couchbase-server-8.0.0-1021 contains kv_engine commit d84e86e with commit message: MB-51805 , MB-51806: Inherit HPS from previous Checkpoint

          Build couchbase-server-7.2.0-1379 contains kv_engine commit 4637d36 with commit message:
          MB-51805: Set PassiveStream::cur_snapshot_prepare on new stream

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-1379 contains kv_engine commit 4637d36 with commit message: MB-51805 : Set PassiveStream::cur_snapshot_prepare on new stream

          Build couchbase-server-8.0.0-1024 contains kv_engine commit 4637d36 with commit message:
          MB-51805: Set PassiveStream::cur_snapshot_prepare on new stream

          build-team Couchbase Build Team added a comment - Build couchbase-server-8.0.0-1024 contains kv_engine commit 4637d36 with commit message: MB-51805 : Set PassiveStream::cur_snapshot_prepare on new stream

          Verified via unit test.

          ben.huddleston Ben Huddleston added a comment - Verified via unit test.

          People

            ben.huddleston Ben Huddleston
            ben.huddleston Ben Huddleston
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty