Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-13386

DCP backfill doesn't read expected data

    XMLWordPrintable

Details

    • Triaged
    • Yes
    • Mar 9 - Mar 27

    Description

      During backup the dcp connection hangs. Below are the logs from the connection of one of the hung vbuckets.

      Wed Feb 4 18:03:36.334143 CST 3: (boss_event_1) DCP (Producer) eq_dcpq:cbbackup-qeYHjBNEGfOVoQKb - (vb 99) stream created with start seqno 0 and end seqno 381517
      Wed Feb 4 18:03:36.433578 CST 3: (boss_event_1) DCP (Producer) eq_dcpq:cbbackup-qeYHjBNEGfOVoQKb - (vb 99) Sending disk snapshot with start seqno 0 and end seqno 381517
      Wed Feb 4 18:03:36.436566 CST 3: (boss_event_1) DCP (Producer) eq_dcpq:cbbackup-qeYHjBNEGfOVoQKb - (vb 99) Backfill complete, 0 items read from disk, last seqno read: 381507
      Wed Feb 4 18:03:36.436577 CST 3: (boss_event_1) Backfill task (1 to 381517) finished for vb 99 disk seqno 381517 memory seqno 381517

      As we can see the backfill task should have read up to sequence number 381517, but it only reads up to 381507. This is the root cause and this behavior is unexpected. Note that we do have a cursor in the checkpoint manager for the dcp connection, but it is not expecting to actually read any items.

      vb_99:eq_dcpq:cbbackup-qeYHjBNEGfOVoQKb:cursor_checkpoint_id: 29498
      vb_99:eq_dcpq:cbbackup-qeYHjBNEGfOVoQKb:cursor_seqno: 381518
      vb_99:eq_dcpq:replication:ns_1@node01.domain.com->ns_1@node03.domain.com:boss_event_1:cursor_checkpoint_id: 29498
      vb_99:eq_dcpq:replication:ns_1@node01.domain.com->ns_1@node03.domain.com:boss_event_1:cursor_seqno: 381517
      vb_99:last_closed_checkpoint_id: 29497
      vb_99:num_checkpoint_items: 1
      vb_99:num_checkpoints: 1
      vb_99:num_items_for_persistence: 0
      vb_99:num_open_checkpoint_items: 0
      vb_99:num_tap_cursors: 2
      vb_99:open_checkpoint_id: 29498
      vb_99:persisted_checkpoint_id: 29497
      vb_99:state: active

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ericcooper Eric Cooper (Inactive)
              mikew Mike Wiederhold [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty