Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62703

DCP backfill with no progress should forcefully be stopped in some circumstances

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • Morpheus
    • 7.6.0
    • couchbase-bucket
    • None
    • Triaged
    • 0
    • No

    Description

      A DCP client which ends up in backfill and that makes no progress with the stream (but is responding to DCP noop) risks the node running out of disk. The issue is that the a disk data is "locked" (e.g. a file is open) and that prevents that snapshot being released when compaction later occurs. In the worst case we could be tripling the disk usage, instead of double (when compacting).

      E.g. for 1 vbucket.

      • n GB for the file held open by backfill (file-v1)
      • n GB for a new file being created by compaction (file-v3)
      • n GB for the old file that compaction is released (file-v2)

      KV can inspect the streams that are in backfill and apply some logic to detect no progress and secondly it could only apply the force-end when it knows the vbucket is growing - i.e. it's fine to hold the backfill if the system is doing nothing.

      Attachments

        For Gerrit Dashboard: MB-62703
        # Subject Branch Project Status CR V

        Activity

          People

            jwalker Jim Walker
            jwalker Jim Walker
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              PagerDuty