Details
-
Task
-
Resolution: Fixed
-
Major
-
3.0
-
Security Level: Public
-
None
-
Mar 9 - Mar 27
Description
Currently ep-engine requires that backfills run from start to end and cannot be paused. This creates a problem for a few reasons. First off, if a user has a large dataset then we will potentially need to backfill a large amount of data from disk and into memory. Without the ability to pause and resume a backfill we cannot control the memory overhead created from reading items off of disk. This can affect the resident ratio if the data that needs to be read by the backfill is large.
A second issue is that this means that we can only run one (or two if there are enough cpu cores) backfill at a time and all backfill must be run serially. In the future we plan on allowing more DCP connections to be created to a server. If many connections require backfill we may have some connections that do not receive data for an extended period of time because these connections are waiting for their backfills to be scheduled.
Attachments
Issue Links
For Gerrit Dashboard: MB-12179 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
41781,6 | MB-12179: Remove EPStats from CouchKVStore | master | ep-engine | Status: MERGED | +2 | +1 |
41826,6 | MB-12179: Refactor database scan code | master | ep-engine | Status: MERGED | +2 | +1 |
41827,6 | MB-12179: Make the scan API's accessable through the KVStore | master | ep-engine | Status: MERGED | +2 | +1 |
41833,4 | MB-12179: Remove the old dump API's from the KVStore | master | ep-engine | Status: MERGED | +2 | +1 |
41834,4 | MB-12179: Move dcp backfill into its own file | master | ep-engine | Status: MERGED | +2 | +1 |
41835,4 | MB-12179: Make DcpBackfill class a state machine | master | ep-engine | Status: MERGED | +2 | +1 |
41977,1 | MB-12179: Add locking to the backfill map | master | ep-engine | Status: ABANDONED | +2 | 0 |
42050,3 | MB-12179: Remove ep_io_* stats | master | ns_server | Status: MERGED | +2 | +1 |
42051,1 | WIP MB-12179: Pausable backfills | master | ep-engine | Status: ABANDONED | -1 | 0 |
42314,10 | MB-12179: Add the backfill manager | master | ep-engine | Status: MERGED | +2 | +1 |
42315,11 | MB-12179: Allow backfills to be intervleaved | master | ep-engine | Status: MERGED | +2 | +1 |
42316,13 | MB-12179: Pause backfills when after 'x' of items or bytes are read | master | ep-engine | Status: MERGED | +2 | +1 |
42319,10 | MB-12179: Add initial max buffer to the BackfillManager | master | ep-engine | Status: MERGED | +2 | +1 |