CheckpointDestroyerTask unable to keep up with heavy write workloads

During heavy write loads, the CheckpointDestroyerTask has been observed to be taking many seconds to execute:

2021-12-08T09:59:59.896711+00:00 WARNING (No Engine) Slow runtime for 'Destroying closed unreferenced checkpoints' on thread NonIoPool0: 6162 ms

This results in a large number of Checkpoints being outstanding for deletion, along with large amounts of memory being tied up in them (pending_destruction_bytes):

This causes much lower resident ratio than we would otherwise have, and increases tempOOM backoffs sent to clients.

It also results in the NonIO thread being tied up and otherwise unable to run other NonIO tasks.