kv_engine needs to own metadata purge interval

Magma can perform background compactions based on some fragmentation percentage that they derive internally. This is not driven by ns_server calling compactDB on the engine.

Currently when instructing kv_engine to perform a compaction ns_server will provide a context with a PurgeBeforeTs (time before which we can purge tombstones). This information is only relevant to the current compaction and is not stored by kv_engine.

For kv_engine to support the purging of tombstones when magma is the actor driving compaction, kv_engine needs to know the metadata purge interval to tell magma if it should keep or drop a tombstone. We would like to modify the compaction API between ns_server and kv_engine. Our ideal API would be to store the metadata purge interval in the engine configuration file (configuration.json) and for ns_server to continue to tell us when to perform a compaction without supplying the PurgeBeforeTs (or to continue to do so and ignore this parameter). kv_engine can then calculate this timestamp itself. Incidentally, this will also allow the metadata purge interval to be configurable on a per-bucket basis.

I believe that the only useful field of the CompactDB payload is the purge_before_ts. From my limited understanding of the ns_server code I believe that purge_before_seq is always set to 0. We ignore the db_file_id field and always use the vBucket ID from the packet header. I believe the drop_deletes field may have some test only usage so we may want to keep this, but functionality should still be available via couch_compact. I believe the only value returned to ns_server is a status code of either SUCCESS or NOT_MY_VBUCKET.

API proposals:

In all cases we would like for ns_server to set the metadata purge interval in the engine configuration.json on CheshireCat nodes. We will use this value to create an appropriate purge_before_ts value for implicit compactions.

CompactDB API:

• Variant 1 - Add a CompactDB v2.

We'd like to simplify the CompactDB API/payload given that we need to make some changes in this area. Ns_server could sent a CompactDB message with the vBucket ID set in the header and with no extras/body. We could identify this as v2 by the body length of 0. Kv_engine could then calculate the appropriate purge_before_ts value based on the metadata purge interval in the configuration.json and pass this to the backend as normal on explicit compactions. This makes the explicit compaction path in kv_engine a good bit easier to follow and reason about.

• Variant 2.1 - Respect body of CompactDB

If ns_server wishes to keep the ability to instruct kv_engine to compact with certain parameters then we could leave the explicit compaction API as is. Kv_engine could instruct the storage engine (couchstore) to compact with the given parameters without change. Only implicit compactions would use the metadata purge interval in the configuration.json.

• Variant 2.2 - Ignore body of CompactDB

We could leave the CompactDB message as is (for the sake of easier backwards compatability in ns_server) but kv_engine could ignore the body. We'd take the vBucket ID from the header of the packet and then calculate the appropriate purge_before_ts value based on the metadata purge interval in the configuration.json. This requires changes to the explicit compaction path in kv_engine but both explicit and implicit compactions would use the value stored in the configuration.json.