Collections/Magma: Keeping collection stats docs until erasure does not account for collection resurrection

Two parts to this

1. Mutation/Deletion + Drop in the same flush batch fails to account for the Mutation/Deletion as code in flush.cc attempts to account for collection resurrection in the same batch (drop + recreation with the same cid).
2. Resurrection of a collection before compaction "inherits" the item count of the old collection as both collections share the same stats doc.

Original Description

The following patch will cause the following failure - https://github.com/jimwwalker/kv_engine/commit/d362eee3a4c13a0341bd7a37bb6812af7a0dfb1e

[ RUN      ] CollectionsEraserTests/CollectionsEraserTest.basic/persistentMagma_full_eviction

../kv_engine/engines/ep/tests/module_tests/collections/evp_store_collections_eraser_test.cc:180: Failure

Expected equality of these values:

  0

  vb->getNumItems()

    Which is: 1

[  FAILED  ] CollectionsEraserTests/CollectionsEraserTest.basic/persistentMagma_full_eviction, where GetParam() = ("persistentMagma", "full_eviction") (68 ms)

Note that magma value-eviction is fine (as item counts come from hash-table) and note that couchstore all variants is fine.

The above patch updates one of our unit tests so that a collection is dropped, but the flush batch that persists the drop-event also stores an insert into the dropped collection.

The later 'purge' incorrectly changes the vbucket item count because magma uses the collection's stat values (stored in a local doc) for the update to the vbucket item-count. The collection stat document is never updated with the values of dropped collections and as such magma is subtracting 2 from the vbucket item count when it should subtract 3. Couchstore is unaffected by this bug because the 'purge' doesn't use the stats document, but instead explicitlly counts how many keys were dropped (as each is visited), couchstore/kv subtracts 3 from the VB total.

Lines of code of interest here:

• Stats document is not updated with a write to a collection being dropped - https://github.com/couchbase/kv_engine/blob/d0eec7ab2ecd07836fbba9926b116135dac43ddf/engines/ep/src/collections/flush.cc#L193
  • Note the comment there is wrong - as only couchstore wipes out the collection stats, magma keeps them until purge completes.
• magma reads stats https://github.com/couchbase/kv_engine/blob/master/engines/ep/src/magma-kvstore/magma-kvstore.cc#L1934
• magma uses stats for update of item count https://github.com/couchbase/kv_engine/blob/d0eec7ab2ecd07836fbba9926b116135dac43ddf/engines/ep/src/magma-kvstore/magma-kvstore.cc#L1935