Possible corruption related to couchstore_commit failures and flusher

couchstore_commit looks like:

couchstore_error_t couchstore_commit_ex(Db* db, uint64_t timestamp) {

    COLLECT_LATENCY();

    couchstore_error_t errcode = precommit(db);

    if (errcode == COUCHSTORE_SUCCESS) {

        db->header.timestamp = timestamp;

        errcode = db_write_header(db);

    }

    if (errcode == COUCHSTORE_SUCCESS) {

        errcode = db->file.ops->sync(&db->file.lastError, db->file.handle);

    }

    if (errcode == COUCHSTORE_SUCCESS) {

        // BufferedFileOps allocates read and write buffers so that we can

        // reduce syscalls. To allow us to cache Db objects in KV_Engine we need

        // to free these buffers as they can use a substantial amount of memory

        // and may not be useful for the next Db usage.

        db->file.ops->free_buffers(db->file.handle);

    }

    return errcode;

}

If db_write_header(db) succeeds but db->file.ops->sync fail the flusher sees this as a failure and assumes that the commit failed and nothing was written to the database. The problem is that if db_write_header put the data on the filesystem and sync failed for some reason we'll see the data from the commit the next time we try to open the database file.

This was detected in the unit test for MB-36739 where we mock the sync call to fail. It manifests itself in the way that we've now got a prepare stored in the database (and the _local/vbstate document contain the correct count), but the next time the flusher tries to write data to the file the flusher don't think there is a prepare in the file and will store an incorrect value in on_disk_prepares. At the time compaction runs we'll now purge that prepare and try to adjust the on_disk_prepares to match whats on disk causing the result to become uint64_t(-1). (in other cases we'll just end up with an incorrect value)