Offline upgrade from 2.x to 3.x loose datatype information

As of 3.0 our servers supports storing metadata information for objects. The 3.0 server will automatically try to detect the JSON datatype from “old” clients and during receival of mutations over TAP. The remaining problem is whenever you try to do an offline upgrade, it’ll just pick up the database files written by the 2.x server and start serving objects off those. These files will not have the metadata bits set, and hence the server will serve these objects as raw data. We don’t want to try to detect the format of the objects every time we read stuff off the disk because that would degrade the performance.

The proposed solution is to utilize the disk_version number in the root block of the couchstore files. Whenever ep-engine opens a new db-file it should check the "disk version" from the block, and whenever it reads out an object it has to detect if it is JSON or compressed and "update" convert the item into that (please note that it should first check if any of those bits are set). During persistence it should just follow the "new" path and store all of the meta information.

The compaction should be extended with a transform() callback that is called for each item going from the old database into the new one. In this callback we should check if the metadata information contains json/snappy etc and if not it should "convert" the objects (please note that this transform() method should only be called if the compactions detects that the file it is about to compact is from the "old" version.