Created to handle this once is fixed

This issue is a known issue with SetWithMeta in Magma first seen with XDCR in MB-48834. The issue is caused by kv_engine not querying Magma's bloomfilters during the get performed to retrieve meta during SetWithMeta resulting in kv_engine performing bg_fetches. Couchstore does not have this issue since the bloomfilters are maintained by kv_engine which avoid IO if the document does not exist.

implemented an API for kv_engine to query Magma's bloomfilter which resulted in GET_META latency dropping from 45us to 10us. This API has to be used by kv_engine to avoid the costly fetch.

I am not sure on the status of the improvement, pinging

Assigned this ticket to storage-engine as this appears to be a Magma issue.

I’m working on testing the performance of cbdatarecovery with Magma. I’ve found that the combination of using SetWithMeta with Magma leads to much slower performance than using either Set or Couchstore. See the graphs attached to this ticket, and also the table below as a summary. All the tests below were done with a 10GB random data set created using pillowfight (10M items of 1KB size). In all cases, an empty bucket was created to restore to (i.e. no conflict resolution).

Is this a known issue? Thanks

The difference in performance between the backup tools and pillowfight seem to be caused by backup using SetWithMeta instead of Set. When the backup code is modified to always use Set (here), Magma/Couchstore performance is the same. Further evidence of this can be seen in “set_with_meta_latency_couchstore_vs_magma.png”, which has the couchstore test/latency on the left, and magma on the right (yellow is the 50th percentile, green is the 90th percentile) when using SetWithMeta. Comparing with “set_latency_couchstore_vs_magma.png”, you can see that SetWithMeta latency is much higher than the latency with Set when using Magma.

The following methods show a significant slowdown (~5x) when the storage engine is Magma instead of Couchstore:

• cbbackupmgr restore

• cbbackupmgr generate

• cbdatarecovery

However, cbc-pillowfight shows the same level of performance regardless of the storage engine used. Either something is wrong with the backup code, or there is something about gocbcore which makes it slower with Magma (pillowfight uses the C SDK). I'm skeptical of the former as AFAICT the backup code works the same way when sending docs to the cluster regardless of what storage engine is used. 

More ideas:

• Make a basic perf testing program akin to pillowfight that can be used to stress test a cluster. I already tried this with gocb, but the ops/sec was too slow to see any difference. Might have to try with gocbcore, like is done in backup.

• Compare memory profiles with different storage engines - where is the slowdown happening?