Performance Testing of Couchbase Lite for Mobile Platforms

mini PRD: https://docs.google.com/document/d/1arzEo3aYXD1o55YmzJu0BEahjTQTeBm81qqqhv3hlYc/edit?usp=sharing

Customers expect a certain level of perf testing to be done on Couchbase Lite in standalone mode for the following reasons

It is not expected to perform perf testing of Couchbase lite on every support hardware platform and it is understood that perf test results depend on a number of variables such as hardware config of device under test, OS version, the volume of data and type of queries.

• For regressions. To ensure new releases are not slower than previous ones. To ensure that updates are not resulting in a slowdown. This is probably the most critical need. To avoid issues such as ones in References. This could be done for most part on lite core although there may be platform specific variations.

• In Lithium, we will be introducing enhancements to query API to support N1QL strings - perf testing of the new API is imperative to ensure it is at par with Querybuilder API

• Comparison across platforms - for instance comparing Xamarin Android and Xamarin iOS versions running on devices with similar hardware configuration .Customers have reported significant variation across platforms specifically Android and iOS, but it isn't clear if that behavior is "expected". There are also reported instances when CBL upgrade has resulted in a slow down on one platform but not the other.

• Ideally if the perf test framework was open sourced and available to developers (not supported product), then they could customize it per their needs (data , queries, index etc) - one common question that comes up is - "I'd like to know what indexes to create by evaluating the performance of my queries [against my data set]" . This is a low priority and nice to have

REFERENCE:
1) Reference the following for the basic tests categories. We will need more comprehensive query coverage to cover aggregate, group by, join, sort etc.
https://sqlite.org/speed.html

2) Attaching a cblite2 file with 80000 JSON documents ( Use cblite to explore data model.. There is a "dataType" field that takes value of "Order", "Product", "UserLogin" and "User" which you can use to query docs of specific type)

Related:
https://github.com/couchbase/couchbase-lite-core/issues/902
https://github.com/couchbase/couchbase-lite-ios/issues/2603

https://www.actian.com/wp-content/uploads/2019/08/Actian-Zen-Edge-vs-SQLite-Benchmark-FINAL.pdf

CC Jim Borden