Smart Batching performance testing

2021-11-17 update: All tests need to disable Autofailover CPU throttling before being run to ensure this is not negatively impacting the results. This can be done by running the following command on any Index node:

curl -u Administrator:password http://localhost:9102/settings -X POST -d '{"indexer.cpu.throttle.target":1.00}'

This needs to be verified in indexer.log – it should log the following message:

2021-11-17T10:18:01.441-08:00 [Info] CpuThrottle::SetCpuTarget: New cpuTarget: 1

–

This task describes new performance testing that should be done for Smart Batching Index Builds During Rebalance feature MB-33546. Testing should be done with indexer.settings.rebalance.redistribute_indexes == true so Rebalance will move indexes onto new nodes. Mainly we want to see Rebalance getting faster (or at least not slower) when running the same test case on 7.1.0 Neo, which has Smart Batching, versus 7.0.0 Cheshire-Cat, which does not.

Highlighting and separating out the following sentence as tests so far have not had enough index nodes:

Tests (other than existing test #1 below) should have at least 3 indexer nodes, a comparable number of KV nodes, and at least 100 indexes.

New:

Each test should be run at least 3 times on the same build level, same machines, and with same parameter settings so we can get an idea of the size of the variance across runs.

Test cases to run (comparing 7.1.0 to 7.0.0 performance):

1. Existing rebalance performance test.
2. Rebalance with many deferred indexes.
3. Rebalance with many replica repairs (i.e. after a Failover has lost a lot of replicas).
4. Rebalance with many partitioned indexes that have a large number of partitions.
5. Rebalance with many collections and many indexes on each collection.

Each of these should be run with at least two settings:

1. indexer.rebalance.transferBatchSize == 3 (default)
2. indexer.rebalance.transferBatchSize == 10

Tests should include background scan and mutation workloads.

Include indexes of different types and combinations of attributes:

• Immediate build
• Deferred build
• 0-3 replicas
• Non-partitioned
• Partitioned (especially with a large number of partitions) 

 Design doc: https://docs.google.com/document/d/14pwgHug41mfmKorHYbd-2b7S4jXSY0GQKqesQhv87Xs/edit#heading=h.pt34o7cubhkf