12 hours to build 5000 indexes

For collections performance testing, most of the tests require 10,000 indexes across 1000 collections with 100M docs in the bucket (100,000 docs per collection). Recently we have been able to create 10,000 indexes (with defer build true) after some issues were fixed. However, now that we are able to run the build phase for all 10,000 indexes, we are seeing that it is taking too long to build. For example, in this test, http://perf.jenkins.couchbase.com/job/hemera/135/, it has taken 12 hours to build 5,000 indexes, which would imply in a perfect world, 24 hours to build all 10,000 indexes. This makes it impractical to run these tests. 

The current set of collections tests for gsi has 17 tests that call for 1000 indexes or less, and 28 tests that call for 10,000 indexes. These tests will take more than 1 month of run time to complete.

Something must be changed or improved. There are a couple option:

Option 1) lower the number of docs from 100M (100,000 per collection) to something lower, say 10M.

Option 2) lower the number of indexes to 1000 indexes at max.

Option 3) increase the indexer build performance by 10x

Here are the logs for http://perf.jenkins.couchbase.com/job/hemera/135/:

705 is indexer node.
https://cb-jira.s3.us-east-2.amazonaws.com/logs/qe/collectinfo-2020-09-11T164315-ns_1%40cen-s705.perf.couchbase.com.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/qe/collectinfo-2020-09-11T164315-ns_1%40cen-s708.perf.couchbase.com.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/qe/collectinfo-2020-09-11T164315-ns_1%40cen-s709.perf.couchbase.com.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/qe/collectinfo-2020-09-11T164315-ns_1%40cen-s710.perf.couchbase.com.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/qe/collectinfo-2020-09-11T164315-ns_1%40cen-s712.perf.couchbase.com.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/qe/collectinfo-2020-09-11T164315-ns_1%40cen-s723.perf.couchbase.com.zip