Shard limit crossed - 2 extra shards created

The automated test has failed with shard limit crossed error -

The test does the following -

Sets flush buffer quota to 0.5 MB.
Sets maxDiskUsagePerShard to 26843545 ( ~ 26 MB).
Start creating indexes in batches of 10 ( Around 60 indexes in total).
During each iteration, fetch the physical shards created. (By querying /getIndexMetadata) and calculate if it has exceeded the limit.

Limit calculations are as follows -

        max_shard_count = settings['indexer.plasma.shardLimitPerTenant']

        flush_buffer_quota = settings['indexer.plasma.flushBufferQuota']

        mem_quota = self.get_indexer_mem_quota()

        shard_limit = math.ceil(mem_quota * 0.9 * flush_buffer_quota / 100)

        if shard_limit % 2 != 0:

            shard_limit += 1

Now at some point, the test has ended up with 6 shards

723809672567083874, 12326645128606720287, 15826462576033473764, 15233062409255651751, 183416447645531922, 12205676462534385018]

This had passed a few times after the fix in https://issues.couchbase.com/browse/MB-59465, but seems to have failed again.

Not really sure if the maxDiskUsagePerShard setting has any effect, so I can change the test depending upon RCA.

s3://cb-customers-secure/shardlimitexceeded/2023-11-21/shardlimitexceeded.zip

Assigning this to Varun Velamuri (because of https://issues.couchbase.com/browse/MB-59465) . Please re-assign if necessary.