Long scan latencies when removing a subset of docs

Cluster version: 6.5.1

Steps to repro:

a. Setup a 1KV+n1ql, 1 indexer nodes in the cluster, "default" bucket

b. Populate 10M documents in the bucket. Make sure the key's prefix changes after every 1M items. E.g., 1st 1M items contain "aaaa" as prefix, 2nd 1M items contains "aaab" as prefix and the 10th 1M items contain "aaaj" as prefix

c. Create a primary index on the bucket - Set the indexer quota so that index is in DGM (<50%)

d. Execute the N1QL query:

select meta().id from default where meta().id like 'aaad%' limit 10000

e. Delete all the documents from step (d) and continue the process until "0" results are being returned from scans i.e. all the documents with given prefix are deleted

f. Observe the increase in scan latencies - In my setup, there were a total of 100 query - delete phases. For the first few scans, the scan latencies were around 200ms but for the last few, the latency increased to 700ms.

We suspect that the increase in scan latency is because internally plasma is spending more time iterating over deleted items in the pages. The same experiment is repeated by reducing the "indexer.plasma.mainIndex.maxNumPageDeltas" to 30 for main index and the peak scan latency has come down from 700ms to 450ms.