Intermittent high reclaim pending

Observed a build up of reclaim pending upto 1.4GB for a 5M scan. Some stats from that window of time:

Fri Oct  7 22:23:27 IST 2022

  "memory_total_storage": 2.5GiB,

  "memory_used_storage": 1.7GiB,

  "default:index1:num_rows_scanned": 77660177,

  "reclaim_pending": 5.4MiB,

  "lss_blk_rdr_reads_bs": 7.4GiB,

...

Fri Oct  7 22:24:27 IST 2022

  "memory_total_storage": 3.2GiB,

  "memory_used_storage": 1.2GiB,

  "default:index1:num_rows_scanned": 82660177,

  "reclaim_pending": 1.4GiB,

  "lss_blk_rdr_reads_bs": 7.9GiB,

Fri Oct  7 22:24:30 IST 2022

  "memory_total_storage": 2.1GiB,

  "memory_used_storage": 1.0GiB,

  "default:index1:num_rows_scanned": 82680177,

  "reclaim_pending": 7.3MiB,

  "lss_blk_rdr_reads_bs": 7.9GiB,

Setup:

1. 32 core, ssd dessert machine with kv+n1ql+index on 6.6.x codebase.
2. load bucket with 84M items and create index - avg_item_size on plasma 108byte
3. continuous background mutations at 1000 updates/sec
4. continuous background scans - 10 scans of 10k rows from some random low-key every second.
5. index at ~10%RR

Then, periodically do large range scans from a random low-key in the index which iterate through 5M items. I monitor memory_used_storage, memory_total_storage and reclaim pending. From time to time, the reclaim_pending climbs to ~1.5GB.