Details
-
Improvement
-
Resolution: Fixed
-
Major
-
7.2.0, 7.1.3
Description
When mutations get queued up in indexer, there is a corresponding tsVbuuid generated every 5ms for bookkeeping and it gets queued up as well.
The size of each TsVbuuid will be atleast 40KB for 1024 vbuckets. TsVbuuid can easily grow to 10k-20k, the memory allocated just for TsVbuuid comes to 400MB-800MB for a single bucket. Indexer allocates 90% of quota for storage and tries to limit the overheads to 10%. But with high memory allocation for TsVbuuid, storage needs to release some of its quota.
Overhead of TsVbuuid data structure can be reduced by merging the queued up tsVbuuid when it goes beyond a threshold.
This is specially helpful in small node configs.
| Issue | Resolution |
| When the indexer was unable to keep up with KV mutations, and there was a queue of mutations within the indexer, there was a large memory overhead from the bookkeeping of queued up mutations. | Indexer has been improved to optimize memory usage so that the bookkeeping overhead is reduced for queued up mutations. |