Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
7.6.0
-
None
-
Untriaged
-
0
-
Unknown
Description
Following areas can be improved for 10k index shard rebalance tests:
a. waitForIndexState optimisations - Currently each transfer token will spawn a goroutine to gather metadata and stats. With 10K indexes, there can be 100's of tokens and this can lead to lot of garbage. This scheme can be optimised to only have one goroutine gather meta every second and all other go-routines can use the data.
b. Drop Index optimisations - Currently, rebalancer issues one drop index request at a time. If 5000 indexes have to be dropped on a node, then 5000 rest calls have to be made. In each call, all the list of indexes have to be iterated. This is a time consuming operation and can slow down entire rebalance. An ideal way to approach this would be to batch all drop requests of a transfer token at a time and issue a single drop request.
c. Planner optimisations - Planner does multiple iterations of all the indexes. With 10K indexes, this is not practical. Identify and address bottlenecks in planner
Attachments
Issue Links
- relates to
-
MB-60900 High heap in use for indexer [CLONE of GSI file-based rebalance-out time is 10+ times more on RC4 build]
- Resolved