Details
-
Improvement
-
Resolution: Fixed
-
Major
-
7.1.3
Description
When a new index node joins the cluster during rebalance, indexer will transfer/build indexes in batches (of 3 by default) on the node. Once the first batch finishes, the indexes become Active and start serving query traffic. Concurrently, the next batch(es) start building and this process goes on till all the transfers are complete. The problem with this approach is that, as index build is a resource intensive process and there is no resource isolation with query workload, this can lead to slow response for the application.
Alternative approach can be to build all the indexes on the new node and once all the resources intensive activity of index builds is complete, switch the query traffic to the new node and drop the indexes on its old source node. This can reduce the impact of rebalance on the application query workload.
This improvement can have following potential benefits:
1. Index build isolation on new node. Old nodes can serve application query traffic reducing the impact of rebalance on the application.
2. Improve rebalance time by building bigger batches on new node. As new node is not serving application traffic, all resources can be utilized for index builds.
3. Reduce number of times DCP data fetched (by using bigger batches).
4. Reduce CPU usage on dcp/projector by reducing the number of DCP conn/vb workers.