Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-57057

[Backport MB-56564 to 7.2.1] Build all indexes on new node before switching query traffic during rebalance

    XMLWordPrintable

Details

    Description

      When a new index node joins the cluster during rebalance, indexer will transfer/build indexes in batches (of 3 by default) on the node. Once the first batch finishes, the indexes become Active and start serving query traffic. Concurrently, the next batch(es) start building and this process goes on till all the transfers are complete. The problem with this approach is that, as index build is a resource intensive process and there is no resource isolation with query workload, this can lead to slow response for the application.

      Alternative approach can be to build all the indexes on the new node and once all the resources intensive activity of index builds is complete, switch the query traffic to the new node and drop the indexes on its old source node. This can reduce the impact of rebalance on the application query workload.

      This improvement can have following potential benefits:

      1. Index build isolation on new node. Old nodes can serve application query traffic reducing the impact of rebalance on the application.
      2. Improve rebalance time by building bigger batches on new node. As new node is not serving application traffic, all resources can be utilized for index builds.
      3. Reduce number of times DCP data fetched (by using bigger batches). 
      4. Reduce CPU usage on dcp/projector by reducing the number of DCP conn/vb workers. 

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              hemant.rajput Hemant Rajput
              amit.kulkarni Amit Kulkarni
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty