Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44125

Scaling bottleneck: Planner issues all metakv lookups on ALL indexer nodes

    XMLWordPrintable

Details

    • 1

    Description

      Planner (planner/proxy.go) sends all metakv token lookup requests to ALL indexer nodes via function restHelperNoLock(). The intent appears to be to make sure all outstanding tokens are found even if metakv has not reached eventual consistency on all nodes or some node is not responding.

      (Related to MB-43285 which is reducing the number of metakv calls and MB-31947 which parallelized the calls for each token type across all nodes simultaneously.)

      This approach will be a bottleneck to unlimited scale-up because the amount of metakv work for each "create index" statement increases with the number of indexer nodes in the cluster, and even the smallest metakv calls consume ~20ms. To remove this scaling barrier we need to limit the number of indexer nodes tokens are requested from to a fixed maximum number, e.g. only 1-3, which can be chosen randomly from the available set.

      Even if Planner does not get any of the tokens, it will still plan against the current known state of index layout, and any tokens that have already been applied but not cleaned up have no effect on the planner's decisions anyway. It is only tokens that have been created but not yet applied that have any impact on the planning decisions, so this is a small window. It is hard to justify introducing such a scaling bottleneck just to make sure Planner sees every last token any node is aware of.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            kevin.cherkauer Kevin Cherkauer (Inactive)
            kevin.cherkauer Kevin Cherkauer (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty