Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-56464

[BP 7.2.1] [Stats] Handle /-/reload timeouts correctly

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      In case if /-/reload takes more time than configured timeout (currently 5 sec), ns_server keeps retrying indefinitely which leads to backed up tcp backlog in prometheus's web server. Which makes it impossible to run any http request against prometheus which we should avoid.
      Probably we should use exponential back off when retrying to reload config (at least in case when

      {error, timeout}

      happens - but this is debatable)

       

      Issue Resolution
      An issue occurred where the Cluster Manager instructed Prometheus to reload the configuration and the reload timeout impacted other requests. The Cluster Manager has been improved to handle timeouts when instructing Prometheus to reload the configuration.

      Attachments

        Issue Links

          is a backport of

          Bug - A problem which impairs or prevents the functions of the product. MB-51618 [Stats] Handle /-/reload timeouts correctly

          • Major - Major loss of function.
          • Closed

          Activity

            People

              steve.watanabe Steve Watanabe
              Abhijeeth.Nuthan Abhijeeth Nuthan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty