Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-21150

Add per-bucket |max_cas - wall_clock| > threshold ep-engine counter

    XMLWordPrintable

Details

    • Task
    • Resolution: Fixed
    • Major
    • 4.6.0
    • 4.6.0
    • couchbase-bucket

    Description

      On any vbucket, at the point we update the max_cas, if the value of it differs greatly from the current wall clock time this is a clear indication that the wall clock on some other node is skewed from the clock on this node, by approximately that amount.

      To allow ns_server to alert admins on this (that they need to examine the clocks across the replication topology) I propose that we track the following stat in ep-engine:

      ep_clock_cas_drift_threshold_exceeded: counter that is incremented every time the max cas is update and its value is greater than the wall clock by more than a threshold value. Note that since the max cas is always at least as large as the wall clock when it's updated, the difference won't actually be negative.

      I propose the default value for the threshold to be: 5 seconds. I'll file a separate ticket to track making this threshold configurable and dynamically changeable.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              jwalker Jim Walker
              dfinlay Dave Finlay
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty