Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-41038

Add packet time-stamping for more granular RTO stats

    XMLWordPrintable

Details

    • 1

    Description

      Provide more granularity in network latencies reported in RTO.

      Possibly use time-stamping of network packets at the kernel level. More details on this topic can be found at -

      https://github.com/torvalds/linux/blob/v3.13/Documentation/networking/timestamping.txt

      https://vilimpoc.org/research/ku-latency/

      From Dave Rigby:

      Currently, the timing information we have for memcached requests / responses is:

      1. [SDK] Time application writes a request to the client OS TCP/IP socket.
      2. [KV-Engine] Time KV-Engine reads request from the server OS TCP/IP socket.
      3. [KV-Engine] Time KV-Engine writes the response to the server OS TCP/IP socket.
      4. [SDK] Time application reads the response from the client OS TCP/IP socket.

      However, if there is say a significant difference between times 1 and 2, it is currently not possible to isolate why

      • Was the client OS / machine slow to send the request across the network?
      • Was the server OS / machine slow to receive the request?
      • Was KV-Engine slow to read the request from the socket (even though it was ready)?
      • Was the server OS slow to send the response back to the client?
      • Was the SDK / application slow to read the response from the socket (even though it was ready)?

      The kernel timestamp APIs mentioned could allow both client and server to timestamp when packets were actually received. They could also be used to timestamp when packets were actually sent, but that information is of limited value as it's too late (after a packet is sent) for us to be able to include it in the RTO payload.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            owend Daniel Owen
            shivani.gupta Shivani Gupta
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty