Details
-
Improvement
-
Resolution: Unresolved
-
Major
-
6.6.0
-
1
Description
Provide more granularity in network latencies reported in RTO.
Possibly use time-stamping of network packets at the kernel level. More details on this topic can be found at -
https://github.com/torvalds/linux/blob/v3.13/Documentation/networking/timestamping.txt
https://vilimpoc.org/research/ku-latency/
From Dave Rigby:
Currently, the timing information we have for memcached requests / responses is:
- [SDK] Time application writes a request to the client OS TCP/IP socket.
- [KV-Engine] Time KV-Engine reads request from the server OS TCP/IP socket.
- [KV-Engine] Time KV-Engine writes the response to the server OS TCP/IP socket.
- [SDK] Time application reads the response from the client OS TCP/IP socket.
However, if there is say a significant difference between times 1 and 2, it is currently not possible to isolate why
- Was the client OS / machine slow to send the request across the network?
- Was the server OS / machine slow to receive the request?
- Was KV-Engine slow to read the request from the socket (even though it was ready)?
- Was the server OS slow to send the response back to the client?
- Was the SDK / application slow to read the response from the socket (even though it was ready)?
The kernel timestamp APIs mentioned could allow both client and server to timestamp when packets were actually received. They could also be used to timestamp when packets were actually sent, but that information is of limited value as it's too late (after a packet is sent) for us to be able to include it in the RTO payload.