Improve stats + logs for long queries

Currently, when capturing scan req and snapshot gen latencies, we have the following distribution:

// 0-2ms, 2ms-5ms, 5ms-10ms, 10ms-20ms, 20ms-30ms, 30ms-50ms, 50ms-100ms, 100ms-Inf

var latencyDist = []int64{0, 2, 5, 10, 20, 30, 50, 100}

// 0-2ms, 2ms-5ms, 5ms-10ms, 10ms-20ms, 20ms-30ms, 30ms-50ms, 50ms-100ms, 100ms-1000ms,

// 1000ms-5000ms, 5000ms-10000ms, 10000ms-Inf

var snapLatencyDist = []int64{0, 2, 5, 10, 20, 30, 50, 100, 1000, 5000, 10000}

// end-end scan request latency

// 0-2ms, 2ms-5ms, 5ms-10ms, 10ms-20ms, 20ms-30ms, 30ms-50ms, 50ms-100ms, 100ms-1000ms,

// 1000ms-5000ms, 5000ms-10000ms, 10000ms-50000ms, 50000ms-Inf

var scanReqLatencyDist = []int64{0, 2, 5, 10, 20, 30, 50, 100, 1000, 5000, 10000, 50000}

but this does help in many customer cases when the request latency is greater than 100ms as we only get data on 100ms-Inf. We should atleast have boundary values like 120 minutes to better understand the request latency in cases of long running queries.

We should also log things like the

• TS of requested snapshot (vs last available TS)
• when did the server receive the query ( <- mainly to cover network layer latencies between server and client as both have separate timeouts)
• time spent on scan (optional)
  if a query times out (or takes too long to respond, ie, elapsed time on a query is ~90%/95% of configured timeouts)