Tune the slow runtime thresholds to reduce noise in memcached.log

Every task in KV has a maxExpectedDuration, which is expected to trigger slow runtime logging only for outlier task runtimes.

    std::chrono::microseconds maxExpectedDuration() const override {

        // Empirical evidence from perf runs suggests this task runs under

        // 210ms 99.9999% of the time.

        return std::chrono::milliseconds(210);

    }

These thresholds are often empirically derived, but even then, what we see is that as datasets and workloads grow, we start seen more logs filled with "slow runtimes" in an otherwise healthy system. We see this problem the most in vBucket visitor tasks like the ones for the ExpiryPager and ItemPager.

These vBucket visitors are derived from InterruptableVBucketVisitor and pause on a vBucket boundary, so the maxDuration should be selected at the 99.(9)%ile of a vBucket visit. But as in-memory workloads grow, we see some customers with large vBuckets which take more than the expected 50ms to visit the data in the vBucket. This causes noise in the logging, often every time the pager runs and causes concern in customers.

We need to tune these durations. It might be possible to make the maxExpectedDuration scale with the size of vBuckets, to provide good threshold values for small and large vBuckets.