Schedule janitor cleanup soon after bucket warms up in EP engine

After the bucket has been warmed up in EP engine, ns_server performs some additional steps such as applying the config, setting up the replication streams before it asks EP engine to enable the traffic to the bucket. It then changes the state of the bucket to “warmed” in ns_memcached.

The above mentioned ns_server steps are performed by the janitor cleanup process. Janitor cleanup is triggered every 10 seconds. It waits for 5 seconds for the bucket to be warmed up in EP engine. If this step timesout, then the next janitor run will be after 5 seconds.
So, there is a window of 5 to 10 seconds between two janitor cleanup runs.

While performing tests involving memcached crash, I noticed that based on when during the above mentioned window, the memcached is restarted or bucket warms up in EP engine, an extra delay of up to 10 seconds may get added before the next cleanup run which applies the config and the bucket can be marked "warmed" in ns_memcached. To reduce this delay, janitor cleanup should ideally be scheduled as soon as the bucket warms up in EP engine.

From auto-failover perspective, the bucket is considered ready only when the bucket’s state is set to “warmed” in ns_memcached. To avoid spurious auto-failovers, bucket warmup needs to be fast and any unnecessary delays like the one mentioned above need to be removed.