Alert if vbucket replica is behind indexer by more than 30mins

If Indexer is ahead of vbucket replica, in the event of hard failover and KV data loss, indexes may need to rollback. This is due to the fact that once the slow replica takes over as the vbucket master, it will ask indexer to rollback to its higher seqno (which can be an old seqno as indexer is ahead). Depending on how far behind the vbucket replica is, this may lead to rollback to zero and production outage.

Indexer already has a provision to detect this situation and retain 2 extra disk snapshots (in addition to the 2 default disk snapshots). Disk snapshot is taken at 10 minute interval.

It is good to raise an alert about this situation, if the oldest of all 4 disk snapshots is not enough to avoid a rollback to 0 i.e. vbucket replica is more than 30mins behind indexer.