listReplicaCount is taking more than 10Sec and timing out

We see that listReplicaCount gets timed out as below
2020-10-08T15:20:31.868+00:00 [Verbose] GetWithAuth: <url to get listReplicaCount> elapsed 10.000339766s
2020-10-08T15:20:31.868+00:00 [Error] Planner::getIndexNumReplica: Error from reading index num replica for node <removed>. Error = Get <url to get listReplicaCount> net/http: request canceled (Client.Timeout exceeded while awaiting headers)

This call is made by Index Planner to get optimal load distribution.
Tried reproducing this issue locally and looks like the http calls to MetaKv to check for DDL tokens is causing this delay.