Lock contention in WatchMetadata and findWatcherByIndexerId

There is a lock contention in WatchMetadata and findWatcherByIndexerId when WatchMetadata gets triggered to updateIndexerList due to poolChangeNotification and findWatcherByIndexerId gets triggered in query plan phase i.e. from Refresh -> safeUpdate -> updateTopology -> FindServiceForIndexer

Steps to repro like in the Customer's environment

• Create a 4 node cluster kv+n1ql, index, index index
• Create 4 or more indexes with one replica each
• Failover an indexer node
• Add delay in handleClientStats of lifecycle manager so that watcher addition gets delayed.
• Add back the indexer node and rebalance
• Now scan any index
• While the watcher is being added back scan must be triggered resulting in long plan phase timings.