Support for zone aware read from replica options

Currently, all read requests from SDK apps are always directed to the primary nodes in the cluster which has the active vbuckets. While this ensures that reads are always consistent, this is not the most optimal approach for the cases mentioned in the following section

Cost: Redirecting reads always to the primary nodes is cost-prohibitive for read-intensive public cloud deployments. In cloud deployments, the server cluster is typically deployed as server groups across multiple AZs for high availability. At high data volumes, egress data charges associated with reads that traverse Availability Zones can be significant. As an example, AWS charges 1c/GB of data transferred both in and out of AZ totaling 2c for a GB of data transferred. In fact, the AWS AZ pricing model has been reported to be rather confusing resulting in unexpected charges.

Consequently, there is a need for read replica requests to be zone aware and to always read from the local zone. In this case, the application is tolerant to slightly inconsistencies in data before the replicas in the local zone catches up.

Speed: While directing read requests always to the local zone has the benefit of reducing costs, it may not be the most optimal solution in all cases. For instance, the vbuckets in the local zone could be overloaded resulting in slower response times compared to requests from remote zone. 

Consequently, there is a need for requests to read from replica to determine the “fastest” zone and to be directed to the vbuckets in the fastest zone. This option would be beneficial to latency sensitive applications that are not read intensive and/or for whom data transfer costs may not be the biggest concern. 

 One Pager: https://docs.google.com/document/d/1u6EkpJUm7VBni3aV-GKCmRE2EN1bufyzGEDC1CiQL9A/edit?usp=sharing