unable to find docs when shards inconsistent

• Test starts with 3 couchbase nodes on source and 2 ElasticSearch nodes on destination
• Replicating single bucket to es index
• Run test that loads 10k items to couchbase
• Run a 2nd pass of data loading that does updating/delete/expiring items to couchbase
• During 2nd pass of data loading, rebalance in a 3rd ElasticSearch node
• When dataloading+rebalancing finishes….xpecting source docs to match dest

• Actual result is src/dest mismatch:
  WARNING: Couchbase has 8790 docs, ElasticSearch all_docs returned 8215 docs
  AssertionError: Document es_xdcr_docs-7001 Missing from ES Index (default)

The following errors are seen in the elastic search logs for this test, suggesting some data was sent while a Shard was not ready to receive data in ES.

[2013-10-03 10:22:26,656][WARN ][org.eclipse.jetty.servlet.ServletHandler] /default%2Fmaster%3B00000000000000000000000000000000/_local/3-f4268e62702130298bf87f17cc481219%2Fdefault%2Fdefault
org.elasticsearch.action.NoShardAvailableActionException: [default][2] null
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.perform(TransportShardSingleOperationAction.java:143)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.start(TransportShardSingleOperationAction.java:123)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:72)
at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.doExecute(TransportShardSingleOperationAction.java:47)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:92)
at org.elasticsearch.client.support.AbstractClient.get(AbstractClient.java:179)
at org.elasticsearch.action.get.GetRequestBuilder.doExecute(GetRequestBuilder.java:112)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:85)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59)
at org.elasticsearch.transport.couchbase.capi.ElasticSearchCAPIBehavior.getDocumentElasticSearch(ElasticSearchCAPIBehavior.java:345)
at org.elasticsearch.transport.couchbase.capi.ElasticSearchCAPIBehavior.getLocalDocument(ElasticSearchCAPIBehavior.java:341)
at com.couchbase.capi.servlet.CAPIServlet.handleDocumentInternal(CAPIServlet.java:264)
at com.couchbase.capi.servlet.CAPIServlet.handleLocalDocument(CAPIServlet.java:242)
at com.couchbase.capi.servlet.CAPIServlet.service(CAPIServlet.java:91)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)