Hard Failover + Rebalance of kv node results in Queries using GSI index returning 0 results

./testrunner -i b/resources/dev-6-nodes-xdcr_n1ql_2i.ini doc-per-day=10,skip_build_tuq=True,cbq_version=sherlock,scan_consistency=request_plus -t 2i.recovery_2i.SecondaryIndexingRecoveryTests.test_failover,before=create_index,in_between=query_ops,after=query_ops,groups=simple,dataset=default,doc-per-day=10,services_init=n1ql:kv-kv-kv-index-index,nodes_init=5,nodes_out=1,nodes_out_dist=kv:1,skip_cleanup=True,scan_consistency=request_plus

1. Create 5 node cluster (1:n1ql, kv 2:kv 3:kv 4:index 5:index)
2. Create default bucket with 20160 items - Employee data set
3. Create Primary index using views
4. Create 10 GSI index using join_mo key for default bucket
5. Hard failover 1 kv node
6. Run n1ql queries using the GSI index
7. Rebalance the cluster

Step 6 and step 7 happen in parallel

Issue Observed

SELECT * FROM default where job_title IS NOT NULL ORDER BY job_title,_id
Results are incorrect.Actual num 0. Expected num: 20160.

EXPLAIN

2015-03-28 12:29:54 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+where+job_title+IS+NOT+NULL+ORDER+BY+job_title%2C_id
2015-03-28 12:29:54 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 54.933133ms
2015-03-28 12:29:54 | INFO | MainProcess | Cluster_Thread | [task.execute] {u'status': u'success', u'metrics':

{u'elapsedTime': u'54.933133ms', u'executionTime': u'54.858338ms', u'resultSize': 2914, u'resultCount': 1}

, u'results': [{u'#operator': u'Sequence', u'~children': [{u'#operator': u'Sequence', u'~children': [{u'index': u'employee5cc5378d1b8044a4b7b36ed31eaf0e8fjob_title', u'namespace': u'default', u'keyspace': u'default', u'#operator': u'IndexScan', u'limit': 9.223372036854776e+18, u'spans': [{u'Range':

{u'High': None, u'Low': [u'null'], u'Inclusion': 0}

, u'Seek': None}], u'using': u'gsi'}, {u'#operator': u'Parallel', u'~child': {u'#operator': u'Sequence', u'~children': [

{u'keyspace': u'default', u'#operator': u'Fetch', u'namespace': u'default'}

,

{u'#operator': u'Filter', u'condition': u'((`default`.`job_title`) is not null)'}

, {u'#operator': u'InitialProject', u'result_terms': [

{u'star': True}

]}]}}]}, {u'#operator': u'Order', u'sort_terms': [

{u'expr': u'(`default`.`job_title`)'}

,

{u'expr': u'(`default`.`_id`)'}

]}, {u'#operator': u'Parallel', u'~child': {u'#operator': u'FinalProject'}}]}], u'requestID': u'fc0fabd1-9d5d-453e-87f4-c8e4ae327534', u'signature': u'json'}

We saw that when the moment rebalance started the query that ran in between in it hit this issue