test_flush_bucket_and_query integration test fails with "Fails to create index. There is no available index service"

The 2i integration test test_flush_bucket_and_query in CI runs is failing with error:

[2015-04-28 17:00:12,270] - [base_2i:456] INFO - CBQError: host 127.0.0.1: ERROR:{u'status': u'errors', u'errors': [

{u'msg': u'GSI CreateIndex() - cause: Fails to create index. There is no available index service that can process this request at this time. Index Service can be in bootstrap, recovery, or non-reachable. Please retry the operation at a later time.', u'code': 5000}

], u'results': [], u'metrics':

{u'elapsedTime': u'5.005641565s', u'executionTime': u'5.005449249s', u'resultSize': 0, u'resultCount': 0, u'errorCount': 1}

, u'requestID': u'f46cc187-2937-4325-8466-2990
ERROR

Test case flow:

• Deletes existing buckets
• Initialized the cluster
• Uses cluster: dev-4-nodes-xdcr_n1ql_2i
• Creates 'default' bucket with per node RAM quota: 8008
• Issues query CREATE PRIMARY INDEX ON default USING GSI

But the index creation is failing. From the test logs:

[2015-04-28 17:00:07,175] - [tuq_helper:343] INFO - CREATE PRIMARY INDEX ON default USING GSI
[2015-04-28 17:00:07,176] - [tuq_helper:72] INFO - RUN QUERY SELECT * FROM system:indexes
[2015-04-28 17:00:07,181] - [rest_client:2216] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes
[2015-04-28 17:00:07,235] - [tuq_helper:354] ERROR - ERROR during index creation CBQError: host 127.0.0.1: ERROR:{u'status': u'errors', u'errors': [

{u'msg': u'System datastore error - cause: Failed to load indexes for keyspace default - cause: Failed to load indexes Keyspace default - cause: all nodes failed to respond', u'code': 11000}

], u'results': [], u'metrics':

{u'elapsedTime': u'52.543469ms', u'executionTime': u'51.888696ms', u'resultSize': 0, u'resultCount': 0, u'errorCount': 1}

, u'requestID': u'8d2e7155-e66e-4610-b4ac-b8bbe9b137a8', u'signature': {u'': u''}}
[2015-04-28 17:00:07,236] - [base_2i:44] INFO - {'in_between':

{'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}

, 'initial':

{'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}

, 'after':

{'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}

, 'before': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}}
[2015-04-28 17:00:07,250] - [basetestcase:1693] INFO - list of all nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2015-04-28 17:00:07,255] - [basetestcase:1693] INFO - list of all nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2015-04-28 17:00:07,258] - [tuq_helper:72] INFO - RUN QUERY CREATE INDEX employeeb95f2fb7a7c944bdab38b5f036d05004join_yr ON default(join_yr) USING GSI
[2015-04-28 17:00:07,263] - [rest_client:2216] INFO - query params : statement=CREATE+INDEX+employeeb95f2fb7a7c944bdab38b5f036d05004join_yr+ON+default%28join_yr%29+USING+GSI+
[2015-04-28 17:00:12,270] - [base_2i:456] INFO - CBQError: host 127.0.0.1: ERROR:{u'status': u'errors', u'errors': [

{u'msg': u'GSI CreateIndex() - cause: Fails to create index. There is no available index service that can process this request at this time. Index Service can be in bootstrap, recovery, or non-reachable. Please retry the operation at a later time.', u'code': 5000}

], u'results': [], u'metrics':

{u'elapsedTime': u'5.005641565s', u'executionTime': u'5.005449249s', u'resultSize': 0, u'resultCount': 0, u'errorCount': 1}

, u'requestID': u'f46cc187-2937-4325-8466-2990
ERROR

Server Logs: http://54.251.41.134/logs-28.04-15.30.tar.gz
Test Logs: http://54.251.41.134/sherlock-28.04-15.30.fail.html