MetaKV crashes during indexer bootstrap

Test case:

./testrunner -i centos.ini skip_build_tuq=True,cbq_version=sherlock,get-cbcollect-info=True,nodes_init=4,doc_per_day=10 -t 2i.indexscans_2i.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=False,skip_cleanup=True,nodes_init=4

Scenario Description

1. Create 4 Node cluster
(1:n1ql, kv 2: kv 3: kv,index 4:index)
2. Create default bucket with 2016 items using employee dataset
3. Create Primary index using GSI

Error for step 3

E+PRIMARY+INDEX+ON+default++USING+GSI
2015-02-26 15:58:27 | ERROR | MainProcess | test_thread | [tuq_helper.create_primary_index_for_3_0_and_greater] ERROR during index creation CBQError: host 10.6.2.144: ERROR:{u'status': u'errors', u'errors': [

{u'msg': u'GSI CreatePrimaryIndex() - cause: Fails to create index. Cannot find available node for new index', u'code': 5000}

], u'results': [], u'metrics':

{u'elapsedTime': u'40.88788ms', u'executionTime': u'40.648398ms', u'resultSize': 0, u'resultCount': 0, u'errorCount': 1}

, u'requestID': u'd500060a-6ca1-491b-951d-904f759c404a', u'signature': None}

The nodes seem to be online from the UI

Drilling down

As seen on nodes with indexer service

15:57:24.951489 [Error] IndexerSettingsManager: metakv notifier failed (Post http://127.0.0.1:8091/_metakv: dial tcp 127.0.0.1:8091: connection refused)..Restarting
15:57:24.951962 [Error] IndexerSettingsManager: metakv notifier failed (Post http://127.0.0.1:8091/_metakv: dial tcp 127.0.0.1:8091: connection refused)..Restarting
15:57:24.952398 [Error] IndexerSettingsManager: metakv notifier failed (Post http://127.0.0.1:8091/_metakv: dial tcp 127.0.0.1:8091: connection refused)..Restarting
15:57:24.952852 [Error] IndexerSettingsManager: metakv notifier failed (Post http://127.0.0.1:8091/_metakv: dial tcp 127.0.0.1:8091: connection refused)..Restarting
15:57:24.953265 [Error] IndexerSettingsManager: metakv notifier failed (Post http://127.0.0.1:8091/_metakv: dial tcp 127.0.0.1:8091: connection refused)..Restarting
15:57:24.953753 [Error] IndexerSettingsManager: metakv notifier failed (Post http://127.0.0.1:8091/_metakv: dial tcp 127.0.0.1:8091: connection refused)..Restarting
15:57:24.954216 [Error] IndexerSettingsManager: metakv notifier failed (Post http://127.0.0.1:8091/_metakv: dial tcp 127.0.0.1:8091: connection refused)..Restarting
15:57:24.954716 [Error] IndexerSettingsManager: metakv notifier failed (Post http://127.0.0.1:8091/_metakv: dial tcp 127.0.0.1:8091: connection refused)..Restarting
15:57:24.955186 [Error] IndexerSettingsManager: metakv notifier failed (Post http://127.0.0.1:8091/_metakv: dial tcp 127.0.0.1:8091: connection refused)..Restarting
15:57:24.955705 [Error] IndexerSettingsManager: metakv notifier failed (Post http://127.0.0.1:8091/_metakv: dial tcp 127.0.0.1:8091: connection refused)..Restarting
[goport] 2015/02/26 15:57:24 got new line on a stdin; terminating.
15:57:58.359685 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -log=2 -vbuckets=1024 -cluster=127.0.0.1:8091 -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i]
15:57:58.361575 [Info] Indexer::NewIndexer Status INIT
15:57:58.370317 [Info] Setting maxcpus = 4
15:57:58.370440 [Info] Setting log level to Info
15:57:58.370541 [Info] Indexer::NewIndexer Starting with Vbuckets 1024
15:57:58.380693 [Info] [Queryport ":9101"] started ...
15:57:58.433226 [Info] Indexer Id 61:93:69:cc:b2:e4:5a:7e
15:57:58.433444 [Info] KVSender::handleCloseStream
Message: MsgStreamUpdate
Type: CLOSE_STREAM
Stream: MAINT_STREAM
BuildTS: []
IndexList: []
RestartTs: <nil>
15:57:58.450010 [Fatal] KVSender::sendShutdownTopic
Unexpected Error During Shutdown Projector 10.6.2.144:9999 Topic MAINT_STREAM_TOPIC_61:93:69:cc:b2:e4:5a:7e. Err projector.topicMissing
15:57:58.450041 [Error] KVSender::closeMutationStream
Error Received projector.topicMissing from 10.6.2.144:9999
15:57:58.450050 [Info] KVSender::closeMutationStream Treating TopicMissing As Success
15:57:58.452057 [Fatal] KVSender::sendShutdownTopic
Unexpected Error During Shutdown Projector 10.6.2.145:9999 Topic MAINT_STREAM_TOPIC_61:93:69:cc:b2:e4:5a:7e. Err projector.topicMissing
15:57:58.452103 [Error] KVSender::closeMutationStream
Error Received projector.topicMissing from 10.6.2.145:9999
15:57:58.452112 [Info] KVSender::closeMutationStream Treating TopicMissing As Success
15:57:58.454204 [Fatal] KVSender::sendShutdownTopic
Unexpected Error During Shutdown Projector 10.6.2.146:9999 Topic MAINT_STREAM_TOPIC_61:93:69:cc:b2:e4:5a:7e. Err projector.topicMissing