[Error] common.ClusterAuthUrl(): CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connection refused

Test failed:
./testrunner.py -i local.ini -p gsi_type=forestdb -t rqg.test_rqg.RQGTests.test_rqg_concurrent_with_secondary,test_file_path=b/resources/rqg/simple_table_db/query_tests_using_templates/query_50000_aggregate_functions.txt.zip,database=simple_table_db,reset_database=True,concurreny_count=10,index_quota_percent=30,password=password,use_mysql=True,replicas=0,run_query_with_secondary=True,run_explain_with_hints=True,total_queries=10000,record_failure=True,check_covering_index=True,failure_record_path=/tmp,skip_cleanup=True

Error seen in query log:
[Error] common.ClusterAuthUrl(): CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: connection refused
2016/08/08 23:23:55 revrpc: Got error (dial tcp 127.0.0.1:8091: connection refused) and will retry in 1s

Error seen in projector log:
2016-08-08T23:23:49.352-07:00 [Error] Settings metakv notifier failed (Get http://127.0.0.1:8091/_metakv/indexing/settings/?feed=continuous: CBAuth database is stale: last reason: dial tcp 127.0.0.1:8091: getsockopt: connection refused)..Restarting

Attaching logs.

I also see core.indexer.* files and core.beam.smp.17614 file in /tmp folder on the same host
Attaching those files also.

Panic seen in indexer.log:

Error occured during cluster info update (invalid character '<' looking for beginning of value) .. Retrying(2)
2016-08-08T23:05:28.384-07:00 [Info] servicesChangeNotifier: Creating new notifier instance for http://%40index-cbauth:f2d648389614d33a81b66121713c4f7d@127.0.0.1:8091, default
2016-08-08T23:05:28.384-07:00 [Error] IndexerSettingsManager: metakv notifier failed (unexpected EOF)..Restarting 1
2016-08-08T23:05:28.385-07:00 [Error] common.NewServicesChangeNotifier(): Get http://127.0.0.1:8091/pools: dial tcp 127.0.0.1:8091: getsockopt: connection refused
2016/08/08 23:05:28 revrpc: Got error (EOF) and will retry in 1s
panic: CBAuth database is stale: last reason: EOF

goroutine 37 [running]:
panic(0xeedaa0, 0xc86cd7b370)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/panic.go:464 +0x3e6 fp=0xc8359c1a30 sp=0xc8359c19b0
github.com/couchbase/indexing/secondary/common.(*CbAuthHandler).GetCredentials(0xc8236c29e0, 0x0, 0x0, 0x0, 0x0)
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:223 +0xbf fp=0xc8359c1aa8 sp=0xc8359c1a30
github.com/couchbase/indexing/secondary/dcp.maybeAddAuth(0xc86ce5e2a0, 0x7fb9c45c6978, 0xc8236c29e0)
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:251 +0x43 fp=0xc8359c1b80 sp=0xc8359c1aa8
github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0xc820170500, 0xc820164ee0, 0x65, 0x7fb9c45c6978, 0xc8236c29e0, 0x10f7340, 0xc8599b5e00, 0x0, 0x0)
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:275 +0x1fd fp=0xc8359c1d90 sp=0xc8359c1b80
github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(0xc82214e538, 0xc820164ee0, 0x65, 0x10f7340, 0xc8599b5e00, 0x0, 0x0)
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:389 +0x72 fp=0xc8359c1de0 sp=0xc8359c1d90
github.com/couchbase/indexing/secondary/dcp.(*Bucket).Refresh(0xc820001b00, 0x0, 0x0)
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:490 +0xd3 fp=0xc8359c1e50 sp=0xc8359c1de0
github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets.func1(0xc8359c1f68)
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:347 +0xd4 fp=0xc8359c1f18 sp=0xc8359c1e50
github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets()
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:352 +0x7e fp=0xc8359c1fa0 sp=0xc8359c1f18
runtime.goexit()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc8359c1fa8 sp=0xc8359c1fa0
created by github.com/couchbase/indexing/secondary/common.init.1
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:34 +0xf3

goroutine 1 [select]:
runtime.gopark(0x1383a98, 0xc8798df228, 0x110f0d0, 0x6, 0x18, 0x2)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:262 +0x163 fp=0xc8798defc0 sp=0xc8798def98
runtime.selectgoImpl(0xc8798df228, 0x0, 0x18)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/select.go:392 +0xa67 fp=0xc8798df180 sp=0xc8798defc0
runtime.selectgo(0xc8798df228)
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/select.go:215 +0x12 fp=0xc8798df1a0 sp=0xc8798df180
github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0xc820092480)
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:479 +0x292 fp=0xc8798df2f8 sp=0xc8798df1a0
github.com/couchbase/indexing/secondary/indexer.NewIndexer(0xc820156360, 0x0, 0x0, 0x0, 0x0)
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:365 +0x3c0e fp=0xc8798dfc20 sp=0xc8798df2f8
main.main()
/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:153 +0x15b5 fp=0xc8798dff40 sp=0xc8798dfc20
runtime.main()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:188 +0x2b0 fp=0xc8798dff90 sp=0xc8798dff40
runtime.goexit()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc8798dff98 sp=0xc8798dff90

goroutine 17 [syscall, 7 minutes, locked to thread]:
runtime.goexit()
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820046fb8 sp=0xc820046fb0