GSI Functional2iTest - TestBufferedScan_BackfillDisabled & TestBufferedScan_BackfillEnabled are failing

2019-04-07T16:03:12.139-07:00 [Info] CreateIndex 9329217356100883660 default/addressidx using:plasma exprType:N1QL whereExpr:<ud>()</ud> secExprs:<ud>([`address`])</ud> desc:[] isPrimary:false scheme:SINGLE  partitionKeys:<ud>([])</ud> with: - elapsed(5.686785521s) err(<nil>)

2019/04/07 16:03:12 Created the secondary index addressidx. Waiting for it become active

2019-04-07T16:03:12.139-07:00 [Info] metadata provider version changed 15 -> 16

2019-04-07T16:03:12.139-07:00 [Info] switched currmeta from 15 -> 16 force false 

2019/04/07 16:03:12 Index is now active

2019-04-07T16:03:12.139-07:00 [Info] MetadataProvider is terminated. Cleaning up ...

2019-04-07T16:03:12.139-07:00 [Info] Unwatching metadata for indexer at 172.23.121.12:9100.

2019-04-07T16:03:12.139-07:00 [Info] [Queryport-connpool:172.23.121.12:9101] ... stopped

2019-04-07T16:03:12.139-07:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 172.23.121.12:9100.  Error = read tcp 172.23.109.52:34178->172.23.121.12:9100: use of closed network connection. Kill Pipe.

2019-04-07T16:03:12.147-07:00 [Info] metadata provider version changed 1357 -> 1358

2019-04-07T16:03:12.147-07:00 [Info] switched currmeta from 1357 -> 1358 force false 

2019-04-07T16:03:12.147-07:00 [Info] GsiClient::UpdateUsecjson: using collatejson as data format between indexer and GsiClient

2019-04-07T16:03:12.147-07:00 [Info] GSIC[default/default-1554678192139343600] started ...

2019-04-07T16:03:12.153-07:00 [Error] [GsiScanClient:"172.23.121.12:9101"] req(bufferedscan) connection "172.23.109.52:50394" closed `EOF` 

2019-04-07T16:03:12.153-07:00 [Error] [GsiScanClient:"172.23.121.12:9101"] Range(bufferedscan) response failed `EOF`

2019-04-07T16:03:12.153-07:00 [Warn] scan failed: requestId bufferedscan queryport 172.23.121.12:9101 inst 7009267350202987538 partition [0]

2019-04-07T16:03:12.153-07:00 [Warn] Scan failed with error for index 9329217356100883660.  Trying scan again with replica, reqId:bufferedscan :  EOF from [172.23.121.12:9101] ...

2019-04-07T16:03:12.153-07:00 [Error] PickRandom: Fail to find indexer for all index partitions. Num partition 1.  Partition with instances 0 

2019-04-07T16:03:12.153-07:00 [Warn] Fail to find indexers to satisfy query request.  Trying scan again for index 9329217356100883660, reqId:bufferedscan :  EOF from [172.23.121.12:9101] ...

2019-04-07T16:03:12.153-07:00 [Info] [Queryport-connpool:172.23.121.12:9101] closing unhealthy connection "172.23.109.52:50394"

2019-04-07T16:03:12.165-07:00 [Info] [Queryport-connpool:172.23.121.12:9101] open new connection ...

2019/04/07 16:03:12 limit=1,chsize=256; received 1 items; took 17.766789ms

2019-04-07T16:03:12.207-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh.

2019-04-07T16:03:12.207-07:00 [Info] Refreshed Indexer List: [172.23.121.12:9100]

2019-04-07T16:03:12.209-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh.

2019-04-07T16:03:12.209-07:00 [Info] Refreshed Indexer List: [172.23.121.12:9100]

2019-04-07T16:03:12.210-07:00 [Info] switched currmeta from 16 -> 16 force true 

2019-04-07T16:03:12.212-07:00 [Info] switched currmeta from 1358 -> 1358 force true 

2019-04-07T16:03:12.450-07:00 [Info] [Queryport-connpool:172.23.121.12:9101] Stopping releaseConnsRoutine

--- FAIL: TestBufferedScan_BackfillDisabled (107.14s)

	common_test.go:123: TestBufferedScan_BackfillDisabled failed: Unexpected backfill file

2019-04-07T16:03:14.489-07:00 [Info] [GsiScanClient:"172.23.121.12:9101"] started ...

2019-04-07T16:03:14.491-07:00 [Info] [Queryport-connpool:172.23.121.12:9101] open new connection ...

2019-04-07T16:03:14.492-07:00 [Info] GsiClient::UpdateUsecjson: using collatejson as data format between indexer and GsiClient

2019-04-07T16:03:14.492-07:00 [Info] MetadataProvider.CheckIndexerStatus(): adminport=172.23.121.12:9100 connected=true

2019/04/07 16:03:14 DBG: ChangeIndexerSettings: nodes = [0xc428cf96e0]

2019/04/07 16:03:14 DBG: ChangeIndexerSettings: adminurl = 172.23.121.12:9100 host 172.23.121.12 sport 9100

2019/04/07 16:03:14 Changing config key queryport.client.settings.backfillLimit to value 0

2019-04-07T16:03:14.508-07:00 [Info] New settings received: 

{"indexer.api.enableTestServer":true,"indexer.settings.allow_large_keys":true,"indexer.settings.bufferPoolBlockSize":16384,"indexer.settings.build.batch_size":5,"indexer.settings.compaction.abort_exceed_interval":false,"indexer.settings.compaction.check_period":30,"indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.compaction.min_frag":30,"indexer.settings.compaction.min_size":524288000,"indexer.settings.compaction.plasma.manual":false,"indexer.settings.compaction.plasma.optional.decrement":5,"indexer.settings.compaction.plasma.optional.min_frag":20,"indexer.settings.compaction.plasma.optional.quota":25,"indexer.settings.corrupt_index_num_backups":1,"indexer.settings.cpuProfFname":"","indexer.settings.cpuProfile":false,"indexer.settings.enable_corrupt_index_backup":false,"indexer.settings.fast_flush_mode":true,"indexer.settings.gc_percent":100,"indexer.settings.inmemory_snapshot.fdb.interval":200,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.inmemory_snapshot.moi.interval":10,"indexer.settings.largeSnapshotThreshold":200,"indexer.settings.log_level":"info","indexer.settings.maxVbQueueLength":0,"indexer.settings.max_array_seckey_size":4096,"indexer.settings.max_cpu_percent":0,"indexer.settings.max_seckey_size":4608,"indexer.settings.max_writer_lock_prob":20,"indexer.settings.memProfFname":"","indexer.settings.memProfile":false,"indexer.settings.memory_quota":1468006400,"indexer.settings.minVbQueueLength":250,"indexer.settings.moi.debug":false,"indexer.settings.moi.persistence_threads":2,"indexer.settings.moi.recovery.max_rollbacks":2,"indexer.settings.moi.recovery_threads":4,"indexer.settings.num_replica":0,"indexer.settings.persisted_snapshot.fdb.interval":5000,"indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.persisted_snapshot.moi.interval":60000,"indexer.settings.persisted_snapshot_init_build.fdb.interval":5000,"indexer.settings.persisted_snapshot_init_build.interval":5000,"indexer.settings.persisted_snapshot_init_build.moi.interval":60000,"indexer.settings.plasma.recovery.max_rollbacks":2,"indexer.settings.recovery.max_rollbacks":5,"indexer.settings.scan_getseqnos_retries":30,"indexer.settings.scan_timeout":0,"indexer.settings.send_buffer_size":1024,"indexer.settings.sliceBufSize":50000,"indexer.settings.smallSnapshotThreshold":30,"indexer.settings.statsLogDumpInterval":60,"indexer.settings.storage_mode":"plasma","indexer.settings.storage_mode.disable_upgrade":true,"indexer.settings.wal_size":4096,"projector.settings.log_level":"info","queryport.client.settings.backfillLimit":0,"queryport.client.settings.minPoolSizeWM":1000,"queryport.client.settings.poolOverflow":30,"queryport.client.settings.poolSize":5000,"queryport.client.settings.relConnBatchSize":100}

2019-04-07T16:03:14.508-07:00 [Info] MetadataProvider is terminated. Cleaning up ...

2019-04-07T16:03:14.508-07:00 [Info] Unwatching metadata for indexer at 172.23.121.12:9100.

2019-04-07T16:03:14.508-07:00 [Info] [Queryport-connpool:172.23.121.12:9101] ... stopped

--- FAIL: TestBufferedScan_BackfillEnabled (1.33s)

	common_test.go:123: TestBufferedScan_BackfillEnabled failed: Unexpected backfill file