============================================================================== couchbase logs (indexer.log) cbbrowse_logs indexer.log ============================================================================== 2015-08-03T20:30:41.475Z-07:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -vbuckets=1024 -cluster=127.0.0.1:8091 -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 -storageDir=/data/@2i] 2015-08-03T20:30:41.509Z-07:00 [Info] Indexer::NewIndexer Status INIT 2015-08-03T20:30:41.513Z-07:00 [Info] Setting maxcpus = 4 2015-08-03T20:30:41.513Z-07:00 [Info] Setting buffer block size to 16384 bytes 2015-08-03T20:30:41.513Z-07:00 [Info] Setting log level to Debug 2015-08-03T20:30:41.513Z-07:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2015-08-03T20:30:41.579Z-07:00 [Info] [Queryport ":9101"] started ... 2015-08-03T20:30:41.583Z-07:00 [Info] New settings received: {"indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.log_level":"debug","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":400,"indexer.settings.recovery.max_rollbacks":5,"indexer.settings.memory_quota":104857600000} 2015-08-03T20:30:41.583Z-07:00 [Info] Setting maxcpus = 4 2015-08-03T20:30:41.584Z-07:00 [Info] Setting log level to Debug 2015-08-03T20:30:41.597Z-07:00 [Debug] Repo.OpenRepositoryWithName(): open repo with name /data/@2i/MetadataStore, buffer cache size 62914560000 2015-08-03T20:30:44.881Z-07:00 [Debug] Repo.Set(): key MagicNumber, len(content) 10 2015-08-03T20:30:44.881Z-07:00 [Debug] Repo.Set(): key CurrentEpoch, len(content) 1 2015-08-03T20:30:44.881Z-07:00 [Debug] Repo.Set(): key AcceptedEpoch, len(content) 1 2015-08-03T20:30:44.882Z-07:00 [Debug] Repo.Set(): key LastLoggedTxid, len(content) 1 2015-08-03T20:30:44.882Z-07:00 [Debug] Repo.Set(): key LastCommittedTxid, len(content) 1 2015-08-03T20:30:44.882Z-07:00 [Debug] EmbeddedServer.runOnce() : Start Running Server 2015-08-03T20:30:44.882Z-07:00 [Debug] LeaderServer.RunLeaderServer(): start leader server :9100 2015-08-03T20:30:44.882Z-07:00 [Debug] LifecycleMgr.processRequest(): LifecycleMgr is ready to proces request 2015-08-03T20:30:44.882Z-07:00 [Debug] ClustMgr:handleGetLocalValue Key IndexerId 2015-08-03T20:30:44.883Z-07:00 [Debug] ClustMgr:handleSetLocalValue Key IndexerId Value e7:5e:71:29:9:be:f1:36 2015-08-03T20:30:44.883Z-07:00 [Debug] Repo.Set(): key IndexerId, len(content) 22 2015-08-03T20:30:44.883Z-07:00 [Debug] Leader.listen(): start listening to message for leader 2015-08-03T20:30:44.898Z-07:00 [Info] Indexer Id e7:5e:71:29:9:be:f1:36 2015-08-03T20:30:44.898Z-07:00 [Debug] LeaderServer.incrementEpoch(): new epoch 1 2015-08-03T20:30:44.898Z-07:00 [Debug] Repo.Set(): key AcceptedEpoch, len(content) 1 2015-08-03T20:30:44.898Z-07:00 [Debug] KVSender::handleCloseStream MAINT_STREAM Message: MsgStreamUpdate Type: CLOSE_STREAM Stream: MAINT_STREAM BuildTS: [] IndexList: [] RestartTs: 2015-08-03T20:30:44.898Z-07:00 [Debug] Repo.Set(): key CurrentEpoch, len(content) 1 2015-08-03T20:30:44.899Z-07:00 [Debug] LeaderServer.processRequest(): Leader Server is ready to proces request 2015-08-03T20:30:44.947Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.70:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:30:44.949Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.70:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:30:44.949Z-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.100.70:9999 2015-08-03T20:30:44.949Z-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:30:44.949Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.71:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:30:44.95Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.71:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:30:44.95Z-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.100.71:9999 2015-08-03T20:30:44.951Z-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:30:44.951Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.72:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:30:44.952Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.72:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:30:44.952Z-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.100.72:9999 2015-08-03T20:30:44.952Z-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:30:44.952Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.73:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:30:44.954Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.73:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:30:44.954Z-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.100.73:9999 2015-08-03T20:30:44.954Z-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:30:44.954Z-07:00 [Debug] KVSender::handleCloseStream INIT_STREAM Message: MsgStreamUpdate Type: CLOSE_STREAM Stream: INIT_STREAM BuildTS: [] IndexList: [] RestartTs: 2015-08-03T20:30:44.983Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.70:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:30:44.984Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.70:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:30:44.984Z-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.100.70:9999 2015-08-03T20:30:44.984Z-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:30:44.984Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.71:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:30:44.985Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.71:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:30:44.985Z-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.100.71:9999 2015-08-03T20:30:44.985Z-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:30:44.985Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.72:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:30:44.986Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.72:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:30:44.986Z-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.100.72:9999 2015-08-03T20:30:44.986Z-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:30:44.986Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.73:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:30:44.986Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.73:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:30:44.986Z-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.100.73:9999 2015-08-03T20:30:44.986Z-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:30:44.987Z-07:00 [Debug] ClustMgr:handleGetGlobalTopology &{map[]} 2015-08-03T20:30:44.987Z-07:00 [Debug] Indexer::initFromPersistedState Recovered IndexInstMap 2015-08-03T20:30:44.987Z-07:00 [Info] MutationMgr::handleUpdateIndexInstMap Message: MsgUpdateInstMap 2015-08-03T20:30:44.987Z-07:00 [Info] MutationMgr::handleUpdateIndexPartnMap Message: MsgUpdatePartnMap 2015-08-03T20:30:44.987Z-07:00 [Debug] ClustMgr:handleIndexerReady &{52 [] {false false false false} 0} 2015-08-03T20:30:44.987Z-07:00 [Debug] IndexManager.NotifyIndexerReady(): making request to notify indexer is ready 2015-08-03T20:30:44.987Z-07:00 [Info] Indexer::NewIndexer Status ACTIVE 2015-08-03T20:30:44.987Z-07:00 [Debug] LifecycleMgr.OnNewRequest(): queuing new request. reqId 4301102030100610354 opCode 57 2015-08-03T20:30:44.987Z-07:00 [Debug] LifecycleMgr.handleRequest(): closing bootstrap channel 2015-08-03T20:30:44.987Z-07:00 [Debug] LifecycleMgr.processRequest(): indexer is ready to process new client request. 2015-08-03T20:30:44.987Z-07:00 [Info] CompactionManager: Refreshing settings 2015-08-03T20:30:51.447Z-07:00 [Info] New settings received: {"indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"debug","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":400,"indexer.settings.recovery.max_rollbacks":5,"indexer.settings.memory_quota":104857600000} 2015-08-03T20:30:51.448Z-07:00 [Info] Setting maxcpus = 4 2015-08-03T20:30:51.448Z-07:00 [Info] Setting log level to Debug 2015-08-03T20:30:51.448Z-07:00 [Info] CompactionManager: Refreshing settings 2015-08-03T20:30:51.486Z-07:00 [Info] New settings received: {"indexer.settings.bufferPoolBlockSize":16384,"indexer.settings.compaction.check_period":30,"indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.compaction.min_frag":15,"indexer.settings.compaction.min_size":1048576,"indexer.settings.cpuProfFname":"","indexer.settings.cpuProfile":false,"indexer.settings.fast_flush_mode":true,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.largeSnapshotThreshold":200,"indexer.settings.log_level":"info","indexer.settings.maxVbQueueLength":10000,"indexer.settings.max_cpu_percent":2400,"indexer.settings.max_writer_lock_prob":20,"indexer.settings.memProfFname":"","indexer.settings.memProfile":false,"indexer.settings.memory_quota":104857600000,"indexer.settings.persisted_snapshot.interval":3000,"indexer.settings.persisted_snapshot_init_build.interval":5000,"indexer.settings.recovery.max_rollbacks":5,"indexer.settings.scan_timeout":0,"indexer.settings.send_buffer_size":1024,"indexer.settings.sliceBufSize":50000,"indexer.settings.smallSnapshotThreshold":30,"indexer.settings.statsLogDumpInterval":60,"indexer.settings.wal_size":40960,"projector.settings.log_level":"info","queryport.client.settings.poolOverflow":30,"queryport.client.settings.poolSize":1000} 2015-08-03T20:30:51.487Z-07:00 [Info] Setting maxcpus = 24 2015-08-03T20:30:51.487Z-07:00 [Info] Setting log level to Info 2015-08-03T20:30:51.487Z-07:00 [Info] CompactionManager: Refreshing settings [goport] 2015/08/03 20:30:52 got new line on a stdin; terminating. 2015-08-03T20:31:01.801Z-07:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -vbuckets=1024 -cluster=127.0.0.1:8091 -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 -storageDir=/data/@2i] 2015-08-03T20:31:01.834Z-07:00 [Info] Indexer::NewIndexer Status INIT 2015-08-03T20:31:01.849Z-07:00 [Info] Setting maxcpus = 24 2015-08-03T20:31:01.849Z-07:00 [Info] Setting buffer block size to 16384 bytes 2015-08-03T20:31:01.849Z-07:00 [Info] Setting log level to Info 2015-08-03T20:31:01.849Z-07:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2015-08-03T20:31:01.918Z-07:00 [Info] [Queryport ":9101"] started ... 2015-08-03T20:31:01.922Z-07:00 [Info] New settings received: {"indexer.settings.bufferPoolBlockSize":16384,"indexer.settings.compaction.check_period":30,"indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.compaction.min_frag":15,"indexer.settings.compaction.min_size":1048576,"indexer.settings.cpuProfFname":"","indexer.settings.cpuProfile":false,"indexer.settings.fast_flush_mode":true,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.largeSnapshotThreshold":200,"indexer.settings.log_level":"info","indexer.settings.maxVbQueueLength":10000,"indexer.settings.max_cpu_percent":2400,"indexer.settings.max_writer_lock_prob":20,"indexer.settings.memProfFname":"","indexer.settings.memProfile":false,"indexer.settings.memory_quota":104857600000,"indexer.settings.persisted_snapshot.interval":3000,"indexer.settings.persisted_snapshot_init_build.interval":5000,"indexer.settings.recovery.max_rollbacks":5,"indexer.settings.scan_timeout":0,"indexer.settings.send_buffer_size":1024,"indexer.settings.sliceBufSize":50000,"indexer.settings.smallSnapshotThreshold":30,"indexer.settings.statsLogDumpInterval":60,"indexer.settings.wal_size":40960,"projector.settings.log_level":"info","queryport.client.settings.poolOverflow":30,"queryport.client.settings.poolSize":1000} 2015-08-03T20:31:01.923Z-07:00 [Info] Setting maxcpus = 24 2015-08-03T20:31:01.923Z-07:00 [Info] Setting log level to Info 2015-08-03T20:31:05.207Z-07:00 [Info] Indexer Id e7:5e:71:29:9:be:f1:36 2015-08-03T20:31:05.233Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.70:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:31:05.235Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.70:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:31:05.235Z-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.100.70:9999 2015-08-03T20:31:05.235Z-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:31:05.235Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.71:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:31:05.236Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.71:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:31:05.236Z-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.100.71:9999 2015-08-03T20:31:05.236Z-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:31:05.236Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.72:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:31:05.238Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.72:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:31:05.238Z-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.100.72:9999 2015-08-03T20:31:05.238Z-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:31:05.238Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.73:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:31:05.239Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.73:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:31:05.24Z-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.100.73:9999 2015-08-03T20:31:05.24Z-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:31:05.27Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.70:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:31:05.271Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.70:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:31:05.271Z-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.100.70:9999 2015-08-03T20:31:05.271Z-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:31:05.271Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.71:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:31:05.272Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.71:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:31:05.272Z-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.100.71:9999 2015-08-03T20:31:05.272Z-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:31:05.272Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.72:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:31:05.272Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.72:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:31:05.272Z-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.100.72:9999 2015-08-03T20:31:05.272Z-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:31:05.272Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.73:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:31:05.273Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.73:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:31:05.273Z-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.100.73:9999 2015-08-03T20:31:05.273Z-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:31:05.273Z-07:00 [Info] MutationMgr::handleUpdateIndexInstMap Message: MsgUpdateInstMap 2015-08-03T20:31:05.273Z-07:00 [Info] MutationMgr::handleUpdateIndexPartnMap Message: MsgUpdatePartnMap 2015-08-03T20:31:05.273Z-07:00 [Info] Indexer::NewIndexer Status ACTIVE 2015-08-03T20:31:05.273Z-07:00 [Info] CompactionManager: Refreshing settings 2015-08-03T20:32:05.207Z-07:00 [Info] PeriodicStats = {"memory_quota":104857600000,"memory_used":143360,"needs_restart":false,"num_connections":0,"timings/stats_response":"63 3704286 255308840318"} 2015-08-03T20:33:05.208Z-07:00 [Info] PeriodicStats = {"memory_quota":104857600000,"memory_used":143360,"needs_restart":false,"num_connections":0,"timings/stats_response":"135 41786256 1174357861429382"} 2015-08-03T20:34:05.208Z-07:00 [Info] PeriodicStats = {"memory_quota":104857600000,"memory_used":143360,"needs_restart":false,"num_connections":0,"timings/stats_response":"207 45640055 1174571897953849"} 2015-08-03T20:35:05.208Z-07:00 [Info] PeriodicStats = {"memory_quota":104857600000,"memory_used":143360,"needs_restart":false,"num_connections":0,"timings/stats_response":"279 67197829 1482857637040647"} 2015-08-03T20:36:05.208Z-07:00 [Info] PeriodicStats = {"memory_quota":104857600000,"memory_used":143360,"needs_restart":false,"num_connections":0,"timings/stats_response":"351 71027182 1483070181742488"} 2015-08-03T20:36:29.681Z-07:00 [Info] clustMgrAgent::OnIndexCreate Notification Received for Create Index DefnId: 16558345882944865108 Name: by_email Using: forestdb Bucket: bucket-1 IsPrimary: false SecExprs: [((`email`.`f`).`f`)] PartitionScheme: SINGLE PartitionKey: WhereExpr: 2015-08-03T20:36:29.681Z-07:00 [Info] Indexer::handleCreateIndex InstId: 16558345882944865108 Defn: DefnId: 16558345882944865108 Name: by_email Using: forestdb Bucket: bucket-1 IsPrimary: false SecExprs: [((`email`.`f`).`f`)] PartitionScheme: SINGLE PartitionKey: WhereExpr: State: INDEX_STATE_CREATED Stream: INVALID_STREAM PartitionContainer: &{map[1:{1 [:9105]}] 1} 2015-08-03T20:36:29.717Z-07:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 16558345882944865108 Partition: PartitionId: 1 Endpoints: [:9105] 2015-08-03T20:36:29.726Z-07:00 [Info] Indexer::initPartnInstance Initialized Slice: Index: 16558345882944865108 Slice: SliceId: 0 File: /data/@2i/bucket-1_by_email_16558345882944865108_0.index Index: 16558345882944865108 2015-08-03T20:36:29.726Z-07:00 [Info] MutationMgr::handleUpdateIndexInstMap Message: MsgUpdateInstMap InstanceId: 16558345882944865108 Name: by_email Bucket: bucket-1 State: INDEX_STATE_CREATED Stream: INVALID_STREAM 2015-08-03T20:36:29.727Z-07:00 [Info] MutationMgr::handleUpdateIndexPartnMap Message: MsgUpdatePartnMap InstanceId: 16558345882944865108 PartitionId: 0 Endpoints: [:9105] 2015-08-03T20:36:29.727Z-07:00 [Info] clustMgrAgent::OnIndexCreate Success for Create Index DefnId: 16558345882944865108 Name: by_email Using: forestdb Bucket: bucket-1 IsPrimary: false SecExprs: [((`email`.`f`).`f`)] PartitionScheme: SINGLE PartitionKey: WhereExpr: 2015-08-03T20:36:29.737Z-07:00 [Info] clustMgrAgent::OnIndexBuild Notification Received for Build Index [16558345882944865108] 2015-08-03T20:36:29.737Z-07:00 [Info] Indexer::handleBuildIndex [16558345882944865108] 2015-08-03T20:36:29.829Z-07:00 [Info] DCPT[secidx:getseqnos-f3:ef:fc:d0:6c:50:8f:98] ##abba feed started ... 2015-08-03T20:36:29.832Z-07:00 [Info] DCPT[secidx:getseqnos-30:46:e2:f3:4f:5c:32:56] ##abba feed started ... 2015-08-03T20:36:29.834Z-07:00 [Info] DCPT[secidx:getseqnos-1a:f2:5b:4e:8e:17:91:14] ##abba feed started ... 2015-08-03T20:36:29.837Z-07:00 [Info] DCPT[secidx:getseqnos-f1:4d:ed:e0:a4:f0:b7:74] ##abba feed started ... 2015-08-03T20:36:29.837Z-07:00 [Info] {bucket,feeds} "bucket-1" created for dcp_seqno cache... 2015-08-03T20:36:29.843Z-07:00 [Info] MutationMgr::handleUpdateIndexInstMap Message: MsgUpdateInstMap InstanceId: 16558345882944865108 Name: by_email Bucket: bucket-1 State: INDEX_STATE_INITIAL Stream: MAINT_STREAM 2015-08-03T20:36:29.845Z-07:00 [Info] DATP[->dataport ":9105"] started ... 2015-08-03T20:36:29.845Z-07:00 [Info] clustMgrAgent::OnIndexBuild returns for Build Index [16558345882944865108] 2015-08-03T20:36:30.34Z-07:00 [Info] KVSender::sendMutationTopicRequest Projector 172.23.100.70:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 bucket-1 Instances [indexInstance: singlePartn: > ] 2015-08-03T20:36:30.353Z-07:00 [Info] DATP[->dataport ":9105"] new connection "172.23.100.70:44774" +1 2015-08-03T20:36:30.562Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1 2015-08-03T20:36:30.664Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1 QueuedCount 11 2015-08-03T20:36:30.958Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10000 2015-08-03T20:36:31.14Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10000 QueuedCount 1349 2015-08-03T20:36:31.298Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20000 2015-08-03T20:36:31.37Z-07:00 [Info] KVSender::sendMutationTopicRequest Projector 172.23.100.71:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 bucket-1 Instances [indexInstance: singlePartn: > ] 2015-08-03T20:36:31.391Z-07:00 [Info] DATP[->dataport ":9105"] new connection "172.23.100.71:47566" +2 2015-08-03T20:36:31.518Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 20000 QueuedCount 3375 2015-08-03T20:36:31.648Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30000 2015-08-03T20:36:31.807Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 40000 2015-08-03T20:36:31.958Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 50000 2015-08-03T20:36:32.133Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 60000 2015-08-03T20:36:32.24Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 30000 QueuedCount 23048 2015-08-03T20:36:32.272Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 70000 2015-08-03T20:36:32.382Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 40000 QueuedCount 25282 2015-08-03T20:36:32.394Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 80000 2015-08-03T20:36:32.439Z-07:00 [Info] KVSender::sendMutationTopicRequest Projector 172.23.100.72:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 bucket-1 Instances [indexInstance: singlePartn: > ] 2015-08-03T20:36:32.454Z-07:00 [Info] DATP[->dataport ":9105"] new connection "172.23.100.72:34130" +3 2015-08-03T20:36:32.539Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 90000 2015-08-03T20:36:32.732Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 100000 2015-08-03T20:36:32.872Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 50000 QueuedCount 50000 2015-08-03T20:36:32.876Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 110000 2015-08-03T20:36:33.033Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 120000 2015-08-03T20:36:33.085Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 60000 QueuedCount 50000 2015-08-03T20:36:33.144Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 130000 2015-08-03T20:36:33.292Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 140000 2015-08-03T20:36:33.445Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 150000 2015-08-03T20:36:33.633Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 160000 2015-08-03T20:36:33.777Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 170000 2015-08-03T20:36:33.805Z-07:00 [Info] KVSender::sendMutationTopicRequest Projector 172.23.100.73:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 bucket-1 Instances [indexInstance: singlePartn: > ] 2015-08-03T20:36:33.875Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 180000 2015-08-03T20:36:33.932Z-07:00 [Info] DATP[->dataport ":9105"] new connection "172.23.100.73:33230" +4 2015-08-03T20:36:33.949Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 70000 QueuedCount 50000 2015-08-03T20:36:33.989Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 190000 2015-08-03T20:36:34.097Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 200000 2015-08-03T20:36:34.21Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 210000 2015-08-03T20:36:34.217Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 80000 QueuedCount 50000 2015-08-03T20:36:34.339Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 220000 2015-08-03T20:36:34.495Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 230000 2015-08-03T20:36:34.647Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 240000 2015-08-03T20:36:34.829Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 250000 2015-08-03T20:36:34.941Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 260000 2015-08-03T20:36:34.986Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 90000 QueuedCount 50000 2015-08-03T20:36:35.037Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 270000 2015-08-03T20:36:35.156Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 280000 2015-08-03T20:36:35.219Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 100000 QueuedCount 47396 2015-08-03T20:36:35.276Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:30035040, Disk:24576, ExtraSnapshotData:0, Fragmentation:0% 2015-08-03T20:36:35.304Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 290000 2015-08-03T20:36:35.456Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 300000 2015-08-03T20:36:35.608Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 310000 2015-08-03T20:36:35.763Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 320000 2015-08-03T20:36:35.901Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 330000 2015-08-03T20:36:35.985Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 110000 QueuedCount 37396 2015-08-03T20:36:36.037Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 340000 2015-08-03T20:36:36.125Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 120000 QueuedCount 27396 2015-08-03T20:36:36.17Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 350000 2015-08-03T20:36:36.302Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 360000 2015-08-03T20:36:36.434Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 370000 2015-08-03T20:36:36.586Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 380000 2015-08-03T20:36:36.644Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 130000 QueuedCount 17396 2015-08-03T20:36:36.711Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 390000 2015-08-03T20:36:36.757Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 140000 QueuedCount 7396 2015-08-03T20:36:36.827Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 400000 2015-08-03T20:36:36.946Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 410000 2015-08-03T20:36:37.071Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 420000 2015-08-03T20:36:37.202Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 430000 2015-08-03T20:36:37.335Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 440000 2015-08-03T20:36:37.459Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 450000 2015-08-03T20:36:37.476Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 2.000082554s CommitTime 321.596381ms TotalFlushTime 5.766702908s TotalCommitTime 321.596381ms 2015-08-03T20:36:37.477Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 8272043821533391390 (SnapshotInfo: seqnos: 147396, 147396, 1 committed:true) SnapCreateDur 2.323874607s SnapOpenDur 173.12µs 2015-08-03T20:36:37.523Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 150000 QueuedCount 8120 2015-08-03T20:36:37.54Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 460000 2015-08-03T20:36:37.606Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 470000 2015-08-03T20:36:37.634Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 160000 QueuedCount 49642 2015-08-03T20:36:37.668Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 480000 2015-08-03T20:36:37.729Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 490000 2015-08-03T20:36:37.812Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 500000 2015-08-03T20:36:37.917Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 510000 2015-08-03T20:36:38.038Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 520000 2015-08-03T20:36:38.148Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 170000 QueuedCount 50000 2015-08-03T20:36:38.156Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 530000 2015-08-03T20:36:38.247Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 540000 2015-08-03T20:36:38.266Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 180000 QueuedCount 50000 2015-08-03T20:36:38.326Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 550000 2015-08-03T20:36:38.43Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 560000 2015-08-03T20:36:38.55Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 570000 2015-08-03T20:36:38.678Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 580000 2015-08-03T20:36:38.684Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 190000 QueuedCount 50000 2015-08-03T20:36:38.763Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 590000 2015-08-03T20:36:38.786Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 200000 QueuedCount 48188 2015-08-03T20:36:38.885Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 600000 2015-08-03T20:36:38.984Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 610000 2015-08-03T20:36:39.108Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 620000 2015-08-03T20:36:39.231Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 630000 2015-08-03T20:36:39.304Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 210000 QueuedCount 50000 2015-08-03T20:36:39.332Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 640000 2015-08-03T20:36:39.405Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 220000 QueuedCount 50000 2015-08-03T20:36:39.414Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 650000 2015-08-03T20:36:39.505Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 660000 2015-08-03T20:36:39.596Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 670000 2015-08-03T20:36:39.747Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 680000 2015-08-03T20:36:39.879Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 690000 2015-08-03T20:36:40.027Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 700000 2015-08-03T20:36:40.181Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 710000 2015-08-03T20:36:40.317Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 720000 2015-08-03T20:36:40.423Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 230000 QueuedCount 50000 2015-08-03T20:36:40.435Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 730000 2015-08-03T20:36:40.529Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 740000 2015-08-03T20:36:40.59Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 240000 QueuedCount 50000 2015-08-03T20:36:40.623Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 750000 2015-08-03T20:36:40.726Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 760000 2015-08-03T20:36:40.848Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 770000 2015-08-03T20:36:40.97Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 780000 2015-08-03T20:36:41.093Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 790000 2015-08-03T20:36:41.192Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 250000 QueuedCount 46512 2015-08-03T20:36:41.223Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 800000 2015-08-03T20:36:41.316Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 260000 QueuedCount 36512 2015-08-03T20:36:41.347Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 810000 2015-08-03T20:36:41.457Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 270000 QueuedCount 26512 2015-08-03T20:36:41.471Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 820000 2015-08-03T20:36:41.59Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 830000 2015-08-03T20:36:41.713Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 840000 2015-08-03T20:36:41.838Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 850000 2015-08-03T20:36:41.97Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 860000 2015-08-03T20:36:42.015Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 280000 QueuedCount 16512 2015-08-03T20:36:42.091Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 870000 2015-08-03T20:36:42.141Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 290000 QueuedCount 6512 2015-08-03T20:36:42.213Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 880000 2015-08-03T20:36:42.336Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 890000 2015-08-03T20:36:42.448Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 900000 2015-08-03T20:36:42.565Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 910000 2015-08-03T20:36:42.68Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 920000 2015-08-03T20:36:42.839Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 930000 2015-08-03T20:36:42.991Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 940000 2015-08-03T20:36:43.074Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.103µs CommitTime 468.766897ms TotalFlushTime 10.717163584s TotalCommitTime 790.363278ms 2015-08-03T20:36:43.074Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 7465490683131855512 (SnapshotInfo: seqnos: 296512, 296512, 2 committed:true) SnapCreateDur 474.777952ms SnapOpenDur 54.193µs 2015-08-03T20:36:43.218Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 300000 QueuedCount 37705 2015-08-03T20:36:43.249Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 950000 2015-08-03T20:36:43.495Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 310000 QueuedCount 50000 2015-08-03T20:36:43.766Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 960000 2015-08-03T20:36:44.098Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 970000 2015-08-03T20:36:44.372Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 980000 2015-08-03T20:36:44.47Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 320000 QueuedCount 50000 2015-08-03T20:36:44.732Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 330000 QueuedCount 50000 2015-08-03T20:36:45.062Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 990000 2015-08-03T20:36:45.366Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1000000 2015-08-03T20:36:45.87Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 340000 QueuedCount 50000 2015-08-03T20:36:46.212Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 350000 QueuedCount 50000 2015-08-03T20:36:47.346Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 360000 QueuedCount 50000 2015-08-03T20:36:47.489Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 370000 QueuedCount 44734 2015-08-03T20:36:48.066Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 380000 QueuedCount 34734 2015-08-03T20:36:48.184Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 390000 QueuedCount 24734 2015-08-03T20:36:48.74Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 400000 QueuedCount 14734 2015-08-03T20:36:48.838Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 410000 QueuedCount 4734 2015-08-03T20:36:49.718Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.036µs CommitTime 820.965812ms TotalFlushTime 16.1971031s TotalCommitTime 1.61132909s 2015-08-03T20:36:49.718Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 414734, 414734, 3 committed:true) SnapCreateDur 831.779699ms SnapOpenDur 57.094µs 2015-08-03T20:36:49.888Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 420000 QueuedCount 49875 2015-08-03T20:36:50.195Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 430000 QueuedCount 50000 2015-08-03T20:36:51.33Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 440000 QueuedCount 50000 2015-08-03T20:36:51.652Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 450000 QueuedCount 49322 2015-08-03T20:36:52.571Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 460000 QueuedCount 50000 2015-08-03T20:36:52.893Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 470000 QueuedCount 47424 2015-08-03T20:36:53.845Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 480000 QueuedCount 37424 2015-08-03T20:36:53.976Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 490000 QueuedCount 27424 2015-08-03T20:36:54.556Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 500000 QueuedCount 17424 2015-08-03T20:36:54.68Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 510000 QueuedCount 7424 2015-08-03T20:36:55.293Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.169µs CommitTime 70.928148ms TotalFlushTime 21.385163665s TotalCommitTime 1.682257238s 2015-08-03T20:36:55.293Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 517424, 517424, 4 committed:true) SnapCreateDur 76.832945ms SnapOpenDur 61.091µs 2015-08-03T20:36:55.383Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 520000 QueuedCount 37845 2015-08-03T20:36:55.686Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 530000 QueuedCount 50000 2015-08-03T20:36:56.866Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 540000 QueuedCount 50000 2015-08-03T20:36:57.162Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 550000 QueuedCount 50000 2015-08-03T20:36:58.336Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 560000 QueuedCount 50000 2015-08-03T20:36:58.494Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 570000 QueuedCount 41673 2015-08-03T20:36:59.128Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 580000 QueuedCount 31673 2015-08-03T20:36:59.248Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 590000 QueuedCount 21673 2015-08-03T20:36:59.813Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 600000 QueuedCount 11673 2015-08-03T20:36:59.905Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 610000 QueuedCount 1673 2015-08-03T20:37:00.229Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.664µs CommitTime 297.947397ms TotalFlushTime 25.75826192s TotalCommitTime 1.980204635s 2015-08-03T20:37:00.23Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 611673, 611673, 5 committed:true) SnapCreateDur 306.167835ms SnapOpenDur 34.571µs 2015-08-03T20:37:00.398Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 620000 QueuedCount 50000 2015-08-03T20:37:00.515Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 630000 QueuedCount 50000 2015-08-03T20:37:01.01Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 640000 QueuedCount 50000 2015-08-03T20:37:01.128Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 650000 QueuedCount 48641 2015-08-03T20:37:01.614Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 660000 QueuedCount 49999 2015-08-03T20:37:01.739Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 670000 QueuedCount 50000 2015-08-03T20:37:02.226Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 680000 QueuedCount 50000 2015-08-03T20:37:02.347Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 690000 QueuedCount 50000 2015-08-03T20:37:02.828Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 700000 QueuedCount 47983 2015-08-03T20:37:02.926Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 710000 QueuedCount 37983 2015-08-03T20:37:03.38Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 720000 QueuedCount 27983 2015-08-03T20:37:03.475Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 730000 QueuedCount 17983 2015-08-03T20:37:03.926Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 740000 QueuedCount 7983 2015-08-03T20:37:04.783Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.477µs CommitTime 751.074286ms TotalFlushTime 29.375673683s TotalCommitTime 2.731278921s 2015-08-03T20:37:04.784Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 747983, 747983, 6 committed:true) SnapCreateDur 775.780096ms SnapOpenDur 47.211µs 2015-08-03T20:37:04.863Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 750000 QueuedCount 32856 2015-08-03T20:37:05.17Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 760000 QueuedCount 50000 2015-08-03T20:37:05.209Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":0,"bucket-1:by_email:avg_scan_wait_latency":0,"bucket-1:by_email:avg_ts_interval":4973038346,"bucket-1:by_email:avg_ts_items_count":121262,"bucket-1:by_email:build_progress":79,"bucket-1:by_email:data_size":232399639,"bucket-1:by_email:delete_bytes":0,"bucket-1:by_email:disk_size":343412736,"bucket-1:by_email:flush_queue_size":50031,"bucket-1:by_email:frag_percent":32,"bucket-1:by_email:get_bytes":0,"bucket-1:by_email:insert_bytes":79823100,"bucket-1:by_email:items_count":747983,"bucket-1:by_email:num_commits":6,"bucket-1:by_email:num_compactions":0,"bucket-1:by_email:num_completed_requests":0,"bucket-1:by_email:num_docs_indexed":798585,"bucket-1:by_email:num_docs_pending":0,"bucket-1:by_email:num_docs_queued":201415,"bucket-1:by_email:num_flush_queued":811045,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":0,"bucket-1:by_email:num_rows_returned":0,"bucket-1:by_email:num_snapshot_waiters":0,"bucket-1:by_email:num_snapshots":6,"bucket-1:by_email:scan_bytes_read":0,"bucket-1:by_email:scan_wait_duration":0,"bucket-1:by_email:since_last_snapshot":4553979598,"bucket-1:by_email:timings/storage_clone_handle":"0 0 0","bucket-1:by_email:timings/storage_commit":"6 2731278921 1655067537116320423","bucket-1:by_email:timings/storage_del":"0 0 0","bucket-1:by_email:timings/storage_get":"761015 5492023289 1110462654184381","bucket-1:by_email:timings/storage_handle_open":"6 331871 20795021657","bucket-1:by_email:timings/storage_iterator_next":"0 0 0","bucket-1:by_email:timings/storage_new_iterator":"0 0 0","bucket-1:by_email:timings/storage_set":"1522030 23457997649 -8216379113981535611","bucket-1:by_email:timings/storage_snapshot_close":"0 0 0","bucket-1:by_email:timings/storage_snapshot_create":"0 0 0","bucket-1:by_email:total_scan_duration":0,"bucket-1:mutation_queue_size":188984,"bucket-1:num_mutations_queued":1000000,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":19,"memory_quota":104857600000,"memory_used":234557440,"needs_restart":false,"num_connections":0,"timings/stats_response":"423 88740678 1489512093499292"} 2015-08-03T20:37:05.276Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:232934474, Disk:343412736, ExtraSnapshotData:0, Fragmentation:32.17069444972478% 2015-08-03T20:37:05.276Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:37:05.296Z-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 296512. Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T20:37:06.354Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 770000 QueuedCount 50000 2015-08-03T20:37:06.662Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 780000 QueuedCount 50000 2015-08-03T20:37:07.793Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 790000 QueuedCount 50000 2015-08-03T20:37:07.983Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 800000 QueuedCount 50000 2015-08-03T20:37:09.533Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 810000 QueuedCount 50000 2015-08-03T20:37:09.797Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 820000 QueuedCount 43004 2015-08-03T20:37:10.651Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 830000 QueuedCount 33004 2015-08-03T20:37:10.803Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 840000 QueuedCount 23004 2015-08-03T20:37:11.4Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 850000 QueuedCount 13004 2015-08-03T20:37:11.523Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 860000 QueuedCount 3004 2015-08-03T20:37:12.403Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.753µs CommitTime 809.231179ms TotalFlushTime 35.845307008s TotalCommitTime 3.5405101s 2015-08-03T20:37:12.404Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 863004, 863004, 7 committed:true) SnapCreateDur 834.064466ms SnapOpenDur 32.616µs 2015-08-03T20:37:12.535Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 870000 QueuedCount 50000 2015-08-03T20:37:12.777Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 880000 QueuedCount 50000 2015-08-03T20:37:13.876Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 890000 QueuedCount 50000 2015-08-03T20:37:14.081Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 900000 QueuedCount 49381 2015-08-03T20:37:15.012Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 910000 QueuedCount 39381 2015-08-03T20:37:15.156Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 920000 QueuedCount 29381 2015-08-03T20:37:15.678Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 930000 QueuedCount 19381 2015-08-03T20:37:17.099Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 940000 QueuedCount 9381 2015-08-03T20:37:18.442Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 23.556µs CommitTime 391.89519ms TotalFlushTime 41.296559983s TotalCommitTime 3.93240529s 2015-08-03T20:37:18.442Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 949381, 949381, 8 committed:true) SnapCreateDur 416.890611ms SnapOpenDur 53.73µs 2015-08-03T20:37:18.48Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 950000 QueuedCount 6194 2015-08-03T20:37:18.758Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 960000 QueuedCount 40000 2015-08-03T20:37:19.866Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 970000 QueuedCount 30000 2015-08-03T20:37:20.016Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 980000 QueuedCount 20000 2015-08-03T20:37:20.181Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 990000 QueuedCount 10000 2015-08-03T20:37:20.801Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1000000 QueuedCount 0 2015-08-03T20:37:21.436Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 13.634µs CommitTime 605.719668ms TotalFlushTime 43.589319249s TotalCommitTime 4.538124958s 2015-08-03T20:37:21.436Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1000000, 1000000, 9 committed:true) SnapCreateDur 630.738039ms SnapOpenDur 53.914µs 2015-08-03T20:37:21.437Z-07:00 [Info] MutationMgr::handleUpdateIndexInstMap Message: MsgUpdateInstMap InstanceId: 16558345882944865108 Name: by_email Bucket: bucket-1 State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM 2015-08-03T20:37:29.458Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 2015-08-03T20:37:31.953Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.2µs CommitTime 41.309308ms TotalFlushTime 43.606403993s TotalCommitTime 4.579434266s 2015-08-03T20:37:31.953Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1001200, 1000600, 10 committed:true) SnapCreateDur 66.794728ms SnapOpenDur 79.883µs 2015-08-03T20:37:32.206Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1003810, 1001905, 0 committed:false) SnapCreateDur 25.304µs SnapOpenDur 99.645278ms 2015-08-03T20:37:32.281Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1004358, 1002179, 0 committed:false) SnapCreateDur 23.753µs SnapOpenDur 5.641814ms 2015-08-03T20:37:32.301Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34141" doReceive() ... 2015-08-03T20:37:32.442Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34146" doReceive() ... 2015-08-03T20:37:32.446Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34147" doReceive() ... 2015-08-03T20:37:32.446Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34148" doReceive() ... 2015-08-03T20:37:32.446Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34149" doReceive() ... 2015-08-03T20:37:32.463Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34150" doReceive() ... 2015-08-03T20:37:32.48Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34151" doReceive() ... 2015-08-03T20:37:32.484Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34152" doReceive() ... 2015-08-03T20:37:32.506Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1006096, 1003048, 0 committed:false) SnapCreateDur 24.01µs SnapOpenDur 9.374727ms 2015-08-03T20:37:32.51Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34153" doReceive() ... 2015-08-03T20:37:32.529Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34154" doReceive() ... 2015-08-03T20:37:32.56Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34155" doReceive() ... 2015-08-03T20:37:32.564Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34156" doReceive() ... 2015-08-03T20:37:32.577Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34157" doReceive() ... 2015-08-03T20:37:32.597Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34158" doReceive() ... 2015-08-03T20:37:32.602Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34159" doReceive() ... 2015-08-03T20:37:32.628Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34160" doReceive() ... 2015-08-03T20:37:32.629Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34161" doReceive() ... 2015-08-03T20:37:32.643Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34162" doReceive() ... 2015-08-03T20:37:32.648Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34163" doReceive() ... 2015-08-03T20:37:32.655Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34164" doReceive() ... 2015-08-03T20:37:32.663Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34165" doReceive() ... 2015-08-03T20:37:32.681Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34166" doReceive() ... 2015-08-03T20:37:32.698Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34167" doReceive() ... 2015-08-03T20:37:32.704Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34168" doReceive() ... 2015-08-03T20:37:32.707Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34169" doReceive() ... 2015-08-03T20:37:32.708Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34170" doReceive() ... 2015-08-03T20:37:32.711Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1007866, 1003933, 0 committed:false) SnapCreateDur 25.333µs SnapOpenDur 14.652504ms 2015-08-03T20:37:32.736Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34171" doReceive() ... 2015-08-03T20:37:32.765Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34172" doReceive() ... 2015-08-03T20:37:32.792Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34173" doReceive() ... 2015-08-03T20:37:32.821Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34174" doReceive() ... 2015-08-03T20:37:32.848Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34175" doReceive() ... 2015-08-03T20:37:32.876Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34176" doReceive() ... 2015-08-03T20:37:32.914Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34177" doReceive() ... 2015-08-03T20:37:32.945Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34178" doReceive() ... 2015-08-03T20:37:32.95Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1011790, 1005895, 0 committed:false) SnapCreateDur 31.44µs SnapOpenDur 22.251508ms 2015-08-03T20:37:32.972Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34179" doReceive() ... 2015-08-03T20:37:33.002Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34180" doReceive() ... 2015-08-03T20:37:33.031Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34181" doReceive() ... 2015-08-03T20:37:33.058Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34182" doReceive() ... 2015-08-03T20:37:33.108Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34183" doReceive() ... 2015-08-03T20:37:33.119Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34184" doReceive() ... 2015-08-03T20:37:33.143Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34185" doReceive() ... 2015-08-03T20:37:33.172Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34186" doReceive() ... 2015-08-03T20:37:33.199Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1015390, 1007695, 0 committed:false) SnapCreateDur 22.853µs SnapOpenDur 33.401212ms 2015-08-03T20:37:33.21Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34187" doReceive() ... 2015-08-03T20:37:33.236Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34188" doReceive() ... 2015-08-03T20:37:33.239Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1010000 2015-08-03T20:37:33.262Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34189" doReceive() ... 2015-08-03T20:37:33.291Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34190" doReceive() ... 2015-08-03T20:37:33.32Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34191" doReceive() ... 2015-08-03T20:37:33.349Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34192" doReceive() ... 2015-08-03T20:37:33.374Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1018986, 1009493, 0 committed:false) SnapCreateDur 32.743µs SnapOpenDur 46.432951ms 2015-08-03T20:37:33.385Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34193" doReceive() ... 2015-08-03T20:37:33.41Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34194" doReceive() ... 2015-08-03T20:37:33.441Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34195" doReceive() ... 2015-08-03T20:37:33.474Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34196" doReceive() ... 2015-08-03T20:37:33.479Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1010000 QueuedCount 1490 2015-08-03T20:37:33.502Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34197" doReceive() ... 2015-08-03T20:37:33.533Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34198" doReceive() ... 2015-08-03T20:37:33.556Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34199" doReceive() ... 2015-08-03T20:37:33.582Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34200" doReceive() ... 2015-08-03T20:37:33.59Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1022656, 1011328, 0 committed:false) SnapCreateDur 37.653µs SnapOpenDur 52.965003ms 2015-08-03T20:37:33.611Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34201" doReceive() ... 2015-08-03T20:37:33.641Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34202" doReceive() ... 2015-08-03T20:37:33.67Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34203" doReceive() ... 2015-08-03T20:37:33.697Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34204" doReceive() ... 2015-08-03T20:37:33.722Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34205" doReceive() ... 2015-08-03T20:37:33.747Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34207" doReceive() ... 2015-08-03T20:37:33.773Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34208" doReceive() ... 2015-08-03T20:37:33.8Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34209" doReceive() ... 2015-08-03T20:37:33.804Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1026260, 1013130, 0 committed:false) SnapCreateDur 28.53µs SnapOpenDur 66.312751ms 2015-08-03T20:37:33.829Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34210" doReceive() ... 2015-08-03T20:37:33.859Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34211" doReceive() ... 2015-08-03T20:37:33.888Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34212" doReceive() ... 2015-08-03T20:37:33.927Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34213" doReceive() ... 2015-08-03T20:37:33.955Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34214" doReceive() ... 2015-08-03T20:37:33.982Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34215" doReceive() ... 2015-08-03T20:37:34.009Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1029948, 1014974, 0 committed:false) SnapCreateDur 24.18µs SnapOpenDur 71.332937ms 2015-08-03T20:37:34.009Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34216" doReceive() ... 2015-08-03T20:37:34.036Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34217" doReceive() ... 2015-08-03T20:37:34.064Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34218" doReceive() ... 2015-08-03T20:37:34.089Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34219" doReceive() ... 2015-08-03T20:37:34.114Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34220" doReceive() ... 2015-08-03T20:37:34.139Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34221" doReceive() ... 2015-08-03T20:37:34.166Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34222" doReceive() ... 2015-08-03T20:37:34.191Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34223" doReceive() ... 2015-08-03T20:37:34.217Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34224" doReceive() ... 2015-08-03T20:37:34.221Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1033726, 1016863, 0 committed:false) SnapCreateDur 33.336µs SnapOpenDur 83.85327ms 2015-08-03T20:37:34.242Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34225" doReceive() ... 2015-08-03T20:37:34.27Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34226" doReceive() ... 2015-08-03T20:37:34.297Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34227" doReceive() ... 2015-08-03T20:37:34.307Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1020000 2015-08-03T20:37:34.325Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34228" doReceive() ... 2015-08-03T20:37:34.353Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34229" doReceive() ... 2015-08-03T20:37:34.38Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34230" doReceive() ... 2015-08-03T20:37:34.403Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34231" doReceive() ... 2015-08-03T20:37:34.43Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34232" doReceive() ... 2015-08-03T20:37:34.43Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1037500, 1018750, 0 committed:false) SnapCreateDur 18.706µs SnapOpenDur 92.257247ms 2015-08-03T20:37:34.455Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34233" doReceive() ... 2015-08-03T20:37:34.481Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34234" doReceive() ... 2015-08-03T20:37:34.501Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1020000 QueuedCount 947 2015-08-03T20:37:34.506Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34235" doReceive() ... 2015-08-03T20:37:34.53Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34236" doReceive() ... 2015-08-03T20:37:34.556Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34237" doReceive() ... 2015-08-03T20:37:34.586Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34238" doReceive() ... 2015-08-03T20:37:34.622Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34239" doReceive() ... 2015-08-03T20:37:34.641Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34240" doReceive() ... 2015-08-03T20:37:34.653Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1041240, 1020620, 0 committed:false) SnapCreateDur 35.614µs SnapOpenDur 115.906605ms 2015-08-03T20:37:34.67Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34241" doReceive() ... 2015-08-03T20:37:34.691Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34242" doReceive() ... 2015-08-03T20:37:34.716Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34243" doReceive() ... 2015-08-03T20:37:34.741Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34244" doReceive() ... 2015-08-03T20:37:34.767Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34245" doReceive() ... 2015-08-03T20:37:34.789Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34246" doReceive() ... 2015-08-03T20:37:34.815Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34247" doReceive() ... 2015-08-03T20:37:34.842Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34248" doReceive() ... 2015-08-03T20:37:34.865Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1044876, 1022438, 0 committed:false) SnapCreateDur 35.697µs SnapOpenDur 117.198341ms 2015-08-03T20:37:34.879Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34249" doReceive() ... 2015-08-03T20:37:34.897Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34250" doReceive() ... 2015-08-03T20:37:34.93Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34251" doReceive() ... 2015-08-03T20:37:34.955Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34252" doReceive() ... 2015-08-03T20:37:34.984Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34253" doReceive() ... 2015-08-03T20:37:35.011Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34254" doReceive() ... 2015-08-03T20:37:35.035Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34255" doReceive() ... 2015-08-03T20:37:35.06Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34256" doReceive() ... 2015-08-03T20:37:35.085Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34257" doReceive() ... 2015-08-03T20:37:35.109Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34258" doReceive() ... 2015-08-03T20:37:35.133Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34259" doReceive() ... 2015-08-03T20:37:35.158Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34260" doReceive() ... 2015-08-03T20:37:35.184Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34261" doReceive() ... 2015-08-03T20:37:35.209Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34262" doReceive() ... 2015-08-03T20:37:35.236Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34263" doReceive() ... 2015-08-03T20:37:35.264Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34264" doReceive() ... 2015-08-03T20:37:35.29Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34265" doReceive() ... 2015-08-03T20:37:35.318Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34266" doReceive() ... 2015-08-03T20:37:35.345Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34267" doReceive() ... 2015-08-03T20:37:35.347Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1030000 2015-08-03T20:37:35.37Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34268" doReceive() ... 2015-08-03T20:37:35.395Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34269" doReceive() ... 2015-08-03T20:37:35.423Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34270" doReceive() ... 2015-08-03T20:37:35.452Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34271" doReceive() ... 2015-08-03T20:37:35.479Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34272" doReceive() ... 2015-08-03T20:37:35.504Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34273" doReceive() ... 2015-08-03T20:37:35.529Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34274" doReceive() ... 2015-08-03T20:37:35.554Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34275" doReceive() ... 2015-08-03T20:37:35.581Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34276" doReceive() ... 2015-08-03T20:37:35.61Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34277" doReceive() ... 2015-08-03T20:37:35.636Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34278" doReceive() ... 2015-08-03T20:37:35.664Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34279" doReceive() ... 2015-08-03T20:37:35.695Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34280" doReceive() ... 2015-08-03T20:37:35.723Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34281" doReceive() ... 2015-08-03T20:37:35.751Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34282" doReceive() ... 2015-08-03T20:37:35.779Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34283" doReceive() ... 2015-08-03T20:37:35.807Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34284" doReceive() ... 2015-08-03T20:37:35.834Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34285" doReceive() ... 2015-08-03T20:37:35.861Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34286" doReceive() ... 2015-08-03T20:37:35.893Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34287" doReceive() ... 2015-08-03T20:37:35.921Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34288" doReceive() ... 2015-08-03T20:37:35.954Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34289" doReceive() ... 2015-08-03T20:37:35.981Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34290" doReceive() ... 2015-08-03T20:37:36.009Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34291" doReceive() ... 2015-08-03T20:37:36.037Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34292" doReceive() ... 2015-08-03T20:37:36.067Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34293" doReceive() ... 2015-08-03T20:37:36.099Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34294" doReceive() ... 2015-08-03T20:37:36.13Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34295" doReceive() ... 2015-08-03T20:37:36.166Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34296" doReceive() ... 2015-08-03T20:37:36.196Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34297" doReceive() ... 2015-08-03T20:37:36.226Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34298" doReceive() ... 2015-08-03T20:37:36.255Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34299" doReceive() ... 2015-08-03T20:37:36.265Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.053µs CommitTime 280.095714ms TotalFlushTime 45.457374332s TotalCommitTime 4.85952998s 2015-08-03T20:37:36.266Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1048768, 1024384, 11 committed:true) SnapCreateDur 305.437435ms SnapOpenDur 63.543µs 2015-08-03T20:37:36.288Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34300" doReceive() ... 2015-08-03T20:37:36.324Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34301" doReceive() ... 2015-08-03T20:37:36.355Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34302" doReceive() ... 2015-08-03T20:37:36.386Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34303" doReceive() ... 2015-08-03T20:37:36.403Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1040000 2015-08-03T20:37:36.413Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34304" doReceive() ... 2015-08-03T20:37:36.434Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1056338, 1028169, 0 committed:false) SnapCreateDur 24.533µs SnapOpenDur 19.694457ms 2015-08-03T20:37:36.447Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34305" doReceive() ... 2015-08-03T20:37:36.481Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34306" doReceive() ... 2015-08-03T20:37:36.501Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1030000 QueuedCount 474 2015-08-03T20:37:36.512Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34307" doReceive() ... 2015-08-03T20:37:36.542Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34308" doReceive() ... 2015-08-03T20:37:36.549Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1059954, 1029977, 0 committed:false) SnapCreateDur 39.58µs SnapOpenDur 30.395957ms 2015-08-03T20:37:36.578Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34309" doReceive() ... 2015-08-03T20:37:36.61Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34310" doReceive() ... 2015-08-03T20:37:36.642Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34311" doReceive() ... 2015-08-03T20:37:36.674Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34312" doReceive() ... 2015-08-03T20:37:36.685Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1063830, 1031915, 0 committed:false) SnapCreateDur 35.147µs SnapOpenDur 40.610353ms 2015-08-03T20:37:36.705Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34313" doReceive() ... 2015-08-03T20:37:36.74Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34314" doReceive() ... 2015-08-03T20:37:36.775Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34315" doReceive() ... 2015-08-03T20:37:36.802Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34316" doReceive() ... 2015-08-03T20:37:36.83Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1067638, 1033819, 0 committed:false) SnapCreateDur 21.414µs SnapOpenDur 49.349452ms 2015-08-03T20:37:36.868Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34318" doReceive() ... 2015-08-03T20:37:36.882Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34319" doReceive() ... 2015-08-03T20:37:36.9Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34320" doReceive() ... 2015-08-03T20:37:36.936Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34321" doReceive() ... 2015-08-03T20:37:36.976Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34322" doReceive() ... 2015-08-03T20:37:36.987Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1071388, 1035694, 0 committed:false) SnapCreateDur 35.126µs SnapOpenDur 62.790544ms 2015-08-03T20:37:37.009Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34323" doReceive() ... 2015-08-03T20:37:37.042Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34324" doReceive() ... 2015-08-03T20:37:37.073Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34325" doReceive() ... 2015-08-03T20:37:37.104Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34326" doReceive() ... 2015-08-03T20:37:37.14Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34327" doReceive() ... 2015-08-03T20:37:37.156Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1075126, 1037563, 0 committed:false) SnapCreateDur 24.334µs SnapOpenDur 74.726168ms 2015-08-03T20:37:37.173Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34328" doReceive() ... 2015-08-03T20:37:37.209Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34329" doReceive() ... 2015-08-03T20:37:37.238Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34330" doReceive() ... 2015-08-03T20:37:37.269Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34331" doReceive() ... 2015-08-03T20:37:37.272Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1040000 QueuedCount 90 2015-08-03T20:37:37.297Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34332" doReceive() ... 2015-08-03T20:37:37.328Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34333" doReceive() ... 2015-08-03T20:37:37.358Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34334" doReceive() ... 2015-08-03T20:37:37.365Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1078890, 1039445, 0 committed:false) SnapCreateDur 21.4µs SnapOpenDur 89.122205ms 2015-08-03T20:37:37.386Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34335" doReceive() ... 2015-08-03T20:37:37.416Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34336" doReceive() ... 2015-08-03T20:37:37.441Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1050000 2015-08-03T20:37:37.453Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34337" doReceive() ... 2015-08-03T20:37:37.486Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34338" doReceive() ... 2015-08-03T20:37:37.516Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34339" doReceive() ... 2015-08-03T20:37:37.546Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34340" doReceive() ... 2015-08-03T20:37:37.566Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1082670, 1041335, 0 committed:false) SnapCreateDur 23.013µs SnapOpenDur 96.047913ms 2015-08-03T20:37:37.583Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34341" doReceive() ... 2015-08-03T20:37:37.615Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34342" doReceive() ... 2015-08-03T20:37:37.645Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34343" doReceive() ... 2015-08-03T20:37:37.674Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34344" doReceive() ... 2015-08-03T20:37:37.703Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34345" doReceive() ... 2015-08-03T20:37:37.735Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34346" doReceive() ... 2015-08-03T20:37:37.764Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34347" doReceive() ... 2015-08-03T20:37:37.781Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1086462, 1043231, 0 committed:false) SnapCreateDur 29.083µs SnapOpenDur 119.537442ms 2015-08-03T20:37:37.795Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34348" doReceive() ... 2015-08-03T20:37:37.823Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34349" doReceive() ... 2015-08-03T20:37:37.855Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34350" doReceive() ... 2015-08-03T20:37:37.888Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34351" doReceive() ... 2015-08-03T20:37:37.921Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34352" doReceive() ... 2015-08-03T20:37:37.955Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34353" doReceive() ... 2015-08-03T20:37:37.992Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34354" doReceive() ... 2015-08-03T20:37:38.02Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34355" doReceive() ... 2015-08-03T20:37:38.052Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34356" doReceive() ... 2015-08-03T20:37:38.082Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34357" doReceive() ... 2015-08-03T20:37:38.114Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34358" doReceive() ... 2015-08-03T20:37:38.146Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34359" doReceive() ... 2015-08-03T20:37:38.18Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34360" doReceive() ... 2015-08-03T20:37:38.212Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34361" doReceive() ... 2015-08-03T20:37:38.242Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34362" doReceive() ... 2015-08-03T20:37:38.273Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34363" doReceive() ... 2015-08-03T20:37:38.306Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34364" doReceive() ... 2015-08-03T20:37:38.343Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34365" doReceive() ... 2015-08-03T20:37:38.376Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34366" doReceive() ... 2015-08-03T20:37:38.409Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34367" doReceive() ... 2015-08-03T20:37:38.44Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34368" doReceive() ... 2015-08-03T20:37:38.481Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34369" doReceive() ... 2015-08-03T20:37:38.486Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1060000 2015-08-03T20:37:38.503Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34370" doReceive() ... 2015-08-03T20:37:38.535Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34371" doReceive() ... 2015-08-03T20:37:38.566Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34372" doReceive() ... 2015-08-03T20:37:38.597Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34373" doReceive() ... 2015-08-03T20:37:38.627Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34374" doReceive() ... 2015-08-03T20:37:38.66Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34375" doReceive() ... 2015-08-03T20:37:38.695Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34376" doReceive() ... 2015-08-03T20:37:38.728Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34377" doReceive() ... 2015-08-03T20:37:38.759Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34378" doReceive() ... 2015-08-03T20:37:38.789Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34379" doReceive() ... 2015-08-03T20:37:38.822Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34380" doReceive() ... 2015-08-03T20:37:38.853Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34381" doReceive() ... 2015-08-03T20:37:38.886Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34382" doReceive() ... 2015-08-03T20:37:38.918Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34383" doReceive() ... 2015-08-03T20:37:38.951Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34384" doReceive() ... 2015-08-03T20:37:38.981Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34385" doReceive() ... 2015-08-03T20:37:39.025Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34386" doReceive() ... 2015-08-03T20:37:39.029Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1090198, 1045099, 0 committed:false) SnapCreateDur 31.44µs SnapOpenDur 83.088165ms 2015-08-03T20:37:39.055Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34387" doReceive() ... 2015-08-03T20:37:39.087Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34388" doReceive() ... 2015-08-03T20:37:39.117Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34389" doReceive() ... 2015-08-03T20:37:39.146Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34390" doReceive() ... 2015-08-03T20:37:39.174Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34391" doReceive() ... 2015-08-03T20:37:39.204Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34392" doReceive() ... 2015-08-03T20:37:39.236Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34393" doReceive() ... 2015-08-03T20:37:39.266Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34394" doReceive() ... 2015-08-03T20:37:39.297Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34395" doReceive() ... 2015-08-03T20:37:39.326Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34396" doReceive() ... 2015-08-03T20:37:39.353Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34397" doReceive() ... 2015-08-03T20:37:39.376Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.198µs CommitTime 247.410542ms TotalFlushTime 47.321898382s TotalCommitTime 5.106940522s 2015-08-03T20:37:39.376Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1093514, 1046757, 12 committed:true) SnapCreateDur 263.216148ms SnapOpenDur 48.518µs 2015-08-03T20:37:39.386Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34398" doReceive() ... 2015-08-03T20:37:39.411Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34399" doReceive() ... 2015-08-03T20:37:39.442Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34400" doReceive() ... 2015-08-03T20:37:39.473Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34401" doReceive() ... 2015-08-03T20:37:39.478Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1050000 QueuedCount 1516 2015-08-03T20:37:39.501Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34402" doReceive() ... 2015-08-03T20:37:39.531Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1070000 2015-08-03T20:37:39.532Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34403" doReceive() ... 2015-08-03T20:37:39.544Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1101424, 1050712, 0 committed:false) SnapCreateDur 20.234µs SnapOpenDur 19.338893ms 2015-08-03T20:37:39.566Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34404" doReceive() ... 2015-08-03T20:37:39.593Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34405" doReceive() ... 2015-08-03T20:37:39.621Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34406" doReceive() ... 2015-08-03T20:37:39.65Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34407" doReceive() ... 2015-08-03T20:37:39.682Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34408" doReceive() ... 2015-08-03T20:37:39.71Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34409" doReceive() ... 2015-08-03T20:37:39.734Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1109076, 1054538, 0 committed:false) SnapCreateDur 23.984µs SnapOpenDur 40.389769ms 2015-08-03T20:37:39.745Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34410" doReceive() ... 2015-08-03T20:37:39.771Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34411" doReceive() ... 2015-08-03T20:37:39.803Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34412" doReceive() ... 2015-08-03T20:37:39.834Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34413" doReceive() ... 2015-08-03T20:37:39.861Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34414" doReceive() ... 2015-08-03T20:37:39.891Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34415" doReceive() ... 2015-08-03T20:37:39.925Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34416" doReceive() ... 2015-08-03T20:37:39.942Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1116536, 1058268, 0 committed:false) SnapCreateDur 25.25µs SnapOpenDur 60.157817ms 2015-08-03T20:37:39.959Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34417" doReceive() ... 2015-08-03T20:37:39.991Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34418" doReceive() ... 2015-08-03T20:37:39.993Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1060000 QueuedCount 3014 2015-08-03T20:37:40.034Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34419" doReceive() ... 2015-08-03T20:37:40.062Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34420" doReceive() ... 2015-08-03T20:37:40.093Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34421" doReceive() ... 2015-08-03T20:37:40.127Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34422" doReceive() ... 2015-08-03T20:37:40.157Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34423" doReceive() ... 2015-08-03T20:37:40.183Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1124054, 1062027, 0 committed:false) SnapCreateDur 32.976µs SnapOpenDur 82.681879ms 2015-08-03T20:37:40.194Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34424" doReceive() ... 2015-08-03T20:37:40.22Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34425" doReceive() ... 2015-08-03T20:37:40.251Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34426" doReceive() ... 2015-08-03T20:37:40.283Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34427" doReceive() ... 2015-08-03T20:37:40.316Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34428" doReceive() ... 2015-08-03T20:37:40.347Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34429" doReceive() ... 2015-08-03T20:37:40.369Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1127862, 1063931, 0 committed:false) SnapCreateDur 29.993µs SnapOpenDur 91.635298ms 2015-08-03T20:37:40.382Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34430" doReceive() ... 2015-08-03T20:37:40.412Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34431" doReceive() ... 2015-08-03T20:37:40.447Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34432" doReceive() ... 2015-08-03T20:37:40.482Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34433" doReceive() ... 2015-08-03T20:37:40.514Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34434" doReceive() ... 2015-08-03T20:37:40.551Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34435" doReceive() ... 2015-08-03T20:37:40.565Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1080000 2015-08-03T20:37:40.576Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1131612, 1065806, 0 committed:false) SnapCreateDur 32.603µs SnapOpenDur 112.619716ms 2015-08-03T20:37:40.587Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34436" doReceive() ... 2015-08-03T20:37:40.618Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34437" doReceive() ... 2015-08-03T20:37:40.649Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34438" doReceive() ... 2015-08-03T20:37:40.684Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34439" doReceive() ... 2015-08-03T20:37:40.714Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34440" doReceive() ... 2015-08-03T20:37:40.744Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34441" doReceive() ... 2015-08-03T20:37:40.778Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34442" doReceive() ... 2015-08-03T20:37:40.79Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1135436, 1067718, 0 committed:false) SnapCreateDur 23.733µs SnapOpenDur 118.675716ms 2015-08-03T20:37:40.809Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34443" doReceive() ... 2015-08-03T20:37:40.841Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34444" doReceive() ... 2015-08-03T20:37:40.876Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34445" doReceive() ... 2015-08-03T20:37:40.909Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34446" doReceive() ... 2015-08-03T20:37:40.942Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34447" doReceive() ... 2015-08-03T20:37:40.971Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34448" doReceive() ... 2015-08-03T20:37:41.001Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34449" doReceive() ... 2015-08-03T20:37:41.044Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34450" doReceive() ... 2015-08-03T20:37:41.075Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34451" doReceive() ... 2015-08-03T20:37:41.108Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34452" doReceive() ... 2015-08-03T20:37:41.138Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34453" doReceive() ... 2015-08-03T20:37:41.176Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34454" doReceive() ... 2015-08-03T20:37:41.213Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34455" doReceive() ... 2015-08-03T20:37:41.246Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34456" doReceive() ... 2015-08-03T20:37:41.279Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34457" doReceive() ... 2015-08-03T20:37:41.317Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34458" doReceive() ... 2015-08-03T20:37:41.355Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34459" doReceive() ... 2015-08-03T20:37:41.393Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34460" doReceive() ... 2015-08-03T20:37:41.425Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34461" doReceive() ... 2015-08-03T20:37:41.456Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34462" doReceive() ... 2015-08-03T20:37:41.488Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34463" doReceive() ... 2015-08-03T20:37:41.562Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34464" doReceive() ... 2015-08-03T20:37:41.566Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34465" doReceive() ... 2015-08-03T20:37:41.6Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34466" doReceive() ... 2015-08-03T20:37:41.611Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1090000 2015-08-03T20:37:41.633Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34467" doReceive() ... 2015-08-03T20:37:41.669Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34468" doReceive() ... 2015-08-03T20:37:41.704Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34469" doReceive() ... 2015-08-03T20:37:41.738Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34470" doReceive() ... 2015-08-03T20:37:41.778Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34471" doReceive() ... 2015-08-03T20:37:41.812Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34472" doReceive() ... 2015-08-03T20:37:41.848Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34473" doReceive() ... 2015-08-03T20:37:41.883Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34474" doReceive() ... 2015-08-03T20:37:41.915Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34475" doReceive() ... 2015-08-03T20:37:41.953Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34476" doReceive() ... 2015-08-03T20:37:41.99Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34477" doReceive() ... 2015-08-03T20:37:42.025Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34478" doReceive() ... 2015-08-03T20:37:42.04Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1070000 QueuedCount 700 2015-08-03T20:37:42.07Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34479" doReceive() ... 2015-08-03T20:37:42.106Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34480" doReceive() ... 2015-08-03T20:37:42.142Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34481" doReceive() ... 2015-08-03T20:37:42.156Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1139216, 1069608, 0 committed:false) SnapCreateDur 24.983µs SnapOpenDur 90.622113ms 2015-08-03T20:37:42.181Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34482" doReceive() ... 2015-08-03T20:37:42.214Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34483" doReceive() ... 2015-08-03T20:37:42.253Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34484" doReceive() ... 2015-08-03T20:37:42.289Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34485" doReceive() ... 2015-08-03T20:37:42.331Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34486" doReceive() ... 2015-08-03T20:37:42.368Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34487" doReceive() ... 2015-08-03T20:37:42.401Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34488" doReceive() ... 2015-08-03T20:37:42.437Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34489" doReceive() ... 2015-08-03T20:37:42.474Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34490" doReceive() ... 2015-08-03T20:37:42.512Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34491" doReceive() ... 2015-08-03T20:37:42.545Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34492" doReceive() ... 2015-08-03T20:37:42.578Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34493" doReceive() ... 2015-08-03T20:37:42.61Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34494" doReceive() ... 2015-08-03T20:37:42.651Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34495" doReceive() ... 2015-08-03T20:37:42.652Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1100000 2015-08-03T20:37:42.687Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34496" doReceive() ... 2015-08-03T20:37:42.693Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.957µs CommitTime 413.896025ms TotalFlushTime 49.316651514s TotalCommitTime 5.520836547s 2015-08-03T20:37:42.693Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1143006, 1071503, 13 committed:true) SnapCreateDur 439.975787ms SnapOpenDur 55.13µs 2015-08-03T20:37:42.721Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34497" doReceive() ... 2015-08-03T20:37:42.757Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34498" doReceive() ... 2015-08-03T20:37:42.798Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34499" doReceive() ... 2015-08-03T20:37:42.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34500" doReceive() ... 2015-08-03T20:37:42.872Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34501" doReceive() ... 2015-08-03T20:37:42.905Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34502" doReceive() ... 2015-08-03T20:37:42.935Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1154424, 1077212, 0 committed:false) SnapCreateDur 25.368µs SnapOpenDur 28.976132ms 2015-08-03T20:37:42.961Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34503" doReceive() ... 2015-08-03T20:37:42.971Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34504" doReceive() ... 2015-08-03T20:37:43.008Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34505" doReceive() ... 2015-08-03T20:37:43.009Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1080000 QueuedCount 2229 2015-08-03T20:37:43.044Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34506" doReceive() ... 2015-08-03T20:37:43.087Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34507" doReceive() ... 2015-08-03T20:37:43.117Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34508" doReceive() ... 2015-08-03T20:37:43.133Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1161910, 1080955, 0 committed:false) SnapCreateDur 23.126µs SnapOpenDur 48.180598ms 2015-08-03T20:37:43.157Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34509" doReceive() ... 2015-08-03T20:37:43.195Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34510" doReceive() ... 2015-08-03T20:37:43.227Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34511" doReceive() ... 2015-08-03T20:37:43.263Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34512" doReceive() ... 2015-08-03T20:37:43.297Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34513" doReceive() ... 2015-08-03T20:37:43.334Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34514" doReceive() ... 2015-08-03T20:37:43.365Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1169504, 1084752, 0 committed:false) SnapCreateDur 25.307µs SnapOpenDur 74.455897ms 2015-08-03T20:37:43.377Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34515" doReceive() ... 2015-08-03T20:37:43.412Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34516" doReceive() ... 2015-08-03T20:37:43.45Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34517" doReceive() ... 2015-08-03T20:37:43.487Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34518" doReceive() ... 2015-08-03T20:37:43.525Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34519" doReceive() ... 2015-08-03T20:37:43.566Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34520" doReceive() ... 2015-08-03T20:37:43.608Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34521" doReceive() ... 2015-08-03T20:37:43.628Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1177026, 1088513, 0 committed:false) SnapCreateDur 26.737µs SnapOpenDur 104.512443ms 2015-08-03T20:37:43.645Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34522" doReceive() ... 2015-08-03T20:37:43.654Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1090000 QueuedCount 1892 2015-08-03T20:37:43.687Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34523" doReceive() ... 2015-08-03T20:37:43.688Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1110000 2015-08-03T20:37:43.721Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34524" doReceive() ... 2015-08-03T20:37:43.76Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34525" doReceive() ... 2015-08-03T20:37:43.798Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34526" doReceive() ... 2015-08-03T20:37:43.836Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34527" doReceive() ... 2015-08-03T20:37:43.872Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34528" doReceive() ... 2015-08-03T20:37:43.912Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34529" doReceive() ... 2015-08-03T20:37:43.948Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:34530" doReceive() ... 2015-08-03T20:37:44.735Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1120000 2015-08-03T20:37:44.754Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1184624, 1092312, 0 committed:false) SnapCreateDur 22.796µs SnapOpenDur 86.539858ms 2015-08-03T20:37:45.084Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1196008, 1098004, 0 committed:false) SnapCreateDur 28.12µs SnapOpenDur 117.638829ms 2015-08-03T20:37:45.119Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1100000 QueuedCount 1448 2015-08-03T20:37:45.328Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1203430, 1101715, 0 committed:false) SnapCreateDur 39.7µs SnapOpenDur 144.007761ms 2015-08-03T20:37:45.774Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1130000 2015-08-03T20:37:46.286Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.857µs CommitTime 835.478641ms TotalFlushTime 51.195905343s TotalCommitTime 6.356315188s 2015-08-03T20:37:46.286Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1207334, 1103667, 14 committed:true) SnapCreateDur 861.504648ms SnapOpenDur 34.289µs 2015-08-03T20:37:46.46Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1110000 QueuedCount 1035 2015-08-03T20:37:46.528Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1218616, 1109308, 0 committed:false) SnapCreateDur 25.906µs SnapOpenDur 29.814243ms 2015-08-03T20:37:46.813Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1230038, 1115019, 0 committed:false) SnapCreateDur 26.654µs SnapOpenDur 62.943867ms 2015-08-03T20:37:46.815Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1140000 2015-08-03T20:37:46.946Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1120000 QueuedCount 711 2015-08-03T20:37:47.057Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1237618, 1118809, 0 committed:false) SnapCreateDur 23.697µs SnapOpenDur 84.550918ms 2015-08-03T20:37:47.338Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1245106, 1122553, 0 committed:false) SnapCreateDur 25.52µs SnapOpenDur 118.900313ms 2015-08-03T20:37:47.866Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1150000 2015-08-03T20:37:48.572Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1252636, 1126318, 0 committed:false) SnapCreateDur 21.98µs SnapOpenDur 92.596182ms 2015-08-03T20:37:48.651Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1130000 QueuedCount 291 2015-08-03T20:37:48.913Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1160000 2015-08-03T20:37:49.094Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.026µs CommitTime 400.495481ms TotalFlushTime 52.992646862s TotalCommitTime 6.756810669s 2015-08-03T20:37:49.095Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1256464, 1128232, 15 committed:true) SnapCreateDur 426.842382ms SnapOpenDur 57.557µs 2015-08-03T20:37:49.336Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1267772, 1133886, 0 committed:false) SnapCreateDur 34.577µs SnapOpenDur 29.496371ms 2015-08-03T20:37:49.495Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1140000 QueuedCount 1754 2015-08-03T20:37:49.625Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1279018, 1139509, 0 committed:false) SnapCreateDur 36.823µs SnapOpenDur 67.028473ms 2015-08-03T20:37:49.949Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1170000 2015-08-03T20:37:49.954Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1290254, 1145127, 0 committed:false) SnapCreateDur 31.003µs SnapOpenDur 95.996204ms 2015-08-03T20:37:50.078Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1150000 QueuedCount 1285 2015-08-03T20:37:50.987Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1180000 2015-08-03T20:37:51.162Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1297784, 1148892, 0 committed:false) SnapCreateDur 21.3µs SnapOpenDur 86.268915ms 2015-08-03T20:37:51.506Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1309090, 1154545, 0 committed:false) SnapCreateDur 25.55µs SnapOpenDur 119.57195ms 2015-08-03T20:37:51.636Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1160000 QueuedCount 2803 2015-08-03T20:37:51.898Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1320462, 1160231, 0 committed:false) SnapCreateDur 42.353µs SnapOpenDur 158.189604ms 2015-08-03T20:37:52.034Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1190000 2015-08-03T20:37:52.888Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 19.397µs CommitTime 867.725548ms TotalFlushTime 55.09848011s TotalCommitTime 7.624536217s 2015-08-03T20:37:52.888Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1324234, 1162117, 16 committed:true) SnapCreateDur 895.31297ms SnapOpenDur 36.15µs 2015-08-03T20:37:53.072Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1200000 2015-08-03T20:37:53.114Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1170000 QueuedCount 510 2015-08-03T20:37:53.173Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1335630, 1167815, 0 committed:false) SnapCreateDur 26.22µs SnapOpenDur 35.995211ms 2015-08-03T20:37:53.463Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1346960, 1173480, 0 committed:false) SnapCreateDur 37.123µs SnapOpenDur 63.63394ms 2015-08-03T20:37:53.613Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1180000 QueuedCount 2038 2015-08-03T20:37:53.783Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1358338, 1179169, 0 committed:false) SnapCreateDur 29.086µs SnapOpenDur 97.45447ms 2015-08-03T20:37:54.112Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1210000 2015-08-03T20:37:55.02Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1365884, 1182942, 0 committed:false) SnapCreateDur 25.294µs SnapOpenDur 58.996339ms 2015-08-03T20:37:55.154Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1220000 2015-08-03T20:37:55.424Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.7µs CommitTime 312.048591ms TotalFlushTime 56.883302364s TotalCommitTime 7.936584808s 2015-08-03T20:37:55.424Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1369544, 1184772, 17 committed:true) SnapCreateDur 338.21355ms SnapOpenDur 44.747µs 2015-08-03T20:37:55.517Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1190000 QueuedCount 3547 2015-08-03T20:37:55.657Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1381008, 1190504, 0 committed:false) SnapCreateDur 19.084µs SnapOpenDur 21.38252ms 2015-08-03T20:37:55.942Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1392504, 1196252, 0 committed:false) SnapCreateDur 35.048µs SnapOpenDur 61.755482ms 2015-08-03T20:37:55.986Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1200000 QueuedCount 1201 2015-08-03T20:37:56.202Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1230000 2015-08-03T20:37:56.262Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1403748, 1201874, 0 committed:false) SnapCreateDur 25.683µs SnapOpenDur 97.871225ms 2015-08-03T20:37:57.253Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1240000 2015-08-03T20:37:57.443Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1210000 QueuedCount 858 2015-08-03T20:37:57.565Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1415094, 1207547, 0 committed:false) SnapCreateDur 21.9µs SnapOpenDur 96.921745ms 2015-08-03T20:37:57.905Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1426434, 1213217, 0 committed:false) SnapCreateDur 22.977µs SnapOpenDur 128.436969ms 2015-08-03T20:37:58.046Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1220000 QueuedCount 2379 2015-08-03T20:37:58.287Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1250000 2015-08-03T20:37:58.298Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1437762, 1218881, 0 committed:false) SnapCreateDur 38.46µs SnapOpenDur 171.372922ms 2015-08-03T20:37:59.329Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1260000 2015-08-03T20:37:59.429Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.71µs CommitTime 995.458276ms TotalFlushTime 59.052840438s TotalCommitTime 8.932043084s 2015-08-03T20:37:59.429Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1441552, 1220776, 18 committed:true) SnapCreateDur 1.024283664s SnapOpenDur 46.684µs 2015-08-03T20:37:59.458Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:345220239, Disk:1050173440, ExtraSnapshotData:0, Fragmentation:67.12731194192075% 2015-08-03T20:37:59.458Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:37:59.482Z-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 1207334. Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T20:37:59.766Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1230000 QueuedCount 77 2015-08-03T20:37:59.794Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1452888, 1226444, 0 committed:false) SnapCreateDur 25.46µs SnapOpenDur 23.455114ms 2015-08-03T20:38:00.02Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1463972, 1231986, 0 committed:false) SnapCreateDur 24.063µs SnapOpenDur 54.227261ms 2015-08-03T20:38:00.2Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1240000 QueuedCount 1460 2015-08-03T20:38:00.346Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1475266, 1237633, 0 committed:false) SnapCreateDur 23.34µs SnapOpenDur 89.634675ms 2015-08-03T20:38:00.378Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1270000 2015-08-03T20:38:01.412Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1280000 2015-08-03T20:38:01.602Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1486612, 1243306, 0 committed:false) SnapCreateDur 23.626µs SnapOpenDur 84.712896ms 2015-08-03T20:38:02.048Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.137µs CommitTime 354.872878ms TotalFlushTime 1m0.878964307s TotalCommitTime 9.286915962s 2015-08-03T20:38:02.048Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1490378, 1245189, 19 committed:true) SnapCreateDur 380.671752ms SnapOpenDur 35.696µs 2015-08-03T20:38:02.087Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1250000 QueuedCount 1051 2015-08-03T20:38:02.208Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1501820, 1250910, 0 committed:false) SnapCreateDur 21.537µs SnapOpenDur 28.188807ms 2015-08-03T20:38:02.399Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1260000 QueuedCount 664 2015-08-03T20:38:02.456Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1290000 2015-08-03T20:38:02.48Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1513060, 1256530, 0 committed:false) SnapCreateDur 21.737µs SnapOpenDur 59.586076ms 2015-08-03T20:38:02.784Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1524426, 1262213, 0 committed:false) SnapCreateDur 22.037µs SnapOpenDur 91.06669ms 2015-08-03T20:38:03.514Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1300000 2015-08-03T20:38:03.926Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1270000 QueuedCount 2249 2015-08-03T20:38:04.064Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1535880, 1267940, 0 committed:false) SnapCreateDur 22.82µs SnapOpenDur 88.105266ms 2015-08-03T20:38:04.398Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1547148, 1273574, 0 committed:false) SnapCreateDur 29.456µs SnapOpenDur 119.726135ms 2015-08-03T20:38:04.487Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1280000 QueuedCount 3776 2015-08-03T20:38:04.616Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1310000 2015-08-03T20:38:04.769Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1558560, 1279280, 0 committed:false) SnapCreateDur 18.843µs SnapOpenDur 149.651122ms 2015-08-03T20:38:05.21Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":2389327465,"bucket-1:by_email:avg_scan_wait_latency":2388727985,"bucket-1:by_email:avg_ts_interval":468515698,"bucket-1:by_email:avg_ts_items_count":5748,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":349013147,"bucket-1:by_email:delete_bytes":10440744,"bucket-1:by_email:disk_size":1066930176,"bucket-1:by_email:flush_queue_size":0,"bucket-1:by_email:frag_percent":67,"bucket-1:by_email:get_bytes":10902220,"bucket-1:by_email:insert_bytes":130494797,"bucket-1:by_email:items_count":1000000,"bucket-1:by_email:num_commits":19,"bucket-1:by_email:num_compactions":2,"bucket-1:by_email:num_completed_requests":3748,"bucket-1:by_email:num_docs_indexed":1246330,"bucket-1:by_email:num_docs_pending":435,"bucket-1:by_email:num_docs_queued":27336,"bucket-1:by_email:num_flush_queued":1285688,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":4132,"bucket-1:by_email:num_rows_returned":53,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":89,"bucket-1:by_email:scan_bytes_read":2120,"bucket-1:by_email:scan_wait_duration":9870224035318,"bucket-1:by_email:since_last_snapshot":371192904,"bucket-1:by_email:timings/storage_clone_handle":"3748 247337279 121530888568649","bucket-1:by_email:timings/storage_commit":"19 9286915962 5968697836523868724","bucket-1:by_email:timings/storage_del":"281162 3906776655 2034574038371167863","bucket-1:by_email:timings/storage_get":"1285688 13192266310 2074156410666738314","bucket-1:by_email:timings/storage_handle_open":"19 911110 48698562214","bucket-1:by_email:timings/storage_iterator_next":"53 923290 24428556830","bucket-1:by_email:timings/storage_new_iterator":"3748 298810812 49443206809288","bucket-1:by_email:timings/storage_set":"2562324 44390645925 5462966571447655015","bucket-1:by_email:timings/storage_snapshot_close":"69 828893149 12459438058324527","bucket-1:by_email:timings/storage_snapshot_create":"70 5284134169 499099751956195715","bucket-1:by_email:total_scan_duration":9872701088055,"bucket-1:mutation_queue_size":29837,"bucket-1:num_mutations_queued":1315525,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":15,"memory_quota":104857600000,"memory_used":449032192,"needs_restart":false,"num_connections":384,"timings/stats_response":"494 114849652 1499561626692928"} 2015-08-03T20:38:05.661Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1320000 2015-08-03T20:38:05.864Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 3.73µs CommitTime 972.132428ms TotalFlushTime 1m2.924937725s TotalCommitTime 10.25904839s 2015-08-03T20:38:05.864Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1562324, 1281162, 20 committed:true) SnapCreateDur 1.001247251s SnapOpenDur 24.555µs 2015-08-03T20:38:06.017Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1290000 QueuedCount 1423 2015-08-03T20:38:06.093Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1573596, 1286798, 0 committed:false) SnapCreateDur 31.767µs SnapOpenDur 28.109126ms 2015-08-03T20:38:06.351Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1584568, 1292284, 0 committed:false) SnapCreateDur 20.623µs SnapOpenDur 55.615927ms 2015-08-03T20:38:06.478Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1300000 QueuedCount 2349 2015-08-03T20:38:06.646Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1595100, 1297550, 0 committed:false) SnapCreateDur 28.633µs SnapOpenDur 84.527903ms 2015-08-03T20:38:06.714Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1330000 2015-08-03T20:38:07.73Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1605990, 1302995, 0 committed:false) SnapCreateDur 12.451µs SnapOpenDur 52.628958ms 2015-08-03T20:38:07.765Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1340000 2015-08-03T20:38:07.824Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1310000 QueuedCount 3708 2015-08-03T20:38:08.553Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 13.433µs CommitTime 588.668231ms TotalFlushTime 1m4.614712921s TotalCommitTime 10.847716621s 2015-08-03T20:38:08.553Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1617428, 1308714, 21 committed:true) SnapCreateDur 617.568962ms SnapOpenDur 59.82µs 2015-08-03T20:38:08.795Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1628720, 1314360, 0 committed:false) SnapCreateDur 23.08µs SnapOpenDur 29.676692ms 2015-08-03T20:38:08.816Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1350000 2015-08-03T20:38:08.836Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1320000 QueuedCount 1354 2015-08-03T20:38:09.086Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1640038, 1320019, 0 committed:false) SnapCreateDur 34.097µs SnapOpenDur 59.407269ms 2015-08-03T20:38:09.277Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1330000 QueuedCount 807 2015-08-03T20:38:09.405Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1651098, 1325549, 0 committed:false) SnapCreateDur 23.66µs SnapOpenDur 95.968707ms 2015-08-03T20:38:09.857Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1360000 2015-08-03T20:38:10.696Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1662402, 1331201, 0 committed:false) SnapCreateDur 20.72µs SnapOpenDur 89.819845ms 2015-08-03T20:38:10.796Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1340000 QueuedCount 2195 2015-08-03T20:38:10.903Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1370000 2015-08-03T20:38:10.955Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1673480, 1336740, 0 committed:false) SnapCreateDur 22.713µs SnapOpenDur 114.719729ms 2015-08-03T20:38:11.792Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 31.01µs CommitTime 712.313743ms TotalFlushTime 1m6.476543723s TotalCommitTime 11.560030364s 2015-08-03T20:38:11.792Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1677226, 1338613, 22 committed:true) SnapCreateDur 740.282478ms SnapOpenDur 44.299µs 2015-08-03T20:38:11.94Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1380000 2015-08-03T20:38:11.952Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1688544, 1344272, 0 committed:false) SnapCreateDur 15.989µs SnapOpenDur 16.869742ms 2015-08-03T20:38:11.979Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1350000 QueuedCount 1795 2015-08-03T20:38:12.233Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1699886, 1349943, 0 committed:false) SnapCreateDur 18.944µs SnapOpenDur 61.123541ms 2015-08-03T20:38:12.401Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1360000 QueuedCount 1399 2015-08-03T20:38:12.538Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1711236, 1355618, 0 committed:false) SnapCreateDur 20.498µs SnapOpenDur 88.852451ms 2015-08-03T20:38:12.981Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1390000 2015-08-03T20:38:14.023Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1400000 2015-08-03T20:38:14.653Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1718784, 1359392, 0 committed:false) SnapCreateDur 19.934µs SnapOpenDur 86.842544ms 2015-08-03T20:38:15.062Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1410000 2015-08-03T20:38:15.097Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.337µs CommitTime 333.754951ms TotalFlushTime 1m9.025922008s TotalCommitTime 11.893785315s 2015-08-03T20:38:15.097Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1722542, 1361271, 23 committed:true) SnapCreateDur 360.086567ms SnapOpenDur 38.72µs 2015-08-03T20:38:15.208Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1370000 QueuedCount 2958 2015-08-03T20:38:15.327Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1734044, 1367022, 0 committed:false) SnapCreateDur 30.337µs SnapOpenDur 28.136338ms 2015-08-03T20:38:15.602Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1745298, 1372649, 0 committed:false) SnapCreateDur 27.757µs SnapOpenDur 59.784863ms 2015-08-03T20:38:15.67Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1380000 QueuedCount 2580 2015-08-03T20:38:15.925Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1756664, 1378332, 0 committed:false) SnapCreateDur 22.636µs SnapOpenDur 91.505651ms 2015-08-03T20:38:16.109Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1420000 2015-08-03T20:38:17.07Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1390000 QueuedCount 217 2015-08-03T20:38:17.138Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1768030, 1384015, 0 committed:false) SnapCreateDur 23.245µs SnapOpenDur 61.578418ms 2015-08-03T20:38:17.155Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1430000 2015-08-03T20:38:17.468Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1779328, 1389664, 0 committed:false) SnapCreateDur 20.746µs SnapOpenDur 118.283399ms 2015-08-03T20:38:17.566Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1400000 QueuedCount 1769 2015-08-03T20:38:17.713Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1790762, 1395381, 0 committed:false) SnapCreateDur 23.846µs SnapOpenDur 105.314081ms 2015-08-03T20:38:18.191Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1440000 2015-08-03T20:38:18.812Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.927µs CommitTime 987.969613ms TotalFlushTime 1m11.035439015s TotalCommitTime 12.881754928s 2015-08-03T20:38:18.812Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1794312, 1397156, 24 committed:true) SnapCreateDur 1.014505382s SnapOpenDur 31.73µs 2015-08-03T20:38:18.984Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1805800, 1402900, 0 committed:false) SnapCreateDur 22.81µs SnapOpenDur 31.520366ms 2015-08-03T20:38:19.017Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1410000 QueuedCount 1379 2015-08-03T20:38:19.161Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1817082, 1408541, 0 committed:false) SnapCreateDur 13.439µs SnapOpenDur 36.544259ms 2015-08-03T20:38:19.236Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1450000 2015-08-03T20:38:19.35Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1420000 QueuedCount 906 2015-08-03T20:38:19.481Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1828362, 1414181, 0 committed:false) SnapCreateDur 30.764µs SnapOpenDur 95.792034ms 2015-08-03T20:38:20.282Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1460000 2015-08-03T20:38:20.736Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1839754, 1419877, 0 committed:false) SnapCreateDur 29.786µs SnapOpenDur 93.640639ms 2015-08-03T20:38:21.285Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.857µs CommitTime 435.52357ms TotalFlushTime 1m12.636064139s TotalCommitTime 13.317278498s 2015-08-03T20:38:21.324Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1470000 2015-08-03T20:38:21.883Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1843434, 1421717, 25 committed:true) SnapCreateDur 464.068522ms SnapOpenDur 598.427715ms 2015-08-03T20:38:21.95Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1430000 QueuedCount 2392 2015-08-03T20:38:22.122Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1854580, 1427290, 0 committed:false) SnapCreateDur 26.62µs SnapOpenDur 27.191505ms 2015-08-03T20:38:22.335Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1440000 QueuedCount 106 2015-08-03T20:38:22.361Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1480000 2015-08-03T20:38:22.409Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1866124, 1433062, 0 committed:false) SnapCreateDur 23.577µs SnapOpenDur 61.524ms 2015-08-03T20:38:22.736Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1877406, 1438703, 0 committed:false) SnapCreateDur 27.57µs SnapOpenDur 94.951476ms 2015-08-03T20:38:23.401Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1490000 2015-08-03T20:38:23.946Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1450000 QueuedCount 1588 2015-08-03T20:38:24.046Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1888720, 1444360, 0 committed:false) SnapCreateDur 13.952µs SnapOpenDur 67.645253ms 2015-08-03T20:38:24.437Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1500000 2015-08-03T20:38:24.633Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.077µs CommitTime 462.665221ms TotalFlushTime 1m14.478372002s TotalCommitTime 13.779943719s 2015-08-03T20:38:24.633Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1892534, 1446267, 26 committed:true) SnapCreateDur 489.558462ms SnapOpenDur 47.437µs 2015-08-03T20:38:24.895Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1903838, 1451919, 0 committed:false) SnapCreateDur 23.174µs SnapOpenDur 29.935652ms 2015-08-03T20:38:24.939Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1460000 QueuedCount 1200 2015-08-03T20:38:25.162Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1915168, 1457584, 0 committed:false) SnapCreateDur 29.94µs SnapOpenDur 55.423991ms 2015-08-03T20:38:25.344Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1470000 QueuedCount 825 2015-08-03T20:38:25.461Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1926562, 1463281, 0 committed:false) SnapCreateDur 23.106µs SnapOpenDur 87.104881ms 2015-08-03T20:38:25.485Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1510000 2015-08-03T20:38:26.515Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1520000 2015-08-03T20:38:26.663Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1937942, 1468971, 0 committed:false) SnapCreateDur 12.235µs SnapOpenDur 60.344226ms 2015-08-03T20:38:26.744Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1480000 QueuedCount 2360 2015-08-03T20:38:26.9Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1949264, 1474632, 0 committed:false) SnapCreateDur 18.936µs SnapOpenDur 114.552239ms 2015-08-03T20:38:27.272Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1960526, 1480263, 0 committed:false) SnapCreateDur 22.986µs SnapOpenDur 151.074238ms 2015-08-03T20:38:27.357Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1490000 QueuedCount 45 2015-08-03T20:38:27.574Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1530000 2015-08-03T20:38:28.338Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.266µs CommitTime 945.794755ms TotalFlushTime 1m16.484862775s TotalCommitTime 14.725738474s 2015-08-03T20:38:28.338Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1964360, 1482180, 27 committed:true) SnapCreateDur 971.914236ms SnapOpenDur 33.882µs 2015-08-03T20:38:28.61Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1540000 2015-08-03T20:38:28.623Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1975804, 1487902, 0 committed:false) SnapCreateDur 20.893µs SnapOpenDur 35.336292ms 2015-08-03T20:38:28.785Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1500000 QueuedCount 1622 2015-08-03T20:38:28.903Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1987142, 1493571, 0 committed:false) SnapCreateDur 32.317µs SnapOpenDur 59.576136ms 2015-08-03T20:38:29.206Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 1998468, 1499234, 0 committed:false) SnapCreateDur 26.517µs SnapOpenDur 88.745968ms 2015-08-03T20:38:29.331Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1510000 QueuedCount 3118 2015-08-03T20:38:29.66Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1550000 2015-08-03T20:38:30.687Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1560000 2015-08-03T20:38:31.733Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1570000 2015-08-03T20:38:31.909Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2009820, 1504910, 0 committed:false) SnapCreateDur 22.003µs SnapOpenDur 110.439404ms 2015-08-03T20:38:32.503Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.59µs CommitTime 473.064212ms TotalFlushTime 1m19.681658834s TotalCommitTime 15.198802686s 2015-08-03T20:38:32.503Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2013678, 1506839, 28 committed:true) SnapCreateDur 499.317615ms SnapOpenDur 49.774µs 2015-08-03T20:38:32.676Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1520000 QueuedCount 783 2015-08-03T20:38:32.738Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2024890, 1512445, 0 committed:false) SnapCreateDur 21.954µs SnapOpenDur 34.119007ms 2015-08-03T20:38:32.774Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1580000 2015-08-03T20:38:33.018Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2036144, 1518072, 0 committed:false) SnapCreateDur 21.804µs SnapOpenDur 55.229283ms 2015-08-03T20:38:33.814Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1590000 2015-08-03T20:38:34.001Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1530000 QueuedCount 2309 2015-08-03T20:38:34.177Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2047550, 1523775, 0 committed:false) SnapCreateDur 20.844µs SnapOpenDur 88.559271ms 2015-08-03T20:38:34.851Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1600000 2015-08-03T20:38:35.889Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1610000 2015-08-03T20:38:36.076Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2058772, 1529386, 0 committed:false) SnapCreateDur 21.667µs SnapOpenDur 98.979201ms 2015-08-03T20:38:36.643Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 3.817µs CommitTime 456.341535ms TotalFlushTime 1m22.898959382s TotalCommitTime 15.655144221s 2015-08-03T20:38:36.644Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2062586, 1531293, 29 committed:true) SnapCreateDur 483.779894ms SnapOpenDur 40.193µs 2015-08-03T20:38:36.667Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1540000 QueuedCount 1842 2015-08-03T20:38:36.81Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2073896, 1536948, 0 committed:false) SnapCreateDur 30.345µs SnapOpenDur 16.024992ms 2015-08-03T20:38:36.941Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1620000 2015-08-03T20:38:36.973Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1550000 QueuedCount 1422 2015-08-03T20:38:37.059Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2085192, 1542596, 0 committed:false) SnapCreateDur 18.342µs SnapOpenDur 34.590131ms 2015-08-03T20:38:37.357Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2096552, 1548276, 0 committed:false) SnapCreateDur 31.007µs SnapOpenDur 84.812192ms 2015-08-03T20:38:37.477Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1560000 QueuedCount 2959 2015-08-03T20:38:37.996Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1630000 2015-08-03T20:38:38.633Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2107940, 1553970, 0 committed:false) SnapCreateDur 35.987µs SnapOpenDur 96.020154ms 2015-08-03T20:38:39.036Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1640000 2015-08-03T20:38:40.083Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1650000 2015-08-03T20:38:40.147Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 2015-08-03T20:38:40.295Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2119266, 1559633, 0 committed:false) SnapCreateDur 26.761µs SnapOpenDur 1.45149611s 2015-08-03T20:38:40.36Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1570000 QueuedCount 661 2015-08-03T20:38:40.646Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 15.497µs CommitTime 229.915492ms TotalFlushTime 1m24.761033523s TotalCommitTime 15.885059713s 2015-08-03T20:38:40.646Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2123106, 1561553, 30 committed:true) SnapCreateDur 257.24977ms SnapOpenDur 43.023µs 2015-08-03T20:38:41.127Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1660000 2015-08-03T20:38:41.417Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2134432, 1567216, 0 committed:false) SnapCreateDur 32.097µs SnapOpenDur 557.485786ms 2015-08-03T20:38:41.56Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1580000 QueuedCount 2248 2015-08-03T20:38:41.7Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2145910, 1572955, 0 committed:false) SnapCreateDur 31.91µs SnapOpenDur 61.358364ms 2015-08-03T20:38:42.024Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2157118, 1578559, 0 committed:false) SnapCreateDur 32.387µs SnapOpenDur 80.810925ms 2015-08-03T20:38:42.127Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1590000 QueuedCount 3752 2015-08-03T20:38:42.162Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1670000 2015-08-03T20:38:43.202Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1680000 2015-08-03T20:38:43.265Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2168560, 1584280, 0 committed:false) SnapCreateDur 21.92µs SnapOpenDur 67.84849ms 2015-08-03T20:38:44.194Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.527µs CommitTime 692.131745ms TotalFlushTime 1m26.656089861s TotalCommitTime 16.577191458s 2015-08-03T20:38:44.194Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2179820, 1589910, 31 committed:true) SnapCreateDur 718.691832ms SnapOpenDur 47.987µs 2015-08-03T20:38:44.234Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1600000 QueuedCount 1446 2015-08-03T20:38:44.238Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1690000 2015-08-03T20:38:44.438Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2191368, 1595684, 0 committed:false) SnapCreateDur 26.083µs SnapOpenDur 31.192345ms 2015-08-03T20:38:44.611Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1610000 QueuedCount 1114 2015-08-03T20:38:44.708Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2202742, 1601371, 0 committed:false) SnapCreateDur 21.363µs SnapOpenDur 58.510033ms 2015-08-03T20:38:45.025Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2213846, 1606923, 0 committed:false) SnapCreateDur 24.894µs SnapOpenDur 93.910563ms 2015-08-03T20:38:45.166Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1620000 QueuedCount 2471 2015-08-03T20:38:45.284Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1700000 2015-08-03T20:38:46.334Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1710000 2015-08-03T20:38:46.495Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2225112, 1612556, 0 committed:false) SnapCreateDur 28.324µs SnapOpenDur 108.575696ms 2015-08-03T20:38:47.081Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.293µs CommitTime 454.893556ms TotalFlushTime 1m28.608920451s TotalCommitTime 17.032085014s 2015-08-03T20:38:47.082Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2228832, 1614416, 32 committed:true) SnapCreateDur 481.510632ms SnapOpenDur 62.333µs 2015-08-03T20:38:47.283Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1630000 QueuedCount 66 2015-08-03T20:38:47.321Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2240052, 1620026, 0 committed:false) SnapCreateDur 24.187µs SnapOpenDur 27.191646ms 2015-08-03T20:38:47.386Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1720000 2015-08-03T20:38:47.603Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2251470, 1625735, 0 committed:false) SnapCreateDur 24.824µs SnapOpenDur 60.069915ms 2015-08-03T20:38:47.771Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1640000 QueuedCount 1631 2015-08-03T20:38:47.935Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2262832, 1631416, 0 committed:false) SnapCreateDur 27.267µs SnapOpenDur 99.21218ms 2015-08-03T20:38:48.437Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1730000 2015-08-03T20:38:49.265Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2274216, 1637108, 0 committed:false) SnapCreateDur 22.38µs SnapOpenDur 117.220115ms 2015-08-03T20:38:49.376Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1650000 QueuedCount 3145 2015-08-03T20:38:49.476Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1740000 2015-08-03T20:38:49.623Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2285500, 1642750, 0 committed:false) SnapCreateDur 24.493µs SnapOpenDur 146.043602ms 2015-08-03T20:38:50.408Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 13.894µs CommitTime 694.133708ms TotalFlushTime 1m30.568481094s TotalCommitTime 17.726218722s 2015-08-03T20:38:50.408Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2289216, 1644608, 33 committed:true) SnapCreateDur 721.478702ms SnapOpenDur 30.219µs 2015-08-03T20:38:50.516Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1750000 2015-08-03T20:38:50.594Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1660000 QueuedCount 806 2015-08-03T20:38:50.651Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2300542, 1650271, 0 committed:false) SnapCreateDur 19.189µs SnapOpenDur 21.658316ms 2015-08-03T20:38:50.928Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2311894, 1655947, 0 committed:false) SnapCreateDur 21.929µs SnapOpenDur 55.212595ms 2015-08-03T20:38:51.118Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1670000 QueuedCount 2331 2015-08-03T20:38:51.341Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2323250, 1661625, 0 committed:false) SnapCreateDur 24.814µs SnapOpenDur 119.019481ms 2015-08-03T20:38:51.561Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1760000 2015-08-03T20:38:52.594Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1770000 2015-08-03T20:38:52.69Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2334528, 1667264, 0 committed:false) SnapCreateDur 26.734µs SnapOpenDur 115.601164ms 2015-08-03T20:38:52.779Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1680000 QueuedCount 1 2015-08-03T20:38:53.364Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.603µs CommitTime 552.695301ms TotalFlushTime 1m32.467232637s TotalCommitTime 18.278914023s 2015-08-03T20:38:53.364Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2338352, 1669176, 34 committed:true) SnapCreateDur 579.546788ms SnapOpenDur 58.886µs 2015-08-03T20:38:53.621Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2349714, 1674857, 0 committed:false) SnapCreateDur 21.803µs SnapOpenDur 32.35553ms 2015-08-03T20:38:53.632Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1780000 2015-08-03T20:38:53.802Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1690000 QueuedCount 1530 2015-08-03T20:38:53.932Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2361064, 1680532, 0 committed:false) SnapCreateDur 21.937µs SnapOpenDur 66.47054ms 2015-08-03T20:38:54.262Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2372418, 1686209, 0 committed:false) SnapCreateDur 22.78µs SnapOpenDur 97.387504ms 2015-08-03T20:38:54.382Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1700000 QueuedCount 3055 2015-08-03T20:38:54.678Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1790000 2015-08-03T20:38:55.501Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2383750, 1691875, 0 committed:false) SnapCreateDur 12.698µs SnapOpenDur 66.948535ms 2015-08-03T20:38:55.714Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1800000 2015-08-03T20:38:55.714Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2395068, 1697534, 0 committed:false) SnapCreateDur 15.313µs SnapOpenDur 83.062493ms 2015-08-03T20:38:55.776Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1710000 QueuedCount 691 2015-08-03T20:38:56.679Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.003µs CommitTime 844.741099ms TotalFlushTime 1m34.359162389s TotalCommitTime 19.123655122s 2015-08-03T20:38:56.679Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2398798, 1699399, 35 committed:true) SnapCreateDur 871.193236ms SnapOpenDur 52.463µs 2015-08-03T20:38:56.761Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1810000 2015-08-03T20:38:56.922Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2409744, 1704872, 0 committed:false) SnapCreateDur 30.51µs SnapOpenDur 31.27807ms 2015-08-03T20:38:57.081Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1720000 QueuedCount 1944 2015-08-03T20:38:57.216Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2420988, 1710494, 0 committed:false) SnapCreateDur 28.933µs SnapOpenDur 62.97346ms 2015-08-03T20:38:57.548Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2432572, 1716286, 0 committed:false) SnapCreateDur 23.99µs SnapOpenDur 99.766431ms 2015-08-03T20:38:57.653Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1730000 QueuedCount 3550 2015-08-03T20:38:57.811Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1820000 2015-08-03T20:38:58.766Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2443816, 1721908, 0 committed:false) SnapCreateDur 13.995µs SnapOpenDur 65.975991ms 2015-08-03T20:38:58.856Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1830000 2015-08-03T20:38:59.12Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.777µs CommitTime 256.982838ms TotalFlushTime 1m36.106449814s TotalCommitTime 19.38063796s 2015-08-03T20:38:59.12Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2447646, 1723823, 36 committed:true) SnapCreateDur 270.109338ms SnapOpenDur 27.872µs 2015-08-03T20:38:59.286Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1740000 QueuedCount 1198 2015-08-03T20:38:59.351Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2458874, 1729437, 0 committed:false) SnapCreateDur 26.366µs SnapOpenDur 20.286069ms 2015-08-03T20:38:59.54Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2470320, 1735160, 0 committed:false) SnapCreateDur 27.06µs SnapOpenDur 57.376062ms 2015-08-03T20:38:59.668Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1750000 QueuedCount 2792 2015-08-03T20:38:59.871Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2481708, 1740854, 0 committed:false) SnapCreateDur 26.34µs SnapOpenDur 99.560707ms 2015-08-03T20:38:59.891Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1840000 2015-08-03T20:39:00.939Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1850000 2015-08-03T20:39:01.186Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2493062, 1746531, 0 committed:false) SnapCreateDur 21.787µs SnapOpenDur 72.409226ms 2015-08-03T20:39:01.265Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1760000 QueuedCount 4237 2015-08-03T20:39:01.55Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2504220, 1752110, 0 committed:false) SnapCreateDur 25.597µs SnapOpenDur 148.776195ms 2015-08-03T20:39:01.773Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1770000 QueuedCount 60 2015-08-03T20:39:01.963Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2515692, 1757846, 0 committed:false) SnapCreateDur 23.42µs SnapOpenDur 181.823593ms 2015-08-03T20:39:01.987Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1860000 2015-08-03T20:39:03.027Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.81µs CommitTime 932.53555ms TotalFlushTime 1m38.248904543s TotalCommitTime 20.31317351s 2015-08-03T20:39:03.027Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2519552, 1759776, 37 committed:true) SnapCreateDur 959.273739ms SnapOpenDur 37.535µs 2015-08-03T20:39:03.038Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1870000 2015-08-03T20:39:03.269Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2530714, 1765357, 0 committed:false) SnapCreateDur 24.68µs SnapOpenDur 31.698075ms 2015-08-03T20:39:03.364Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1780000 QueuedCount 3522 2015-08-03T20:39:03.543Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2542222, 1771111, 0 committed:false) SnapCreateDur 23.127µs SnapOpenDur 62.767902ms 2015-08-03T20:39:03.861Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2553554, 1776777, 0 committed:false) SnapCreateDur 21.37µs SnapOpenDur 96.474337ms 2015-08-03T20:39:03.911Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1790000 QueuedCount 3129 2015-08-03T20:39:04.157Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1880000 2015-08-03T20:39:05.202Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1890000 2015-08-03T20:39:05.208Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2564852, 1782426, 0 committed:false) SnapCreateDur 31.333µs SnapOpenDur 115.32656ms 2015-08-03T20:39:05.21Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":3950457408,"bucket-1:by_email:avg_scan_wait_latency":3949596424,"bucket-1:by_email:avg_ts_interval":851459398,"bucket-1:by_email:avg_ts_items_count":5625,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":415589747,"bucket-1:by_email:delete_bytes":32153250,"bucket-1:by_email:disk_size":1734041600,"bucket-1:by_email:flush_queue_size":0,"bucket-1:by_email:frag_percent":76,"bucket-1:by_email:get_bytes":33311885,"bucket-1:by_email:insert_bytes":183513750,"bucket-1:by_email:items_count":1000000,"bucket-1:by_email:num_commits":37,"bucket-1:by_email:num_compactions":2,"bucket-1:by_email:num_completed_requests":7335,"bucket-1:by_email:num_docs_indexed":1763887,"bucket-1:by_email:num_docs_pending":264,"bucket-1:by_email:num_docs_queued":90599,"bucket-1:by_email:num_flush_queued":1795003,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":7719,"bucket-1:by_email:num_rows_returned":111,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":189,"bucket-1:by_email:scan_bytes_read":4440,"bucket-1:by_email:scan_wait_duration":30486934800899,"bucket-1:by_email:since_last_snapshot":1347094608,"bucket-1:by_email:timings/storage_clone_handle":"7335 495685775 2717735617411199","bucket-1:by_email:timings/storage_commit":"37 20313173510 -4685922761959368558","bucket-1:by_email:timings/storage_del":"782426 9991710945 4786669658364912593","bucket-1:by_email:timings/storage_get":"1795003 21774903688 4918913725428496706","bucket-1:by_email:timings/storage_handle_open":"37 599993050 358110082743815370","bucket-1:by_email:timings/storage_iterator_next":"111 1864831 42163433453","bucket-1:by_email:timings/storage_new_iterator":"7335 651634231 2760827115813059","bucket-1:by_email:timings/storage_set":"3564852 66063058813 2778395197939188893","bucket-1:by_email:timings/storage_snapshot_close":"150 2646498788 673519985670653330","bucket-1:by_email:timings/storage_snapshot_create":"152 13187259134 3447873982329021302","bucket-1:by_email:total_scan_duration":30493580733722,"bucket-1:mutation_queue_size":95064,"bucket-1:num_mutations_queued":1890067,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":48,"memory_quota":104857600000,"memory_used":544960512,"needs_restart":false,"num_connections":384,"timings/stats_response":"566 140404842 1508944851527064"} 2015-08-03T20:39:05.824Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 14.507µs CommitTime 513.350914ms TotalFlushTime 1m40.04311067s TotalCommitTime 20.826524424s 2015-08-03T20:39:05.824Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2568734, 1784367, 38 committed:true) SnapCreateDur 540.020899ms SnapOpenDur 49.331µs 2015-08-03T20:39:05.939Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1800000 QueuedCount 2755 2015-08-03T20:39:06.054Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2580064, 1790032, 0 committed:false) SnapCreateDur 25.286µs SnapOpenDur 28.802833ms 2015-08-03T20:39:06.247Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1900000 2015-08-03T20:39:06.346Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2591322, 1795661, 0 committed:false) SnapCreateDur 27.89µs SnapOpenDur 69.483954ms 2015-08-03T20:39:06.42Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1810000 QueuedCount 4144 2015-08-03T20:39:06.658Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2602526, 1801263, 0 committed:false) SnapCreateDur 28.651µs SnapOpenDur 101.071738ms 2015-08-03T20:39:07.301Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1910000 2015-08-03T20:39:07.88Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2613898, 1806949, 0 committed:false) SnapCreateDur 28.506µs SnapOpenDur 112.684279ms 2015-08-03T20:39:07.906Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1820000 QueuedCount 1814 2015-08-03T20:39:08.254Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2625174, 1812587, 0 committed:false) SnapCreateDur 34.997µs SnapOpenDur 140.587209ms 2015-08-03T20:39:08.339Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1920000 2015-08-03T20:39:09.134Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.77µs CommitTime 757.986306ms TotalFlushTime 1m41.920820611s TotalCommitTime 21.58451073s 2015-08-03T20:39:09.134Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2629042, 1814521, 39 committed:true) SnapCreateDur 785.357056ms SnapOpenDur 50.82µs 2015-08-03T20:39:09.241Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1830000 QueuedCount 3332 2015-08-03T20:39:09.384Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1930000 2015-08-03T20:39:09.389Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2640316, 1820158, 0 committed:false) SnapCreateDur 26.467µs SnapOpenDur 32.441809ms 2015-08-03T20:39:09.675Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2651770, 1825885, 0 committed:false) SnapCreateDur 34.246µs SnapOpenDur 64.147335ms 2015-08-03T20:39:09.732Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1840000 QueuedCount 2880 2015-08-03T20:39:10.012Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2663040, 1831520, 0 committed:false) SnapCreateDur 33.77µs SnapOpenDur 104.469241ms 2015-08-03T20:39:10.147Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:430208754, Disk:1941630976, ExtraSnapshotData:0, Fragmentation:77.84291869476232% 2015-08-03T20:39:10.147Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:39:10.17Z-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 2398798. Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T20:39:10.425Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1940000 2015-08-03T20:39:11.416Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1850000 QueuedCount 633 2015-08-03T20:39:11.458Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2674404, 1837202, 0 committed:false) SnapCreateDur 26.85µs SnapOpenDur 13.938169ms 2015-08-03T20:39:11.463Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1950000 2015-08-03T20:39:12.089Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 15.45µs CommitTime 497.869894ms TotalFlushTime 1m43.957474462s TotalCommitTime 22.082380624s 2015-08-03T20:39:12.09Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2678162, 1839081, 40 committed:true) SnapCreateDur 537.096097ms SnapOpenDur 78.926µs 2015-08-03T20:39:12.379Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1860000 QueuedCount 193 2015-08-03T20:39:12.434Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2693154, 1846577, 0 committed:false) SnapCreateDur 31.623µs SnapOpenDur 44.47738ms 2015-08-03T20:39:12.505Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1960000 2015-08-03T20:39:12.746Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2704570, 1852285, 0 committed:false) SnapCreateDur 24.984µs SnapOpenDur 79.36265ms 2015-08-03T20:39:12.903Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1870000 QueuedCount 1643 2015-08-03T20:39:13.052Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2715730, 1857865, 0 committed:false) SnapCreateDur 20.458µs SnapOpenDur 85.414869ms 2015-08-03T20:39:13.538Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1970000 2015-08-03T20:39:14.324Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2725884, 1862942, 0 committed:false) SnapCreateDur 30.236µs SnapOpenDur 110.367911ms 2015-08-03T20:39:14.449Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1880000 QueuedCount 2207 2015-08-03T20:39:14.583Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1980000 2015-08-03T20:39:14.66Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2736514, 1868257, 0 committed:false) SnapCreateDur 21.743µs SnapOpenDur 135.612492ms 2015-08-03T20:39:15.626Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1990000 2015-08-03T20:39:15.636Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.826µs CommitTime 825.030678ms TotalFlushTime 1m46.004104053s TotalCommitTime 22.907411302s 2015-08-03T20:39:15.636Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2740532, 1870266, 41 committed:true) SnapCreateDur 851.686187ms SnapOpenDur 41.529µs 2015-08-03T20:39:15.84Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1890000 QueuedCount 1930 2015-08-03T20:39:15.949Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2755670, 1877835, 0 committed:false) SnapCreateDur 16.87µs SnapOpenDur 38.717282ms 2015-08-03T20:39:16.206Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2766986, 1883493, 0 committed:false) SnapCreateDur 22.845µs SnapOpenDur 46.089682ms 2015-08-03T20:39:16.307Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1900000 QueuedCount 3412 2015-08-03T20:39:16.531Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2778270, 1889135, 0 committed:false) SnapCreateDur 16.884µs SnapOpenDur 103.74538ms 2015-08-03T20:39:16.664Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2000000 2015-08-03T20:39:17.714Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2010000 2015-08-03T20:39:17.755Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2789390, 1894695, 0 committed:false) SnapCreateDur 22.783µs SnapOpenDur 112.505472ms 2015-08-03T20:39:17.802Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1910000 QueuedCount 989 2015-08-03T20:39:18.49Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.76µs CommitTime 634.354783ms TotalFlushTime 1m47.724217092s TotalCommitTime 23.541766085s 2015-08-03T20:39:18.49Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2793164, 1896582, 42 committed:true) SnapCreateDur 663.769924ms SnapOpenDur 33.173µs 2015-08-03T20:39:18.644Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2808316, 1904158, 0 committed:false) SnapCreateDur 10.485µs SnapOpenDur 18.735994ms 2015-08-03T20:39:18.713Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1920000 QueuedCount 3314 2015-08-03T20:39:18.76Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2020000 2015-08-03T20:39:18.946Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2819778, 1909889, 0 committed:false) SnapCreateDur 28.9µs SnapOpenDur 81.518664ms 2015-08-03T20:39:19.166Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1930000 QueuedCount 233 2015-08-03T20:39:19.294Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2831056, 1915528, 0 committed:false) SnapCreateDur 37.154µs SnapOpenDur 115.689062ms 2015-08-03T20:39:19.802Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2030000 2015-08-03T20:39:20.661Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2842326, 1921163, 0 committed:false) SnapCreateDur 25.516µs SnapOpenDur 115.40267ms 2015-08-03T20:39:20.825Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1940000 QueuedCount 1622 2015-08-03T20:39:20.838Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2040000 2015-08-03T20:39:21.024Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2853534, 1926767, 0 committed:false) SnapCreateDur 21.057µs SnapOpenDur 142.459888ms 2015-08-03T20:39:21.887Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2050000 2015-08-03T20:39:21.943Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.307µs CommitTime 799.071163ms TotalFlushTime 1m49.704065568s TotalCommitTime 24.340837248s 2015-08-03T20:39:21.943Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2857492, 1928746, 43 committed:true) SnapCreateDur 825.130248ms SnapOpenDur 41.041µs 2015-08-03T20:39:22.096Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1950000 QueuedCount 1337 2015-08-03T20:39:22.171Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2872682, 1936341, 0 committed:false) SnapCreateDur 21.46µs SnapOpenDur 42.121549ms 2015-08-03T20:39:22.928Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2060000 2015-08-03T20:39:23.33Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2884048, 1942024, 0 committed:false) SnapCreateDur 22.323µs SnapOpenDur 79.647748ms 2015-08-03T20:39:23.456Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1960000 QueuedCount 4849 2015-08-03T20:39:23.966Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2070000 2015-08-03T20:39:24.688Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2899288, 1949644, 0 committed:false) SnapCreateDur 15.847µs SnapOpenDur 73.476185ms 2015-08-03T20:39:25.003Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2080000 2015-08-03T20:39:25.041Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.421µs CommitTime 244.142515ms TotalFlushTime 1m52.217471754s TotalCommitTime 24.584979763s 2015-08-03T20:39:25.041Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2903002, 1951501, 44 committed:true) SnapCreateDur 257.699966ms SnapOpenDur 47.36µs 2015-08-03T20:39:25.121Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1970000 QueuedCount 2514 2015-08-03T20:39:25.204Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2918176, 1959088, 0 committed:false) SnapCreateDur 12.118µs SnapOpenDur 18.617632ms 2015-08-03T20:39:25.405Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1980000 QueuedCount 2118 2015-08-03T20:39:25.538Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2933348, 1966674, 0 committed:false) SnapCreateDur 13.854µs SnapOpenDur 60.80845ms 2015-08-03T20:39:26.048Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2090000 2015-08-03T20:39:26.763Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2944630, 1972315, 0 committed:false) SnapCreateDur 21.088µs SnapOpenDur 63.665564ms 2015-08-03T20:39:26.859Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1990000 QueuedCount 5559 2015-08-03T20:39:27.107Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2100000 2015-08-03T20:39:27.172Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2959820, 1979910, 0 committed:false) SnapCreateDur 23.31µs SnapOpenDur 132.863813ms 2015-08-03T20:39:27.341Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2000000 QueuedCount 3153 2015-08-03T20:39:27.622Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2974736, 1987368, 0 committed:false) SnapCreateDur 32.476µs SnapOpenDur 176.643249ms 2015-08-03T20:39:28.145Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2110000 2015-08-03T20:39:28.956Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2986106, 1993053, 0 committed:false) SnapCreateDur 25.897µs SnapOpenDur 103.991762ms 2015-08-03T20:39:29.19Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2120000 2015-08-03T20:39:29.395Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2010000 QueuedCount 743 2015-08-03T20:39:29.872Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 13.293µs CommitTime 426.045979ms TotalFlushTime 1m55.8224014s TotalCommitTime 25.011025742s 2015-08-03T20:39:29.872Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 2989664, 1994832, 45 committed:true) SnapCreateDur 451.935294ms SnapOpenDur 45.293µs 2015-08-03T20:39:30.198Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3004994, 2002497, 0 committed:false) SnapCreateDur 21.517µs SnapOpenDur 46.512286ms 2015-08-03T20:39:30.233Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2130000 2015-08-03T20:39:30.273Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2020000 QueuedCount 4282 2015-08-03T20:39:30.579Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3020006, 2010003, 0 committed:false) SnapCreateDur 22.81µs SnapOpenDur 86.147979ms 2015-08-03T20:39:30.737Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2030000 QueuedCount 3848 2015-08-03T20:39:31.266Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2140000 2015-08-03T20:39:31.798Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3035118, 2017559, 0 committed:false) SnapCreateDur 9.752µs SnapOpenDur 64.987951ms 2015-08-03T20:39:31.929Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2040000 QueuedCount 1547 2015-08-03T20:39:32.11Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3050270, 2025135, 0 committed:false) SnapCreateDur 21.483µs SnapOpenDur 148.624424ms 2015-08-03T20:39:32.311Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2150000 2015-08-03T20:39:33.028Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.457µs CommitTime 796.567073ms TotalFlushTime 1m57.643268013s TotalCommitTime 25.807592815s 2015-08-03T20:39:33.029Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3054030, 2027015, 46 committed:true) SnapCreateDur 824.290607ms SnapOpenDur 27.428µs 2015-08-03T20:39:33.179Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2050000 QueuedCount 1161 2015-08-03T20:39:33.25Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3069152, 2034576, 0 committed:false) SnapCreateDur 29.277µs SnapOpenDur 45.364963ms 2015-08-03T20:39:33.349Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2160000 2015-08-03T20:39:33.517Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3084212, 2042106, 0 committed:false) SnapCreateDur 22.857µs SnapOpenDur 91.798493ms 2015-08-03T20:39:33.563Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2060000 QueuedCount 1507 2015-08-03T20:39:34.389Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2170000 2015-08-03T20:39:35.433Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2180000 2015-08-03T20:39:35.812Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3095438, 2047719, 0 committed:false) SnapCreateDur 26.374µs SnapOpenDur 104.16432ms 2015-08-03T20:39:36.359Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 18.673µs CommitTime 426.202442ms TotalFlushTime 2m0.168114049s TotalCommitTime 26.233795257s 2015-08-03T20:39:36.359Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3099308, 2049654, 47 committed:true) SnapCreateDur 452.233975ms SnapOpenDur 51.65µs 2015-08-03T20:39:36.478Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2190000 2015-08-03T20:39:36.496Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2070000 QueuedCount 4219 2015-08-03T20:39:36.675Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3114496, 2057248, 0 committed:false) SnapCreateDur 21.951µs SnapOpenDur 40.7019ms 2015-08-03T20:39:36.884Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2080000 QueuedCount 1813 2015-08-03T20:39:37.037Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3129472, 2064736, 0 committed:false) SnapCreateDur 24.643µs SnapOpenDur 87.109541ms 2015-08-03T20:39:37.523Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2200000 2015-08-03T20:39:38.564Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2210000 2015-08-03T20:39:39.458Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3144776, 2072388, 0 committed:false) SnapCreateDur 29.283µs SnapOpenDur 114.609584ms 2015-08-03T20:39:39.495Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2090000 QueuedCount 1475 2015-08-03T20:39:39.612Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2220000 2015-08-03T20:39:40.148Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 15.89µs CommitTime 535.756046ms TotalFlushTime 2m2.982935042s TotalCommitTime 26.769551303s 2015-08-03T20:39:40.149Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3148496, 2074248, 48 committed:true) SnapCreateDur 595.790116ms SnapOpenDur 46.507µs 2015-08-03T20:39:40.46Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3163414, 2081707, 0 committed:false) SnapCreateDur 27.452µs SnapOpenDur 36.490347ms 2015-08-03T20:39:40.516Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2100000 QueuedCount 2817 2015-08-03T20:39:40.646Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2230000 2015-08-03T20:39:40.832Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3178414, 2089207, 0 committed:false) SnapCreateDur 32.051µs SnapOpenDur 88.918831ms 2015-08-03T20:39:40.971Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2110000 QueuedCount 4369 2015-08-03T20:39:41.688Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2240000 2015-08-03T20:39:42.19Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3193544, 2096772, 0 committed:false) SnapCreateDur 21.221µs SnapOpenDur 73.113376ms 2015-08-03T20:39:42.328Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2120000 QueuedCount 2037 2015-08-03T20:39:42.475Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3208604, 2104302, 0 committed:false) SnapCreateDur 24.199µs SnapOpenDur 100.88047ms 2015-08-03T20:39:42.736Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2250000 2015-08-03T20:39:43.193Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.379µs CommitTime 608.659116ms TotalFlushTime 2m4.936805098s TotalCommitTime 27.378210419s 2015-08-03T20:39:43.193Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3212390, 2106195, 49 committed:true) SnapCreateDur 623.847389ms SnapOpenDur 40.75µs 2015-08-03T20:39:43.411Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2130000 QueuedCount 1649 2015-08-03T20:39:43.51Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3227524, 2113762, 0 committed:false) SnapCreateDur 29.413µs SnapOpenDur 42.541611ms 2015-08-03T20:39:43.778Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2260000 2015-08-03T20:39:43.89Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3242668, 2121334, 0 committed:false) SnapCreateDur 22.256µs SnapOpenDur 88.614313ms 2015-08-03T20:39:43.935Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2140000 QueuedCount 1473 2015-08-03T20:39:44.81Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2270000 2015-08-03T20:39:45.245Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3257794, 2128897, 0 committed:false) SnapCreateDur 22.193µs SnapOpenDur 110.459291ms 2015-08-03T20:39:45.372Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2150000 QueuedCount 4766 2015-08-03T20:39:45.682Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3272992, 2136496, 0 committed:false) SnapCreateDur 21.21µs SnapOpenDur 151.781739ms 2015-08-03T20:39:45.854Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2280000 2015-08-03T20:39:46.743Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.83µs CommitTime 939.598418ms TotalFlushTime 2m6.956185155s TotalCommitTime 28.317808837s 2015-08-03T20:39:46.926Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2290000 2015-08-03T20:39:47.962Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2300000 2015-08-03T20:39:49.002Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2310000 2015-08-03T20:39:49.021Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3276668, 2138334, 50 committed:true) SnapCreateDur 967.358296ms SnapOpenDur 2.277701093s 2015-08-03T20:39:49.405Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 5.403µs CommitTime 262.041357ms TotalFlushTime 2m7.019739636s TotalCommitTime 28.579850194s 2015-08-03T20:39:49.405Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3280480, 2140240, 51 committed:true) SnapCreateDur 289.517538ms SnapOpenDur 57.39µs 2015-08-03T20:39:49.472Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2160000 QueuedCount 2453 2015-08-03T20:39:50.05Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2320000 2015-08-03T20:39:50.661Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3295638, 2147819, 0 committed:false) SnapCreateDur 25.353µs SnapOpenDur 40.420762ms 2015-08-03T20:39:51.081Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2330000 2015-08-03T20:39:51.375Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2170000 QueuedCount 3935 2015-08-03T20:39:51.596Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3310738, 2155369, 0 committed:false) SnapCreateDur 20.986µs SnapOpenDur 80.99074ms 2015-08-03T20:39:52.125Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2340000 2015-08-03T20:39:52.725Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2180000 QueuedCount 1539 2015-08-03T20:39:52.872Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3325714, 2162857, 0 committed:false) SnapCreateDur 23.38µs SnapOpenDur 111.853483ms 2015-08-03T20:39:53.16Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2350000 2015-08-03T20:39:53.277Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 1.978µs CommitTime 298.739482ms TotalFlushTime 2m10.216924686s TotalCommitTime 28.878589676s 2015-08-03T20:39:53.277Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3329820, 2164910, 52 committed:true) SnapCreateDur 311.44137ms SnapOpenDur 35.444µs 2015-08-03T20:39:53.442Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2190000 QueuedCount 1247 2015-08-03T20:39:53.512Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3344810, 2172405, 0 committed:false) SnapCreateDur 20.956µs SnapOpenDur 39.543778ms 2015-08-03T20:39:53.865Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3359734, 2179867, 0 committed:false) SnapCreateDur 21.883µs SnapOpenDur 78.293963ms 2015-08-03T20:39:53.929Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2200000 QueuedCount 2721 2015-08-03T20:39:54.207Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2360000 2015-08-03T20:39:55.239Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2370000 2015-08-03T20:39:55.262Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3374826, 2187413, 0 committed:false) SnapCreateDur 23.04µs SnapOpenDur 109.73609ms 2015-08-03T20:39:55.35Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2210000 QueuedCount 2259 2015-08-03T20:39:55.575Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3389816, 2194908, 0 committed:false) SnapCreateDur 20.993µs SnapOpenDur 147.972888ms 2015-08-03T20:39:56.286Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2380000 2015-08-03T20:39:57.337Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2390000 2015-08-03T20:39:57.438Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 2015-08-03T20:39:57.471Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2220000 QueuedCount 1870 2015-08-03T20:39:57.547Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3404992, 2202496, 0 committed:false) SnapCreateDur 22.437µs SnapOpenDur 16.432494ms 2015-08-03T20:39:58.374Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2400000 2015-08-03T20:39:58.935Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.336µs CommitTime 259.664121ms TotalFlushTime 2m13.996455086s TotalCommitTime 29.138253797s 2015-08-03T20:39:58.935Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3408778, 2204389, 53 committed:true) SnapCreateDur 1.304161033s SnapOpenDur 34.285µs 2015-08-03T20:39:59.161Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2230000 QueuedCount 1514 2015-08-03T20:39:59.263Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3423906, 2211953, 0 committed:false) SnapCreateDur 23.386µs SnapOpenDur 42.533027ms 2015-08-03T20:39:59.412Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2410000 2015-08-03T20:39:59.668Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3438966, 2219483, 0 committed:false) SnapCreateDur 31.545µs SnapOpenDur 99.989458ms 2015-08-03T20:39:59.724Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2240000 QueuedCount 3035 2015-08-03T20:40:00.458Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2420000 2015-08-03T20:40:00.932Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3454104, 2227052, 0 committed:false) SnapCreateDur 32.387µs SnapOpenDur 116.017325ms 2015-08-03T20:40:01.495Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2430000 2015-08-03T20:40:01.66Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.24µs CommitTime 604.304296ms TotalFlushTime 2m15.686772701s TotalCommitTime 29.742558093s 2015-08-03T20:40:01.66Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3457970, 2228985, 54 committed:true) SnapCreateDur 632.640187ms SnapOpenDur 73.544µs 2015-08-03T20:40:01.725Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2250000 QueuedCount 2632 2015-08-03T20:40:02.005Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3472964, 2236482, 0 committed:false) SnapCreateDur 29.487µs SnapOpenDur 46.851067ms 2015-08-03T20:40:02.156Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2260000 QueuedCount 4111 2015-08-03T20:40:02.406Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3488028, 2244014, 0 committed:false) SnapCreateDur 36.123µs SnapOpenDur 95.316516ms 2015-08-03T20:40:02.541Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2440000 2015-08-03T20:40:03.614Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2450000 2015-08-03T20:40:03.635Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2270000 QueuedCount 1803 2015-08-03T20:40:03.803Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3503174, 2251587, 0 committed:false) SnapCreateDur 23.59µs SnapOpenDur 123.14106ms 2015-08-03T20:40:04.134Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3518422, 2259211, 0 committed:false) SnapCreateDur 30.606µs SnapOpenDur 156.227217ms 2015-08-03T20:40:04.166Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2280000 QueuedCount 1422 2015-08-03T20:40:04.724Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2460000 2015-08-03T20:40:05.051Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.022µs CommitTime 836.541899ms TotalFlushTime 2m17.62765848s TotalCommitTime 30.579099992s 2015-08-03T20:40:05.052Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3522116, 2261058, 55 committed:true) SnapCreateDur 853.235298ms SnapOpenDur 38.714µs 2015-08-03T20:40:05.211Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":5742145819,"bucket-1:by_email:avg_scan_wait_latency":5741300430,"bucket-1:by_email:avg_ts_interval":777950579,"bucket-1:by_email:avg_ts_items_count":4698,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":397784727,"bucket-1:by_email:delete_bytes":52975097,"bucket-1:by_email:disk_size":1534418944,"bucket-1:by_email:flush_queue_size":3558,"bucket-1:by_email:frag_percent":74,"bucket-1:by_email:get_bytes":54604410,"bucket-1:by_email:insert_bytes":234357795,"bucket-1:by_email:items_count":1000000,"bucket-1:by_email:num_commits":55,"bucket-1:by_email:num_compactions":3,"bucket-1:by_email:num_completed_requests":9219,"bucket-1:by_email:num_docs_indexed":2256293,"bucket-1:by_email:num_docs_pending":373,"bucket-1:by_email:num_docs_queued":178141,"bucket-1:by_email:num_flush_queued":2288916,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":9603,"bucket-1:by_email:num_rows_returned":142,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":272,"bucket-1:by_email:scan_bytes_read":5680,"bucket-1:by_email:scan_wait_duration":55133708036441,"bucket-1:by_email:since_last_snapshot":917678097,"bucket-1:by_email:timings/storage_clone_handle":"9219 585075649 2722728710270091","bucket-1:by_email:timings/storage_commit":"55 30579099992 2009576856470669938","bucket-1:by_email:timings/storage_del":"1264927 15434229717 6963672853412641903","bucket-1:by_email:timings/storage_get":"2285360 31004810451 8115895441555331043","bucket-1:by_email:timings/storage_handle_open":"55 2878407160 5546006710493331720","bucket-1:by_email:timings/storage_iterator_next":"142 2437734 53701106062","bucket-1:by_email:timings/storage_new_iterator":"9219 789157530 2782617903476858","bucket-1:by_email:timings/storage_set":"4529852 88476666652 2534745431657763516","bucket-1:by_email:timings/storage_snapshot_close":"217 3558833306 688944549332391990","bucket-1:by_email:timings/storage_snapshot_create":"217 18664582295 4008882593064907941","bucket-1:by_email:total_scan_duration":55141826307490,"bucket-1:mutation_queue_size":175700,"bucket-1:num_mutations_queued":2464616,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":91,"memory_quota":104857600000,"memory_used":504676352,"needs_restart":false,"num_connections":384,"timings/stats_response":"638 165828249 1518263431528907"} 2015-08-03T20:40:05.372Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3536872, 2268436, 0 committed:false) SnapCreateDur 23.492µs SnapOpenDur 33.262639ms 2015-08-03T20:40:05.434Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2290000 QueuedCount 2750 2015-08-03T20:40:05.768Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2470000 2015-08-03T20:40:05.832Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3551970, 2275985, 0 committed:false) SnapCreateDur 33.671µs SnapOpenDur 121.341714ms 2015-08-03T20:40:06.008Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2300000 QueuedCount 4251 2015-08-03T20:40:06.836Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2480000 2015-08-03T20:40:07.226Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3567054, 2283527, 0 committed:false) SnapCreateDur 14.53µs SnapOpenDur 76.824306ms 2015-08-03T20:40:07.869Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 15.146µs CommitTime 522.887458ms TotalFlushTime 2m19.529678733s TotalCommitTime 31.10198745s 2015-08-03T20:40:07.87Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3570828, 2285414, 56 committed:true) SnapCreateDur 549.419616ms SnapOpenDur 49.8µs 2015-08-03T20:40:07.886Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2490000 2015-08-03T20:40:08.021Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2310000 QueuedCount 3841 2015-08-03T20:40:08.194Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3585922, 2292961, 0 committed:false) SnapCreateDur 36.988µs SnapOpenDur 39.474914ms 2015-08-03T20:40:08.432Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2320000 QueuedCount 1549 2015-08-03T20:40:08.576Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3601088, 2300544, 0 committed:false) SnapCreateDur 24.957µs SnapOpenDur 84.325145ms 2015-08-03T20:40:08.928Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2500000 2015-08-03T20:40:09.86Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3616180, 2308090, 0 committed:false) SnapCreateDur 22.503µs SnapOpenDur 109.029506ms 2015-08-03T20:40:09.911Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2330000 QueuedCount 3101 2015-08-03T20:40:09.961Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2510000 2015-08-03T20:40:10.306Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3631282, 2315641, 0 committed:false) SnapCreateDur 34.576µs SnapOpenDur 160.701985ms 2015-08-03T20:40:11.002Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2520000 2015-08-03T20:40:11.326Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 14.264µs CommitTime 896.681105ms TotalFlushTime 2m21.500840567s TotalCommitTime 31.998668555s 2015-08-03T20:40:11.326Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3635082, 2317541, 57 committed:true) SnapCreateDur 925.365303ms SnapOpenDur 38.216µs 2015-08-03T20:40:11.387Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2340000 QueuedCount 2728 2015-08-03T20:40:11.643Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3650210, 2325105, 0 committed:false) SnapCreateDur 31.055µs SnapOpenDur 31.421604ms 2015-08-03T20:40:11.794Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2350000 QueuedCount 4220 2015-08-03T20:40:12.038Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2530000 2015-08-03T20:40:12.043Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3665398, 2332699, 0 committed:false) SnapCreateDur 25.177µs SnapOpenDur 94.646872ms 2015-08-03T20:40:13.084Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2540000 2015-08-03T20:40:13.205Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2360000 QueuedCount 1854 2015-08-03T20:40:13.308Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3680430, 2340215, 0 committed:false) SnapCreateDur 20.663µs SnapOpenDur 68.105794ms 2015-08-03T20:40:13.724Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.645µs CommitTime 307.125158ms TotalFlushTime 2m23.254959639s TotalCommitTime 32.305793713s 2015-08-03T20:40:13.724Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3684416, 2342208, 58 committed:true) SnapCreateDur 321.210883ms SnapOpenDur 33.005µs 2015-08-03T20:40:13.832Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2370000 QueuedCount 1529 2015-08-03T20:40:13.877Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3699458, 2349729, 0 committed:false) SnapCreateDur 18.694µs SnapOpenDur 18.548373ms 2015-08-03T20:40:14.116Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2550000 2015-08-03T20:40:14.224Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3714662, 2357331, 0 committed:false) SnapCreateDur 21.894µs SnapOpenDur 83.454683ms 2015-08-03T20:40:14.272Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2380000 QueuedCount 3036 2015-08-03T20:40:15.163Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2560000 2015-08-03T20:40:15.604Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3729588, 2364794, 0 committed:false) SnapCreateDur 22.054µs SnapOpenDur 75.060094ms 2015-08-03T20:40:15.69Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2390000 QueuedCount 2603 2015-08-03T20:40:15.873Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3744630, 2372315, 0 committed:false) SnapCreateDur 12.939µs SnapOpenDur 95.433658ms 2015-08-03T20:40:15.998Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2400000 QueuedCount 2189 2015-08-03T20:40:16.155Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3759758, 2379879, 0 committed:false) SnapCreateDur 18.963µs SnapOpenDur 116.683565ms 2015-08-03T20:40:16.199Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2570000 2015-08-03T20:40:17.205Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3774888, 2387444, 0 committed:false) SnapCreateDur 30.443µs SnapOpenDur 156.184588ms 2015-08-03T20:40:17.231Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2410000 QueuedCount 1807 2015-08-03T20:40:17.241Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2580000 2015-08-03T20:40:17.781Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.482µs CommitTime 486.369799ms TotalFlushTime 2m26.050073716s TotalCommitTime 32.792163512s 2015-08-03T20:40:17.782Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3778702, 2389351, 59 committed:true) SnapCreateDur 501.669485ms SnapOpenDur 36.811µs 2015-08-03T20:40:18.092Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3793834, 2396917, 0 committed:false) SnapCreateDur 17.495µs SnapOpenDur 26.52602ms 2015-08-03T20:40:18.125Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2420000 QueuedCount 1403 2015-08-03T20:40:18.287Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2590000 2015-08-03T20:40:18.409Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3808936, 2404468, 0 committed:false) SnapCreateDur 24.606µs SnapOpenDur 91.623196ms 2015-08-03T20:40:18.535Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2430000 QueuedCount 4862 2015-08-03T20:40:19.333Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2600000 2015-08-03T20:40:19.778Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3824086, 2412043, 0 committed:false) SnapCreateDur 27.156µs SnapOpenDur 122.943035ms 2015-08-03T20:40:19.978Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2440000 QueuedCount 2491 2015-08-03T20:40:20.235Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3839110, 2419555, 0 committed:false) SnapCreateDur 25.11µs SnapOpenDur 171.856128ms 2015-08-03T20:40:20.369Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2610000 2015-08-03T20:40:21.341Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.907µs CommitTime 974.625423ms TotalFlushTime 2m28.018141659s TotalCommitTime 33.766788935s 2015-08-03T20:40:21.341Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3842890, 2421445, 60 committed:true) SnapCreateDur 1.001547461s SnapOpenDur 53.187µs 2015-08-03T20:40:21.412Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2620000 2015-08-03T20:40:21.547Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2450000 QueuedCount 1757 2015-08-03T20:40:21.648Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3857350, 2428675, 0 committed:false) SnapCreateDur 24.457µs SnapOpenDur 42.689645ms 2015-08-03T20:40:22.009Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3871304, 2435652, 0 committed:false) SnapCreateDur 25.493µs SnapOpenDur 95.605044ms 2015-08-03T20:40:22.076Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2460000 QueuedCount 2626 2015-08-03T20:40:22.457Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2630000 2015-08-03T20:40:23.288Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3886310, 2443155, 0 committed:false) SnapCreateDur 13.122µs SnapOpenDur 62.74079ms 2015-08-03T20:40:23.491Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2640000 2015-08-03T20:40:23.914Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.723µs CommitTime 515.726302ms TotalFlushTime 2m29.725568607s TotalCommitTime 34.282515237s 2015-08-03T20:40:23.914Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3889616, 2444808, 61 committed:true) SnapCreateDur 542.226391ms SnapOpenDur 44.45µs 2015-08-03T20:40:24.002Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2470000 QueuedCount 4090 2015-08-03T20:40:24.248Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3905094, 2452547, 0 committed:false) SnapCreateDur 38.067µs SnapOpenDur 47.974357ms 2015-08-03T20:40:24.411Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2480000 QueuedCount 3497 2015-08-03T20:40:24.536Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2650000 2015-08-03T20:40:24.627Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3919862, 2459931, 0 committed:false) SnapCreateDur 26.743µs SnapOpenDur 92.699887ms 2015-08-03T20:40:25.577Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2660000 2015-08-03T20:40:25.768Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2490000 QueuedCount 1181 2015-08-03T20:40:25.863Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3934990, 2467495, 0 committed:false) SnapCreateDur 10.962µs SnapOpenDur 69.6351ms 2015-08-03T20:40:26.339Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3950090, 2475045, 0 committed:false) SnapCreateDur 25.873µs SnapOpenDur 181.233065ms 2015-08-03T20:40:26.406Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2500000 QueuedCount 807 2015-08-03T20:40:26.616Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2670000 2015-08-03T20:40:27.439Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:439156300, Disk:2093395968, ExtraSnapshotData:0, Fragmentation:79.02182354829108% 2015-08-03T20:40:27.439Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:40:27.442Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.35µs CommitTime 971.829172ms TotalFlushTime 2m31.690919257s TotalCommitTime 35.254344409s 2015-08-03T20:40:27.442Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3953920, 2476960, 62 committed:true) SnapCreateDur 998.59426ms SnapOpenDur 40.998µs 2015-08-03T20:40:27.465Z-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 3684416. Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T20:40:27.66Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2680000 2015-08-03T20:40:27.806Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3968970, 2484485, 0 committed:false) SnapCreateDur 28.553µs SnapOpenDur 48.475245ms 2015-08-03T20:40:27.889Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2510000 QueuedCount 4227 2015-08-03T20:40:28.216Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3984078, 2492039, 0 committed:false) SnapCreateDur 25.58µs SnapOpenDur 93.688663ms 2015-08-03T20:40:28.387Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2520000 QueuedCount 3851 2015-08-03T20:40:28.711Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2690000 2015-08-03T20:40:29.604Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 3999270, 2499635, 0 committed:false) SnapCreateDur 35.707µs SnapOpenDur 100.783523ms 2015-08-03T20:40:29.74Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2700000 2015-08-03T20:40:30.27Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.207µs CommitTime 545.666795ms TotalFlushTime 2m33.567423694s TotalCommitTime 35.800011204s 2015-08-03T20:40:30.27Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4003090, 2501545, 63 committed:true) SnapCreateDur 572.101475ms SnapOpenDur 26.33µs 2015-08-03T20:40:30.427Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2530000 QueuedCount 3467 2015-08-03T20:40:30.574Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4018226, 2509113, 0 committed:false) SnapCreateDur 22.094µs SnapOpenDur 30.337941ms 2015-08-03T20:40:30.741Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2540000 QueuedCount 1169 2015-08-03T20:40:30.787Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2710000 2015-08-03T20:40:30.875Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4033384, 2516692, 0 committed:false) SnapCreateDur 25.05µs SnapOpenDur 87.431825ms 2015-08-03T20:40:31.827Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2720000 2015-08-03T20:40:32.176Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4048524, 2524262, 0 committed:false) SnapCreateDur 17.775µs SnapOpenDur 76.259808ms 2015-08-03T20:40:32.272Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2550000 QueuedCount 4627 2015-08-03T20:40:32.687Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4063644, 2531822, 0 committed:false) SnapCreateDur 27.907µs SnapOpenDur 175.000714ms 2015-08-03T20:40:32.862Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2730000 2015-08-03T20:40:33.608Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.87µs CommitTime 799.080423ms TotalFlushTime 2m35.518580321s TotalCommitTime 36.599091627s 2015-08-03T20:40:33.608Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4067538, 2533769, 64 committed:true) SnapCreateDur 825.752047ms SnapOpenDur 35.627µs 2015-08-03T20:40:33.661Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2560000 QueuedCount 2322 2015-08-03T20:40:33.806Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4082740, 2541370, 0 committed:false) SnapCreateDur 18.791µs SnapOpenDur 23.209344ms 2015-08-03T20:40:33.899Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2570000 QueuedCount 3498 2015-08-03T20:40:33.91Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2740000 2015-08-03T20:40:34.015Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4097704, 2548852, 0 committed:false) SnapCreateDur 14.967µs SnapOpenDur 46.143205ms 2015-08-03T20:40:34.951Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2750000 2015-08-03T20:40:35.132Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2580000 QueuedCount 1492 2015-08-03T20:40:35.273Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4112788, 2556394, 0 committed:false) SnapCreateDur 29.443µs SnapOpenDur 104.836294ms 2015-08-03T20:40:35.509Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4127880, 2563940, 0 committed:false) SnapCreateDur 11.398µs SnapOpenDur 81.213149ms 2015-08-03T20:40:35.56Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2590000 QueuedCount 2989 2015-08-03T20:40:35.961Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4143094, 2571547, 0 committed:false) SnapCreateDur 29.56µs SnapOpenDur 167.131104ms 2015-08-03T20:40:35.987Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2760000 2015-08-03T20:40:37.035Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2770000 2015-08-03T20:40:37.198Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 13.37µs CommitTime 1.124996467s TotalFlushTime 2m37.338196382s TotalCommitTime 37.724088094s 2015-08-03T20:40:37.198Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4146770, 2573385, 65 committed:true) SnapCreateDur 1.151754997s SnapOpenDur 37.484µs 2015-08-03T20:40:37.243Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2600000 QueuedCount 726 2015-08-03T20:40:37.385Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4161864, 2580932, 0 committed:false) SnapCreateDur 12.492µs SnapOpenDur 20.471738ms 2015-08-03T20:40:37.464Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2610000 QueuedCount 2175 2015-08-03T20:40:37.585Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4176986, 2588493, 0 committed:false) SnapCreateDur 11.291µs SnapOpenDur 44.254718ms 2015-08-03T20:40:38.097Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2780000 2015-08-03T20:40:38.702Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2620000 QueuedCount 1847 2015-08-03T20:40:38.858Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4192146, 2596073, 0 committed:false) SnapCreateDur 20.943µs SnapOpenDur 112.527418ms 2015-08-03T20:40:39.133Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2790000 2015-08-03T20:40:39.318Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4207246, 2603623, 0 committed:false) SnapCreateDur 26.544µs SnapOpenDur 156.055137ms 2015-08-03T20:40:39.357Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2630000 QueuedCount 1356 2015-08-03T20:40:40.166Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2800000 2015-08-03T20:40:40.351Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.99µs CommitTime 920.627719ms TotalFlushTime 2m39.03902304s TotalCommitTime 38.644715813s 2015-08-03T20:40:40.351Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4210826, 2605413, 66 committed:true) SnapCreateDur 948.17822ms SnapOpenDur 34.679µs 2015-08-03T20:40:40.682Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4226120, 2613060, 0 committed:false) SnapCreateDur 20.007µs SnapOpenDur 48.508625ms 2015-08-03T20:40:40.734Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2640000 QueuedCount 2961 2015-08-03T20:40:41.06Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4241368, 2620684, 0 committed:false) SnapCreateDur 23.697µs SnapOpenDur 91.504851ms 2015-08-03T20:40:41.193Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2650000 QueuedCount 4571 2015-08-03T20:40:41.211Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2810000 2015-08-03T20:40:42.251Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2820000 2015-08-03T20:40:42.335Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4256542, 2628271, 0 committed:false) SnapCreateDur 27.26µs SnapOpenDur 110.027411ms 2015-08-03T20:40:43.063Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.144µs CommitTime 607.584276ms TotalFlushTime 2m40.72835733s TotalCommitTime 39.252300089s 2015-08-03T20:40:43.287Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2830000 2015-08-03T20:40:43.837Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4260328, 2630164, 67 committed:true) SnapCreateDur 634.287271ms SnapOpenDur 774.489338ms 2015-08-03T20:40:43.977Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2660000 QueuedCount 4154 2015-08-03T20:40:44.156Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4275398, 2637699, 0 committed:false) SnapCreateDur 19.593µs SnapOpenDur 43.141482ms 2015-08-03T20:40:44.334Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2840000 2015-08-03T20:40:44.38Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2670000 QueuedCount 1789 2015-08-03T20:40:44.54Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4290444, 2645222, 0 committed:false) SnapCreateDur 20.07µs SnapOpenDur 89.523249ms 2015-08-03T20:40:45.371Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2850000 2015-08-03T20:40:45.978Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4305542, 2652771, 0 committed:false) SnapCreateDur 18.327µs SnapOpenDur 111.702733ms 2015-08-03T20:40:46.017Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2680000 QueuedCount 1348 2015-08-03T20:40:46.416Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2860000 2015-08-03T20:40:46.687Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.747µs CommitTime 599.188014ms TotalFlushTime 2m42.577186119s TotalCommitTime 39.851488103s 2015-08-03T20:40:46.687Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4309252, 2654626, 68 committed:true) SnapCreateDur 625.420734ms SnapOpenDur 52.633µs 2015-08-03T20:40:47.006Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4324400, 2662200, 0 committed:false) SnapCreateDur 30.637µs SnapOpenDur 42.607183ms 2015-08-03T20:40:47.048Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2690000 QueuedCount 876 2015-08-03T20:40:47.285Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4339572, 2669786, 0 committed:false) SnapCreateDur 20.606µs SnapOpenDur 85.169515ms 2015-08-03T20:40:47.435Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2700000 QueuedCount 4424 2015-08-03T20:40:47.463Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2870000 2015-08-03T20:40:48.509Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2880000 2015-08-03T20:40:48.621Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4354718, 2677359, 0 committed:false) SnapCreateDur 12.655µs SnapOpenDur 72.372102ms 2015-08-03T20:40:48.833Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2710000 QueuedCount 2107 2015-08-03T20:40:49.003Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4369834, 2684917, 0 committed:false) SnapCreateDur 12.621µs SnapOpenDur 97.147907ms 2015-08-03T20:40:49.549Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2890000 2015-08-03T20:40:49.936Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.677µs CommitTime 843.046045ms TotalFlushTime 2m44.496257915s TotalCommitTime 40.694534148s 2015-08-03T20:40:49.936Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4373608, 2686804, 69 committed:true) SnapCreateDur 869.170433ms SnapOpenDur 32.988µs 2015-08-03T20:40:50.078Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2720000 QueuedCount 1620 2015-08-03T20:40:50.155Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4388514, 2694257, 0 committed:false) SnapCreateDur 28.356µs SnapOpenDur 43.623704ms 2015-08-03T20:40:50.45Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4403810, 2701905, 0 committed:false) SnapCreateDur 24.136µs SnapOpenDur 69.552621ms 2015-08-03T20:40:50.496Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2730000 QueuedCount 1410 2015-08-03T20:40:50.59Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2900000 2015-08-03T20:40:51.633Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2910000 2015-08-03T20:40:52.666Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2920000 2015-08-03T20:40:53.236Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4418892, 2709446, 0 committed:false) SnapCreateDur 25.127µs SnapOpenDur 121.676413ms 2015-08-03T20:40:53.711Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2930000 2015-08-03T20:40:53.885Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.353µs CommitTime 527.520293ms TotalFlushTime 2m47.536296182s TotalCommitTime 41.222054441s 2015-08-03T20:40:53.885Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4422656, 2711328, 70 committed:true) SnapCreateDur 555.049573ms SnapOpenDur 39.646µs 2015-08-03T20:40:53.94Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2740000 QueuedCount 2897 2015-08-03T20:40:54.214Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4437866, 2718933, 0 committed:false) SnapCreateDur 19.18µs SnapOpenDur 43.995561ms 2015-08-03T20:40:54.351Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2750000 QueuedCount 4381 2015-08-03T20:40:54.587Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4452920, 2726460, 0 committed:false) SnapCreateDur 27.873µs SnapOpenDur 88.16283ms 2015-08-03T20:40:54.756Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2940000 2015-08-03T20:40:55.714Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2760000 QueuedCount 2038 2015-08-03T20:40:55.791Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2950000 2015-08-03T20:40:55.834Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4467994, 2733997, 0 committed:false) SnapCreateDur 13.821µs SnapOpenDur 71.699152ms 2015-08-03T20:40:56.138Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4483038, 2741519, 0 committed:false) SnapCreateDur 21.744µs SnapOpenDur 149.696232ms 2015-08-03T20:40:56.166Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2770000 QueuedCount 1598 2015-08-03T20:40:56.525Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4497878, 2748939, 0 committed:false) SnapCreateDur 21.676µs SnapOpenDur 200.62545ms 2015-08-03T20:40:56.853Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2960000 2015-08-03T20:40:57.892Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2970000 2015-08-03T20:40:58.937Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2980000 2015-08-03T20:40:59.261Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.877µs CommitTime 316.62769ms TotalFlushTime 2m51.817450194s TotalCommitTime 41.538682131s 2015-08-03T20:40:59.261Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4501704, 2750852, 71 committed:true) SnapCreateDur 342.669795ms SnapOpenDur 37.96µs 2015-08-03T20:40:59.314Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2780000 QueuedCount 2930 2015-08-03T20:40:59.585Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4516794, 2758397, 0 committed:false) SnapCreateDur 34.26µs SnapOpenDur 48.469418ms 2015-08-03T20:40:59.981Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2990000 2015-08-03T20:41:00.402Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 3.82µs CommitTime 707.107221ms TotalFlushTime 2m52.120440244s TotalCommitTime 42.245789352s 2015-08-03T20:41:00.402Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4520586, 2760293, 72 committed:true) SnapCreateDur 733.301481ms SnapOpenDur 33.743µs 2015-08-03T20:41:00.472Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2790000 QueuedCount 4179 2015-08-03T20:41:00.736Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4535724, 2767862, 0 committed:false) SnapCreateDur 31.97µs SnapOpenDur 40.483124ms 2015-08-03T20:41:01.012Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3000000 2015-08-03T20:41:01.669Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2800000 QueuedCount 4145 2015-08-03T20:41:01.902Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4550924, 2775462, 0 committed:false) SnapCreateDur 23.62µs SnapOpenDur 81.189353ms 2015-08-03T20:41:02.061Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3010000 2015-08-03T20:41:03.075Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2810000 QueuedCount 1827 2015-08-03T20:41:03.108Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3020000 2015-08-03T20:41:03.221Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4566062, 2783031, 0 committed:false) SnapCreateDur 21.771µs SnapOpenDur 105.416949ms 2015-08-03T20:41:03.939Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 5.207µs CommitTime 595.40285ms TotalFlushTime 2m54.669100484s TotalCommitTime 42.841192202s 2015-08-03T20:41:03.939Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4569912, 2784956, 73 committed:true) SnapCreateDur 623.708146ms SnapOpenDur 41.914µs 2015-08-03T20:41:04.157Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2820000 QueuedCount 1443 2015-08-03T20:41:04.227Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3030000 2015-08-03T20:41:04.244Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4584980, 2792490, 0 committed:false) SnapCreateDur 18.95µs SnapOpenDur 39.918896ms 2015-08-03T20:41:04.603Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4600084, 2800042, 0 committed:false) SnapCreateDur 21.5µs SnapOpenDur 83.909142ms 2015-08-03T20:41:04.662Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2830000 QueuedCount 2951 2015-08-03T20:41:05.211Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":7065484543,"bucket-1:by_email:avg_scan_wait_latency":7064569722,"bucket-1:by_email:avg_ts_interval":483368043,"bucket-1:by_email:avg_ts_items_count":6901,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":491939532,"bucket-1:by_email:delete_bytes":76544343,"bucket-1:by_email:disk_size":2802987008,"bucket-1:by_email:flush_queue_size":3568,"bucket-1:by_email:frag_percent":82,"bucket-1:by_email:get_bytes":78686431,"bucket-1:by_email:insert_bytes":291910562,"bucket-1:by_email:items_count":1000000,"bucket-1:by_email:num_commits":73,"bucket-1:by_email:num_compactions":4,"bucket-1:by_email:num_completed_requests":10371,"bucket-1:by_email:num_docs_indexed":2811942,"bucket-1:by_email:num_docs_pending":93,"bucket-1:by_email:num_docs_queued":203181,"bucket-1:by_email:num_flush_queued":2836866,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":10755,"bucket-1:by_email:num_rows_returned":160,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":357,"bucket-1:by_email:scan_bytes_read":6400,"bucket-1:by_email:scan_wait_duration":75979447367303,"bucket-1:by_email:since_last_snapshot":358770258,"bucket-1:by_email:timings/storage_clone_handle":"10371 642787298 2728593328092644","bucket-1:by_email:timings/storage_commit":"73 42841192202 -7147115488827690116","bucket-1:by_email:timings/storage_del":"1804141 22372663176 -7958931375458137382","bucket-1:by_email:timings/storage_get":"2833299 37520341648 8117481036520110476","bucket-1:by_email:timings/storage_handle_open":"73 3653497774 6145830880995954460","bucket-1:by_email:timings/storage_iterator_next":"160 2837109 64031815317","bucket-1:by_email:timings/storage_new_iterator":"10371 901902896 2812751275542996","bucket-1:by_email:timings/storage_set":"5608280 111830240668 2382196497756650774","bucket-1:by_email:timings/storage_snapshot_close":"283 4553094267 708874804114892391","bucket-1:by_email:timings/storage_snapshot_create":"284 24382187414 4620990875264311192","bucket-1:by_email:total_scan_duration":75989286269971,"bucket-1:mutation_queue_size":202374,"bucket-1:num_mutations_queued":3039240,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":105,"memory_quota":104857600000,"memory_used":1730621440,"needs_restart":false,"num_connections":384,"timings/stats_response":"710 191506063 1527772778489849"} 2015-08-03T20:41:05.286Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3040000 2015-08-03T20:41:05.856Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4615276, 2807638, 0 committed:false) SnapCreateDur 9.879µs SnapOpenDur 68.10789ms 2015-08-03T20:41:05.935Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2840000 QueuedCount 2583 2015-08-03T20:41:06.174Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4630362, 2815181, 0 committed:false) SnapCreateDur 31.837µs SnapOpenDur 153.846846ms 2015-08-03T20:41:06.333Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3050000 2015-08-03T20:41:06.37Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2850000 QueuedCount 2242 2015-08-03T20:41:07.382Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3060000 2015-08-03T20:41:07.852Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.06µs CommitTime 1.378044779s TotalFlushTime 2m56.668354632s TotalCommitTime 44.219236981s 2015-08-03T20:41:08.428Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3070000 2015-08-03T20:41:08.746Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4645586, 2822793, 74 committed:true) SnapCreateDur 1.404285158s SnapOpenDur 893.926412ms 2015-08-03T20:41:09.074Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4660686, 2830343, 0 committed:false) SnapCreateDur 29.066µs SnapOpenDur 43.151925ms 2015-08-03T20:41:09.098Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2860000 QueuedCount 1768 2015-08-03T20:41:09.418Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4675576, 2837788, 0 committed:false) SnapCreateDur 23.006µs SnapOpenDur 80.891802ms 2015-08-03T20:41:09.461Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3080000 2015-08-03T20:41:10.504Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3090000 2015-08-03T20:41:10.617Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 15.927µs CommitTime 1.089431799s TotalFlushTime 2m57.216211733s TotalCommitTime 45.30866878s 2015-08-03T20:41:11.547Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3100000 2015-08-03T20:41:11.731Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4679348, 2839674, 75 committed:true) SnapCreateDur 1.11541992s SnapOpenDur 1.113583215s 2015-08-03T20:41:11.731Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 2015-08-03T20:41:11.773Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2870000 QueuedCount 1306 2015-08-03T20:41:12.066Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4694430, 2847215, 0 committed:false) SnapCreateDur 39.597µs SnapOpenDur 49.547072ms 2015-08-03T20:41:12.195Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2880000 QueuedCount 4732 2015-08-03T20:41:12.462Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4709530, 2854765, 0 committed:false) SnapCreateDur 24.35µs SnapOpenDur 99.939652ms 2015-08-03T20:41:12.59Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3110000 2015-08-03T20:41:13.626Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3120000 2015-08-03T20:41:14.665Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3130000 2015-08-03T20:41:14.801Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.63µs CommitTime 976.701456ms TotalFlushTime 2m57.802328533s TotalCommitTime 46.285370236s 2015-08-03T20:41:14.801Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4713390, 2856695, 76 committed:true) SnapCreateDur 2.23429207s SnapOpenDur 27.436µs 2015-08-03T20:41:14.931Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2890000 QueuedCount 4362 2015-08-03T20:41:15.123Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4728490, 2864245, 0 committed:false) SnapCreateDur 47.977µs SnapOpenDur 46.789817ms 2015-08-03T20:41:15.34Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2900000 QueuedCount 2043 2015-08-03T20:41:15.51Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4743582, 2871791, 0 committed:false) SnapCreateDur 23.743µs SnapOpenDur 91.967942ms 2015-08-03T20:41:15.709Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3140000 2015-08-03T20:41:16.688Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.55µs CommitTime 1.0554548s TotalFlushTime 2m58.368929369s TotalCommitTime 47.340825036s 2015-08-03T20:41:16.688Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4747392, 2873696, 77 committed:true) SnapCreateDur 1.082037794s SnapOpenDur 34.197µs 2015-08-03T20:41:16.758Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3150000 2015-08-03T20:41:16.817Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2910000 QueuedCount 1671 2015-08-03T20:41:16.894Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4762528, 2881264, 0 committed:false) SnapCreateDur 22.094µs SnapOpenDur 40.123978ms 2015-08-03T20:41:17.267Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4777590, 2888795, 0 committed:false) SnapCreateDur 25.331µs SnapOpenDur 88.215093ms 2015-08-03T20:41:17.315Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2920000 QueuedCount 3166 2015-08-03T20:41:17.794Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3160000 2015-08-03T20:41:18.672Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4792736, 2896368, 0 committed:false) SnapCreateDur 31.22µs SnapOpenDur 118.730312ms 2015-08-03T20:41:18.84Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3170000 2015-08-03T20:41:19.346Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 14.913µs CommitTime 552.045058ms TotalFlushTime 3m0.075101627s TotalCommitTime 47.892870094s 2015-08-03T20:41:19.346Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4796522, 2898261, 78 committed:true) SnapCreateDur 578.671289ms SnapOpenDur 51.045µs 2015-08-03T20:41:19.405Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2930000 QueuedCount 2790 2015-08-03T20:41:19.676Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4811684, 2905842, 0 committed:false) SnapCreateDur 33.777µs SnapOpenDur 43.778696ms 2015-08-03T20:41:19.822Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2940000 QueuedCount 4326 2015-08-03T20:41:19.884Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3180000 2015-08-03T20:41:20.062Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4826790, 2913395, 0 committed:false) SnapCreateDur 25.167µs SnapOpenDur 89.561419ms 2015-08-03T20:41:20.919Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3190000 2015-08-03T20:41:21.217Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2950000 QueuedCount 2006 2015-08-03T20:41:21.377Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4841896, 2920948, 0 committed:false) SnapCreateDur 25.283µs SnapOpenDur 117.149588ms 2015-08-03T20:41:21.633Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4856720, 2928360, 0 committed:false) SnapCreateDur 14.176µs SnapOpenDur 90.472151ms 2015-08-03T20:41:21.677Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2960000 QueuedCount 1429 2015-08-03T20:41:21.961Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3200000 2015-08-03T20:41:23.01Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3210000 2015-08-03T20:41:23.36Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.717µs CommitTime 1.45133255s TotalFlushTime 3m2.091940839s TotalCommitTime 49.344202644s 2015-08-03T20:41:23.361Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4867962, 2933981, 79 committed:true) SnapCreateDur 1.480563757s SnapOpenDur 69.68µs 2015-08-03T20:41:23.558Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2970000 QueuedCount 2950 2015-08-03T20:41:23.711Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4883086, 2941543, 0 committed:false) SnapCreateDur 24.477µs SnapOpenDur 43.855358ms 2015-08-03T20:41:23.988Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2980000 QueuedCount 658 2015-08-03T20:41:24.053Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3220000 2015-08-03T20:41:24.114Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4898264, 2949132, 0 committed:false) SnapCreateDur 27.423µs SnapOpenDur 93.632433ms 2015-08-03T20:41:25.089Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3230000 2015-08-03T20:41:25.373Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4913320, 2956660, 0 committed:false) SnapCreateDur 22.375µs SnapOpenDur 73.160949ms 2015-08-03T20:41:25.456Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 2990000 QueuedCount 229 2015-08-03T20:41:26.137Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3240000 2015-08-03T20:41:26.181Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.547µs CommitTime 683.869622ms TotalFlushTime 3m3.852813183s TotalCommitTime 50.028072266s 2015-08-03T20:41:26.181Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4917084, 2958542, 80 committed:true) SnapCreateDur 712.360235ms SnapOpenDur 57.93µs 2015-08-03T20:41:26.511Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4932262, 2966131, 0 committed:false) SnapCreateDur 26.157µs SnapOpenDur 44.425234ms 2015-08-03T20:41:26.612Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3000000 QueuedCount 5524 2015-08-03T20:41:26.964Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4947210, 2973605, 0 committed:false) SnapCreateDur 39.13µs SnapOpenDur 115.742657ms 2015-08-03T20:41:27.178Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3250000 2015-08-03T20:41:28.145Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3010000 QueuedCount 3286 2015-08-03T20:41:28.215Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3260000 2015-08-03T20:41:28.35Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4962462, 2981231, 0 committed:false) SnapCreateDur 42.407µs SnapOpenDur 128.767179ms 2015-08-03T20:41:29.264Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3270000 2015-08-03T20:41:29.266Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.814µs CommitTime 722.640179ms TotalFlushTime 3m5.76657717s TotalCommitTime 50.750712445s 2015-08-03T20:41:29.267Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4970056, 2985028, 81 committed:true) SnapCreateDur 748.355926ms SnapOpenDur 31.76µs 2015-08-03T20:41:29.35Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3020000 QueuedCount 2739 2015-08-03T20:41:29.478Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4984870, 2992435, 0 committed:false) SnapCreateDur 22.676µs SnapOpenDur 26.435867ms 2015-08-03T20:41:29.616Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3030000 QueuedCount 1555 2015-08-03T20:41:29.708Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 4998444, 2999222, 0 committed:false) SnapCreateDur 16.418µs SnapOpenDur 55.496898ms 2015-08-03T20:41:30.302Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3280000 2015-08-03T20:41:31.085Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5013476, 3006738, 0 committed:false) SnapCreateDur 22.423µs SnapOpenDur 112.341912ms 2015-08-03T20:41:31.144Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3040000 QueuedCount 3085 2015-08-03T20:41:31.34Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3290000 2015-08-03T20:41:31.567Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5028520, 3014260, 0 committed:false) SnapCreateDur 27.753µs SnapOpenDur 165.851138ms 2015-08-03T20:41:32.378Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3300000 2015-08-03T20:41:32.502Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 16.454µs CommitTime 813.780004ms TotalFlushTime 3m7.639018667s TotalCommitTime 51.564492449s 2015-08-03T20:41:32.502Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5032408, 3016204, 82 committed:true) SnapCreateDur 840.078993ms SnapOpenDur 48.88µs 2015-08-03T20:41:32.563Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3050000 QueuedCount 2558 2015-08-03T20:41:32.833Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5047328, 3023664, 0 committed:false) SnapCreateDur 25.037µs SnapOpenDur 45.859754ms 2015-08-03T20:41:32.988Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3060000 QueuedCount 4062 2015-08-03T20:41:33.236Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5062396, 3031198, 0 committed:false) SnapCreateDur 27.25µs SnapOpenDur 94.448108ms 2015-08-03T20:41:33.422Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3310000 2015-08-03T20:41:34.456Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3070000 QueuedCount 1743 2015-08-03T20:41:34.462Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3320000 2015-08-03T20:41:34.614Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5077550, 3038775, 0 committed:false) SnapCreateDur 22.384µs SnapOpenDur 120.688454ms 2015-08-03T20:41:35.28Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.103µs CommitTime 544.25652ms TotalFlushTime 3m9.449757991s TotalCommitTime 52.108748969s 2015-08-03T20:41:35.28Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5081254, 3040627, 83 committed:true) SnapCreateDur 571.380064ms SnapOpenDur 41.725µs 2015-08-03T20:41:35.508Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3330000 2015-08-03T20:41:35.517Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3080000 QueuedCount 1306 2015-08-03T20:41:35.613Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5096372, 3048186, 0 committed:false) SnapCreateDur 32.377µs SnapOpenDur 46.693021ms 2015-08-03T20:41:36.002Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5111608, 3055804, 0 committed:false) SnapCreateDur 34.181µs SnapOpenDur 92.762422ms 2015-08-03T20:41:36.062Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3090000 QueuedCount 2864 2015-08-03T20:41:36.552Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3340000 2015-08-03T20:41:37.3Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5126668, 3063334, 0 committed:false) SnapCreateDur 14.475µs SnapOpenDur 70.63084ms 2015-08-03T20:41:37.435Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3100000 QueuedCount 4428 2015-08-03T20:41:37.599Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3350000 2015-08-03T20:41:37.705Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5141886, 3070943, 0 committed:false) SnapCreateDur 26.263µs SnapOpenDur 110.629496ms 2015-08-03T20:41:38.639Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3360000 2015-08-03T20:41:38.851Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.953µs CommitTime 1.043960987s TotalFlushTime 3m11.467800004s TotalCommitTime 53.152709956s 2015-08-03T20:41:38.851Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5145608, 3072804, 84 committed:true) SnapCreateDur 1.071094578s SnapOpenDur 33.943µs 2015-08-03T20:41:38.95Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3110000 QueuedCount 4050 2015-08-03T20:41:39.083Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5160826, 3080413, 0 committed:false) SnapCreateDur 24.337µs SnapOpenDur 47.192492ms 2015-08-03T20:41:39.32Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3120000 QueuedCount 1704 2015-08-03T20:41:39.477Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5175854, 3087927, 0 committed:false) SnapCreateDur 29.093µs SnapOpenDur 97.312252ms 2015-08-03T20:41:39.685Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3370000 2015-08-03T20:41:40.719Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3380000 2015-08-03T20:41:40.851Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5191002, 3095501, 0 committed:false) SnapCreateDur 22.89µs SnapOpenDur 127.019366ms 2015-08-03T20:41:40.896Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3130000 QueuedCount 1329 2015-08-03T20:41:41.615Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.536µs CommitTime 643.74003ms TotalFlushTime 3m13.160946107s TotalCommitTime 53.796449986s 2015-08-03T20:41:41.615Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5194818, 3097409, 85 committed:true) SnapCreateDur 670.060597ms SnapOpenDur 57.37µs 2015-08-03T20:41:41.731Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:437220919, Disk:2037760000, ExtraSnapshotData:0, Fragmentation:78.54404252708856% 2015-08-03T20:41:41.731Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:41:41.753Z-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 4970056. Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T20:41:41.765Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3390000 2015-08-03T20:41:42.336Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5209882, 3104941, 0 committed:false) SnapCreateDur 26.05µs SnapOpenDur 31.625166ms 2015-08-03T20:41:42.398Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3140000 QueuedCount 2870 2015-08-03T20:41:42.734Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5224998, 3112499, 0 committed:false) SnapCreateDur 21.834µs SnapOpenDur 81.681799ms 2015-08-03T20:41:42.812Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3400000 2015-08-03T20:41:42.879Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3150000 QueuedCount 4281 2015-08-03T20:41:43.84Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3410000 2015-08-03T20:41:44.085Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5239964, 3119982, 0 committed:false) SnapCreateDur 31.19µs SnapOpenDur 105.751495ms 2015-08-03T20:41:44.599Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 15.193µs CommitTime 391.750738ms TotalFlushTime 3m15.380070156s TotalCommitTime 54.188200724s 2015-08-03T20:41:44.599Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5243822, 3121911, 86 committed:true) SnapCreateDur 418.46083ms SnapOpenDur 62.45µs 2015-08-03T20:41:44.752Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3160000 QueuedCount 3871 2015-08-03T20:41:44.884Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3420000 2015-08-03T20:41:44.931Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5258874, 3129437, 0 committed:false) SnapCreateDur 25.187µs SnapOpenDur 46.377861ms 2015-08-03T20:41:45.156Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3170000 QueuedCount 1607 2015-08-03T20:41:45.31Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5274104, 3137052, 0 committed:false) SnapCreateDur 31.667µs SnapOpenDur 93.053023ms 2015-08-03T20:41:45.928Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3430000 2015-08-03T20:41:46.677Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5289202, 3144601, 0 committed:false) SnapCreateDur 31.1µs SnapOpenDur 112.057755ms 2015-08-03T20:41:46.728Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3180000 QueuedCount 3068 2015-08-03T20:41:46.972Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3440000 2015-08-03T20:41:47.146Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5304360, 3152180, 0 committed:false) SnapCreateDur 33.02µs SnapOpenDur 172.919292ms 2015-08-03T20:41:48.028Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3450000 2015-08-03T20:41:48.129Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.534µs CommitTime 861.606057ms TotalFlushTime 3m17.424581873s TotalCommitTime 55.049806781s 2015-08-03T20:41:48.129Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5308166, 3154083, 87 committed:true) SnapCreateDur 888.83399ms SnapOpenDur 35.616µs 2015-08-03T20:41:48.175Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3190000 QueuedCount 977 2015-08-03T20:41:48.419Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5323114, 3161557, 0 committed:false) SnapCreateDur 20.708µs SnapOpenDur 24.83807ms 2015-08-03T20:41:48.507Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3200000 QueuedCount 2281 2015-08-03T20:41:48.68Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5338396, 3169198, 0 committed:false) SnapCreateDur 25.493µs SnapOpenDur 96.675713ms 2015-08-03T20:41:49.064Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3460000 2015-08-03T20:41:49.742Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3210000 QueuedCount 1882 2015-08-03T20:41:49.899Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5353538, 3176769, 0 committed:false) SnapCreateDur 21.85µs SnapOpenDur 116.119401ms 2015-08-03T20:41:50.098Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3470000 2015-08-03T20:41:50.345Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5368750, 3184375, 0 committed:false) SnapCreateDur 21.28µs SnapOpenDur 162.559856ms 2015-08-03T20:41:50.382Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3220000 QueuedCount 1501 2015-08-03T20:41:51.145Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3480000 2015-08-03T20:41:51.445Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 13.153µs CommitTime 979.33806ms TotalFlushTime 3m19.170759922s TotalCommitTime 56.029144841s 2015-08-03T20:41:51.445Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5372474, 3186237, 88 committed:true) SnapCreateDur 1.005525208s SnapOpenDur 39.379µs 2015-08-03T20:41:51.619Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5387544, 3193772, 0 committed:false) SnapCreateDur 23.369µs SnapOpenDur 27.054633ms 2015-08-03T20:41:51.657Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3230000 QueuedCount 1091 2015-08-03T20:41:51.91Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5402714, 3201357, 0 committed:false) SnapCreateDur 21.946µs SnapOpenDur 96.006805ms 2015-08-03T20:41:52.049Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3240000 QueuedCount 4487 2015-08-03T20:41:52.185Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3490000 2015-08-03T20:41:53.213Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3500000 2015-08-03T20:41:53.278Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5417708, 3208854, 0 committed:false) SnapCreateDur 26.284µs SnapOpenDur 121.950106ms 2015-08-03T20:41:53.51Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3250000 QueuedCount 2155 2015-08-03T20:41:54.265Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3510000 2015-08-03T20:41:54.436Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.317µs CommitTime 824.900316ms TotalFlushTime 3m20.928489309s TotalCommitTime 56.854045157s 2015-08-03T20:41:54.436Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5432802, 3216401, 89 committed:true) SnapCreateDur 852.168656ms SnapOpenDur 37.153µs 2015-08-03T20:41:54.771Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5447882, 3223941, 0 committed:false) SnapCreateDur 32.103µs SnapOpenDur 38.776859ms 2015-08-03T20:41:54.798Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3260000 QueuedCount 1756 2015-08-03T20:41:55.193Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5462942, 3231471, 0 committed:false) SnapCreateDur 25.93µs SnapOpenDur 95.616526ms 2015-08-03T20:41:55.291Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3520000 2015-08-03T20:41:55.307Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3270000 QueuedCount 5201 2015-08-03T20:41:56.337Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3530000 2015-08-03T20:41:56.346Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5478094, 3239047, 0 committed:false) SnapCreateDur 11.923µs SnapOpenDur 66.679844ms 2015-08-03T20:41:57.27Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.84µs CommitTime 739.683548ms TotalFlushTime 3m22.658247488s TotalCommitTime 57.593728705s 2015-08-03T20:41:57.39Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3540000 2015-08-03T20:41:58.435Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3550000 2015-08-03T20:41:58.447Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5485688, 3242844, 90 committed:true) SnapCreateDur 766.236046ms SnapOpenDur 1.177060841s 2015-08-03T20:41:58.509Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3280000 QueuedCount 2878 2015-08-03T20:41:58.799Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5500754, 3250377, 0 committed:false) SnapCreateDur 24.68µs SnapOpenDur 46.25792ms 2015-08-03T20:41:58.935Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3290000 QueuedCount 4421 2015-08-03T20:41:59.177Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5515912, 3257956, 0 committed:false) SnapCreateDur 22.893µs SnapOpenDur 92.817136ms 2015-08-03T20:41:59.473Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3560000 2015-08-03T20:42:00.513Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3570000 2015-08-03T20:42:01.221Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3300000 QueuedCount 2100 2015-08-03T20:42:01.433Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5531034, 3265517, 0 committed:false) SnapCreateDur 23.534µs SnapOpenDur 130.555426ms 2015-08-03T20:42:01.558Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3580000 2015-08-03T20:42:02.591Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3590000 2015-08-03T20:42:02.75Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 13.176µs CommitTime 692.999224ms TotalFlushTime 3m25.837115979s TotalCommitTime 58.286727929s 2015-08-03T20:42:02.75Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5534814, 3267407, 91 committed:true) SnapCreateDur 721.936579ms SnapOpenDur 28.214µs 2015-08-03T20:42:02.974Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3310000 QueuedCount 1755 2015-08-03T20:42:03.064Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5550012, 3275006, 0 committed:false) SnapCreateDur 13.887µs SnapOpenDur 25.974044ms 2015-08-03T20:42:03.441Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5565136, 3282568, 0 committed:false) SnapCreateDur 25.99µs SnapOpenDur 82.788544ms 2015-08-03T20:42:03.481Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3320000 QueuedCount 1332 2015-08-03T20:42:03.661Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3600000 2015-08-03T20:42:04.766Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3610000 2015-08-03T20:42:05.107Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5580222, 3290111, 0 committed:false) SnapCreateDur 21.134µs SnapOpenDur 111.610132ms 2015-08-03T20:42:05.211Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":8330415818,"bucket-1:by_email:avg_scan_wait_latency":8329501762,"bucket-1:by_email:avg_ts_interval":1139767687,"bucket-1:by_email:avg_ts_items_count":7307,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":468054288,"bucket-1:by_email:delete_bytes":97719994,"bucket-1:by_email:disk_size":2493906944,"bucket-1:by_email:flush_queue_size":0,"bucket-1:by_email:frag_percent":81,"bucket-1:by_email:get_bytes":100271356,"bucket-1:by_email:insert_bytes":343618485,"bucket-1:by_email:items_count":1000000,"bucket-1:by_email:num_commits":91,"bucket-1:by_email:num_compactions":5,"bucket-1:by_email:num_completed_requests":11139,"bucket-1:by_email:num_docs_indexed":3312026,"bucket-1:by_email:num_docs_pending":339,"bucket-1:by_email:num_docs_queued":283439,"bucket-1:by_email:num_flush_queued":3328996,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":11523,"bucket-1:by_email:num_rows_returned":174,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":433,"bucket-1:by_email:scan_bytes_read":6960,"bucket-1:by_email:scan_wait_duration":95980848811871,"bucket-1:by_email:since_last_snapshot":1666613644,"bucket-1:by_email:timings/storage_clone_handle":"11139 678837897 2730590758255319","bucket-1:by_email:timings/storage_commit":"91 58286727929 7433176246726437165","bucket-1:by_email:timings/storage_del":"2291967 28635706383 -5264852881415562887","bucket-1:by_email:timings/storage_get":"3328996 44290141214 8278373570733585302","bucket-1:by_email:timings/storage_handle_open":"91 6838647962 -8876305252573736636","bucket-1:by_email:timings/storage_iterator_next":"174 3123835 71556692115","bucket-1:by_email:timings/storage_new_iterator":"11139 959076552 2819979340192364","bucket-1:by_email:timings/storage_set":"6583934 130725963627 -3016178325697590305","bucket-1:by_email:timings/storage_snapshot_close":"341 5379669935 723330479474889539","bucket-1:by_email:timings/storage_snapshot_create":"342 29296829884 5119099118273595342","bucket-1:by_email:total_scan_duration":95991381477736,"bucket-1:mutation_queue_size":285207,"bucket-1:num_mutations_queued":3614203,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":148,"memory_quota":104857600000,"memory_used":982429696,"needs_restart":false,"num_connections":384,"timings/stats_response":"782 217782671 1537738465498481"} 2015-08-03T20:42:05.809Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 3.46µs CommitTime 581.937814ms TotalFlushTime 3m27.926654337s TotalCommitTime 58.868665743s 2015-08-03T20:42:05.81Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5583934, 3291967, 92 committed:true) SnapCreateDur 608.316266ms SnapOpenDur 33.879µs 2015-08-03T20:42:05.813Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3620000 2015-08-03T20:42:05.864Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3330000 QueuedCount 2850 2015-08-03T20:42:06.148Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5599128, 3299564, 0 committed:false) SnapCreateDur 24.954µs SnapOpenDur 44.54494ms 2015-08-03T20:42:06.254Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3340000 QueuedCount 4338 2015-08-03T20:42:06.48Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5614120, 3307060, 0 committed:false) SnapCreateDur 26.356µs SnapOpenDur 84.053966ms 2015-08-03T20:42:06.865Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3630000 2015-08-03T20:42:07.639Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3350000 QueuedCount 1957 2015-08-03T20:42:07.811Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5629156, 3314578, 0 committed:false) SnapCreateDur 23.45µs SnapOpenDur 125.851485ms 2015-08-03T20:42:07.915Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3640000 2015-08-03T20:42:08.206Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5644336, 3322168, 0 committed:false) SnapCreateDur 28.643µs SnapOpenDur 97.136633ms 2015-08-03T20:42:08.24Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3360000 QueuedCount 1557 2015-08-03T20:42:08.958Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3650000 2015-08-03T20:42:09.354Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.197µs CommitTime 1.027666387s TotalFlushTime 3m29.895161318s TotalCommitTime 59.89633213s 2015-08-03T20:42:09.354Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5648020, 3324010, 93 committed:true) SnapCreateDur 1.053849747s SnapOpenDur 37.845µs 2015-08-03T20:42:09.676Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5663102, 3331551, 0 committed:false) SnapCreateDur 34.496µs SnapOpenDur 46.022698ms 2015-08-03T20:42:09.725Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3370000 QueuedCount 3077 2015-08-03T20:42:10.009Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3660000 2015-08-03T20:42:10.16Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5678286, 3339143, 0 committed:false) SnapCreateDur 32.44µs SnapOpenDur 88.842215ms 2015-08-03T20:42:10.295Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3380000 QueuedCount 4619 2015-08-03T20:42:11.036Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3670000 2015-08-03T20:42:11.437Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5693400, 3346700, 0 committed:false) SnapCreateDur 23.927µs SnapOpenDur 122.676072ms 2015-08-03T20:42:12.077Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3680000 2015-08-03T20:42:12.232Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.57µs CommitTime 705.056106ms TotalFlushTime 3m31.651036543s TotalCommitTime 1m0.601388236s 2015-08-03T20:42:12.232Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5697206, 3348603, 94 committed:true) SnapCreateDur 731.186762ms SnapOpenDur 34.884µs 2015-08-03T20:42:12.365Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3390000 QueuedCount 4178 2015-08-03T20:42:12.526Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5712250, 3356125, 0 committed:false) SnapCreateDur 16.147µs SnapOpenDur 26.471861ms 2015-08-03T20:42:12.669Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3400000 QueuedCount 1901 2015-08-03T20:42:12.803Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5727470, 3363735, 0 committed:false) SnapCreateDur 22.313µs SnapOpenDur 91.731755ms 2015-08-03T20:42:13.115Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3690000 2015-08-03T20:42:14.155Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5742598, 3371299, 0 committed:false) SnapCreateDur 20.68µs SnapOpenDur 115.523032ms 2015-08-03T20:42:14.162Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3700000 2015-08-03T20:42:14.192Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3410000 QueuedCount 1485 2015-08-03T20:42:14.614Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5757656, 3378828, 0 committed:false) SnapCreateDur 37.168µs SnapOpenDur 154.473271ms 2015-08-03T20:42:15.201Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3710000 2015-08-03T20:42:15.635Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.61µs CommitTime 898.803534ms TotalFlushTime 3m33.567764665s TotalCommitTime 1m1.50019177s 2015-08-03T20:42:15.635Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5761398, 3380699, 95 committed:true) SnapCreateDur 926.937499ms SnapOpenDur 34.978µs 2015-08-03T20:42:15.673Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3420000 QueuedCount 1100 2015-08-03T20:42:15.824Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5776412, 3388206, 0 committed:false) SnapCreateDur 12.992µs SnapOpenDur 22.845205ms 2015-08-03T20:42:15.951Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3430000 QueuedCount 4541 2015-08-03T20:42:16.145Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5791658, 3395829, 0 committed:false) SnapCreateDur 21.417µs SnapOpenDur 47.086631ms 2015-08-03T20:42:16.236Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3720000 2015-08-03T20:42:17.229Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3440000 QueuedCount 2181 2015-08-03T20:42:17.286Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3730000 2015-08-03T20:42:17.408Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5806696, 3403348, 0 committed:false) SnapCreateDur 21.41µs SnapOpenDur 129.33261ms 2015-08-03T20:42:17.872Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5821848, 3410924, 0 committed:false) SnapCreateDur 21.153µs SnapOpenDur 169.244179ms 2015-08-03T20:42:17.898Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3450000 QueuedCount 1802 2015-08-03T20:42:18.327Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3740000 2015-08-03T20:42:18.981Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.477µs CommitTime 988.496798ms TotalFlushTime 3m35.368143157s TotalCommitTime 1m2.488688568s 2015-08-03T20:42:18.981Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5825642, 3412821, 96 committed:true) SnapCreateDur 1.015012049s SnapOpenDur 37.84µs 2015-08-03T20:42:19.199Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5840630, 3420315, 0 committed:false) SnapCreateDur 21.63µs SnapOpenDur 43.041212ms 2015-08-03T20:42:19.243Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3460000 QueuedCount 1366 2015-08-03T20:42:19.384Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3750000 2015-08-03T20:42:19.582Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5855814, 3427907, 0 committed:false) SnapCreateDur 22.056µs SnapOpenDur 93.028666ms 2015-08-03T20:42:19.7Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3470000 QueuedCount 4809 2015-08-03T20:42:20.411Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3760000 2015-08-03T20:42:20.867Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5870890, 3435445, 0 committed:false) SnapCreateDur 20.39µs SnapOpenDur 120.54869ms 2015-08-03T20:42:21.45Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3770000 2015-08-03T20:42:21.792Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.336µs CommitTime 741.868016ms TotalFlushTime 3m37.018014117s TotalCommitTime 1m3.230556584s 2015-08-03T20:42:21.792Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5878480, 3439240, 97 committed:true) SnapCreateDur 767.920994ms SnapOpenDur 37.608µs 2015-08-03T20:42:21.868Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3480000 QueuedCount 4436 2015-08-03T20:42:22.121Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5893706, 3446853, 0 committed:false) SnapCreateDur 20.726µs SnapOpenDur 45.162451ms 2015-08-03T20:42:22.263Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3490000 QueuedCount 4078 2015-08-03T20:42:22.483Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5908794, 3454397, 0 committed:false) SnapCreateDur 36.62µs SnapOpenDur 88.588581ms 2015-08-03T20:42:22.488Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3780000 2015-08-03T20:42:23.533Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3790000 2015-08-03T20:42:24.565Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3800000 2015-08-03T20:42:24.999Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 2015-08-03T20:42:25.088Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3500000 QueuedCount 1823 2015-08-03T20:42:25.186Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5924104, 3462052, 0 committed:false) SnapCreateDur 34.401µs SnapOpenDur 26.382052ms 2015-08-03T20:42:25.613Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3810000 2015-08-03T20:42:26.664Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3820000 2015-08-03T20:42:27.178Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 16.793µs CommitTime 395.644476ms TotalFlushTime 3m40.251822689s TotalCommitTime 1m3.62620106s 2015-08-03T20:42:27.178Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5927872, 3463936, 98 committed:true) SnapCreateDur 1.857631375s SnapOpenDur 35.62µs 2015-08-03T20:42:27.317Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3510000 QueuedCount 3422 2015-08-03T20:42:27.495Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5950654, 3475327, 0 committed:false) SnapCreateDur 25.647µs SnapOpenDur 75.169839ms 2015-08-03T20:42:27.698Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3520000 QueuedCount 6800 2015-08-03T20:42:27.71Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3830000 2015-08-03T20:42:28.752Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3840000 2015-08-03T20:42:28.939Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5973296, 3486648, 0 committed:false) SnapCreateDur 22.514µs SnapOpenDur 131.724746ms 2015-08-03T20:42:29.757Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.974µs CommitTime 696.304805ms TotalFlushTime 3m41.800709922s TotalCommitTime 1m4.322505865s 2015-08-03T20:42:29.757Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5977020, 3488510, 99 committed:true) SnapCreateDur 723.159241ms SnapOpenDur 33.29µs 2015-08-03T20:42:29.788Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3850000 2015-08-03T20:42:29.807Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3530000 QueuedCount 2484 2015-08-03T20:42:30.019Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3540000 QueuedCount 127 2015-08-03T20:42:30.096Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 5999546, 3499773, 0 committed:false) SnapCreateDur 30.643µs SnapOpenDur 67.651069ms 2015-08-03T20:42:30.835Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3860000 2015-08-03T20:42:31.141Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3550000 QueuedCount 1632 2015-08-03T20:42:31.255Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6022196, 3511098, 0 committed:false) SnapCreateDur 14.994µs SnapOpenDur 74.726255ms 2015-08-03T20:42:31.575Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3560000 QueuedCount 3170 2015-08-03T20:42:31.866Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3870000 2015-08-03T20:42:31.879Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6044916, 3522458, 0 committed:false) SnapCreateDur 19.536µs SnapOpenDur 192.335776ms 2015-08-03T20:42:32.908Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3880000 2015-08-03T20:42:33.119Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3570000 QueuedCount 785 2015-08-03T20:42:33.744Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.38µs CommitTime 572.967038ms TotalFlushTime 3m44.70410113s TotalCommitTime 1m4.895472903s 2015-08-03T20:42:33.744Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6059888, 3529944, 100 committed:true) SnapCreateDur 599.419524ms SnapOpenDur 45.483µs 2015-08-03T20:42:33.943Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3890000 2015-08-03T20:42:34.083Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3580000 QueuedCount 2390 2015-08-03T20:42:34.244Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6082752, 3541376, 0 committed:false) SnapCreateDur 33.153µs SnapOpenDur 76.843469ms 2015-08-03T20:42:34.986Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3900000 2015-08-03T20:42:35.506Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3590000 QueuedCount 3929 2015-08-03T20:42:35.731Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6105460, 3552730, 0 committed:false) SnapCreateDur 24.74µs SnapOpenDur 142.067584ms 2015-08-03T20:42:36.033Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3910000 2015-08-03T20:42:36.56Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 15.79µs CommitTime 725.674599ms TotalFlushTime 3m46.436349258s TotalCommitTime 1m5.621147502s 2015-08-03T20:42:36.56Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6109230, 3554615, 101 committed:true) SnapCreateDur 755.024184ms SnapOpenDur 47.323µs 2015-08-03T20:42:36.732Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3600000 QueuedCount 6470 2015-08-03T20:42:37.022Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6130102, 3565051, 0 committed:false) SnapCreateDur 25.383µs SnapOpenDur 68.434791ms 2015-08-03T20:42:37.082Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3920000 2015-08-03T20:42:37.18Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3610000 QueuedCount 7971 2015-08-03T20:42:38.113Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3930000 2015-08-03T20:42:38.436Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6152744, 3576372, 0 committed:false) SnapCreateDur 24.46µs SnapOpenDur 119.727299ms 2015-08-03T20:42:38.538Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3620000 QueuedCount 5616 2015-08-03T20:42:39.084Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6175164, 3587582, 0 committed:false) SnapCreateDur 36.63µs SnapOpenDur 212.567808ms 2015-08-03T20:42:39.132Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3630000 QueuedCount 1283 2015-08-03T20:42:39.158Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3940000 2015-08-03T20:42:40.199Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3950000 2015-08-03T20:42:40.299Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.473µs CommitTime 1.081879597s TotalFlushTime 3m48.518791013s TotalCommitTime 1m6.703027099s 2015-08-03T20:42:40.299Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6178936, 3589468, 102 committed:true) SnapCreateDur 1.110982073s SnapOpenDur 27.247µs 2015-08-03T20:42:40.481Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3640000 QueuedCount 2753 2015-08-03T20:42:40.614Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6201486, 3600743, 0 committed:false) SnapCreateDur 29.453µs SnapOpenDur 77.663728ms 2015-08-03T20:42:41.236Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3960000 2015-08-03T20:42:41.774Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3650000 QueuedCount 4285 2015-08-03T20:42:41.984Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6224194, 3612097, 0 committed:false) SnapCreateDur 33.323µs SnapOpenDur 126.675881ms 2015-08-03T20:42:42.123Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3660000 QueuedCount 3923 2015-08-03T20:42:42.277Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3970000 2015-08-03T20:42:43.314Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3980000 2015-08-03T20:42:43.488Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 13.69µs CommitTime 1.224645528s TotalFlushTime 3m50.140317822s TotalCommitTime 1m7.927672627s 2015-08-03T20:42:43.489Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6246896, 3623448, 103 committed:true) SnapCreateDur 1.251037135s SnapOpenDur 59.73µs 2015-08-03T20:42:43.596Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3670000 QueuedCount 3478 2015-08-03T20:42:43.827Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6269658, 3634829, 0 committed:false) SnapCreateDur 28.783µs SnapOpenDur 75.074095ms 2015-08-03T20:42:43.947Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3680000 QueuedCount 6968 2015-08-03T20:42:44.353Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3990000 2015-08-03T20:42:45.333Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6292338, 3646169, 0 committed:false) SnapCreateDur 33.647µs SnapOpenDur 133.343938ms 2015-08-03T20:42:45.388Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4000000 2015-08-03T20:42:45.397Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3690000 QueuedCount 826 2015-08-03T20:42:45.848Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.137µs CommitTime 406.460499ms TotalFlushTime 3m51.77014349s TotalCommitTime 1m8.334133126s 2015-08-03T20:42:45.848Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6296124, 3648062, 104 committed:true) SnapCreateDur 420.252125ms SnapOpenDur 34.018µs 2015-08-03T20:42:46.084Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3700000 QueuedCount 2368 2015-08-03T20:42:46.189Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6318814, 3659407, 0 committed:false) SnapCreateDur 25.305µs SnapOpenDur 46.595922ms 2015-08-03T20:42:46.436Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4010000 2015-08-03T20:42:47.044Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3710000 QueuedCount 3930 2015-08-03T20:42:47.255Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6341556, 3670778, 0 committed:false) SnapCreateDur 22.754µs SnapOpenDur 131.676212ms 2015-08-03T20:42:47.486Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4020000 2015-08-03T20:42:47.507Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3720000 QueuedCount 5386 2015-08-03T20:42:47.925Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6364052, 3682026, 0 committed:false) SnapCreateDur 43.501µs SnapOpenDur 216.611693ms 2015-08-03T20:42:48.536Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4030000 2015-08-03T20:42:49.12Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3730000 QueuedCount 4906 2015-08-03T20:42:49.317Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6382806, 3691403, 0 committed:false) SnapCreateDur 22.362µs SnapOpenDur 92.945274ms 2015-08-03T20:42:49.57Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4040000 2015-08-03T20:42:50.292Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.336µs CommitTime 853.892041ms TotalFlushTime 3m54.670229848s TotalCommitTime 1m9.188025167s 2015-08-03T20:42:50.292Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6386708, 3693354, 105 committed:true) SnapCreateDur 880.395096ms SnapOpenDur 36.643µs 2015-08-03T20:42:50.379Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3740000 QueuedCount 2653 2015-08-03T20:42:50.599Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6409456, 3704728, 0 committed:false) SnapCreateDur 20.711µs SnapOpenDur 43.845113ms 2015-08-03T20:42:50.611Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4050000 2015-08-03T20:42:50.679Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3750000 QueuedCount 4163 2015-08-03T20:42:51.655Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4060000 2015-08-03T20:42:51.98Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6432110, 3716055, 0 committed:false) SnapCreateDur 33.85µs SnapOpenDur 128.723699ms 2015-08-03T20:42:52.011Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3760000 QueuedCount 1841 2015-08-03T20:42:52.369Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3770000 QueuedCount 1467 2015-08-03T20:42:52.646Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6454794, 3727397, 0 committed:false) SnapCreateDur 38.557µs SnapOpenDur 215.94767ms 2015-08-03T20:42:52.7Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4070000 2015-08-03T20:42:53.737Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4080000 2015-08-03T20:42:54.203Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.707µs CommitTime 1.423429178s TotalFlushTime 3m56.59256114s TotalCommitTime 1m10.611454345s 2015-08-03T20:42:54.203Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6458596, 3729298, 106 committed:true) SnapCreateDur 1.452217906s SnapOpenDur 59.753µs 2015-08-03T20:42:54.509Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3780000 QueuedCount 4930 2015-08-03T20:42:54.764Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6481338, 3740669, 0 committed:false) SnapCreateDur 27.795µs SnapOpenDur 76.539961ms 2015-08-03T20:42:54.777Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4090000 2015-08-03T20:42:54.984Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3790000 QueuedCount 6462 2015-08-03T20:42:54.999Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:439506257, Disk:2008133632, ExtraSnapshotData:0, Fragmentation:78.11369472646729% 2015-08-03T20:42:54.999Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:42:55.021Z-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 6178936. Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T20:42:55.81Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4100000 2015-08-03T20:42:56.381Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6504018, 3752009, 0 committed:false) SnapCreateDur 24.037µs SnapOpenDur 30.729494ms 2015-08-03T20:42:56.86Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4110000 2015-08-03T20:42:57.225Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.28µs CommitTime 721.988427ms TotalFlushTime 3m58.655121158s TotalCommitTime 1m11.333442772s 2015-08-03T20:42:57.225Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6507882, 3753941, 107 committed:true) SnapCreateDur 749.033792ms SnapOpenDur 28.28µs 2015-08-03T20:42:57.299Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3800000 QueuedCount 4170 2015-08-03T20:42:57.69Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6530320, 3765160, 0 committed:false) SnapCreateDur 18.933µs SnapOpenDur 63.69035ms 2015-08-03T20:42:57.718Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3810000 QueuedCount 1878 2015-08-03T20:42:57.913Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4120000 2015-08-03T20:42:58.812Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3820000 QueuedCount 1379 2015-08-03T20:42:58.91Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6553030, 3776515, 0 committed:false) SnapCreateDur 13.04µs SnapOpenDur 68.060272ms 2015-08-03T20:42:58.954Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4130000 2015-08-03T20:42:59.088Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3830000 QueuedCount 2858 2015-08-03T20:42:59.33Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6575640, 3787820, 0 committed:false) SnapCreateDur 25.437µs SnapOpenDur 188.090789ms 2015-08-03T20:42:59.99Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4140000 2015-08-03T20:43:01.036Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4150000 2015-08-03T20:43:01.245Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3840000 QueuedCount 547 2015-08-03T20:43:01.854Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.69µs CommitTime 562.401317ms TotalFlushTime 4m2.237776551s TotalCommitTime 1m11.895844089s 2015-08-03T20:43:01.854Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6590748, 3795374, 108 committed:true) SnapCreateDur 588.786036ms SnapOpenDur 38µs 2015-08-03T20:43:02.073Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4160000 2015-08-03T20:43:02.176Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3850000 QueuedCount 2050 2015-08-03T20:43:02.32Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6613398, 3806699, 0 committed:false) SnapCreateDur 16.99µs SnapOpenDur 62.828877ms 2015-08-03T20:43:03.115Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4170000 2015-08-03T20:43:03.578Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3860000 QueuedCount 1717 2015-08-03T20:43:04.038Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.766µs CommitTime 392.235566ms TotalFlushTime 4m3.859166187s TotalCommitTime 1m12.288079655s 2015-08-03T20:43:04.038Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6632406, 3816203, 109 committed:true) SnapCreateDur 420.270586ms SnapOpenDur 37.79µs 2015-08-03T20:43:04.213Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4180000 2015-08-03T20:43:04.332Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3870000 QueuedCount 3223 2015-08-03T20:43:04.518Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6655070, 3827535, 0 committed:false) SnapCreateDur 30.33µs SnapOpenDur 68.301829ms 2015-08-03T20:43:05.212Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":9785502798,"bucket-1:by_email:avg_scan_wait_latency":9784598674,"bucket-1:by_email:avg_ts_interval":929716396,"bucket-1:by_email:avg_ts_items_count":10764,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":453984957,"bucket-1:by_email:delete_bytes":120999592,"bucket-1:by_email:disk_size":2194030592,"bucket-1:by_email:flush_queue_size":4896,"bucket-1:by_email:frag_percent":79,"bucket-1:by_email:get_bytes":123995316,"bucket-1:by_email:insert_bytes":400464077,"bucket-1:by_email:items_count":1000000,"bucket-1:by_email:num_commits":109,"bucket-1:by_email:num_compactions":6,"bucket-1:by_email:num_completed_requests":11907,"bucket-1:by_email:num_docs_indexed":3862085,"bucket-1:by_email:num_docs_pending":122,"bucket-1:by_email:num_docs_queued":313776,"bucket-1:by_email:num_flush_queued":3884850,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":12291,"bucket-1:by_email:num_rows_returned":190,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":500,"bucket-1:by_email:scan_bytes_read":7600,"bucket-1:by_email:scan_wait_duration":120262502312924,"bucket-1:by_email:since_last_snapshot":480244920,"bucket-1:by_email:timings/storage_clone_handle":"11907 718580672 2733624131274468","bucket-1:by_email:timings/storage_commit":"109 72288079655 1292021537320246945","bucket-1:by_email:timings/storage_del":"2834145 36995767439 200133883663779897","bucket-1:by_email:timings/storage_get":"3879955 50911974367 8280038773968867753","bucket-1:by_email:timings/storage_handle_open":"109 6839276138 -8876305229400674790","bucket-1:by_email:timings/storage_iterator_next":"190 3370945 75764807711","bucket-1:by_email:timings/storage_new_iterator":"11907 1021759822 2827528389599204","bucket-1:by_email:timings/storage_set":"7668289 151558532634 -6740164147703035768","bucket-1:by_email:timings/storage_snapshot_close":"390 6119854036 736732490286758776","bucket-1:by_email:timings/storage_snapshot_create":"391 34087736046 5714100579214825406","bucket-1:by_email:total_scan_duration":120273614890764,"bucket-1:mutation_queue_size":304008,"bucket-1:num_mutations_queued":4188858,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":158,"memory_quota":104857600000,"memory_used":799920128,"needs_restart":false,"num_connections":384,"timings/stats_response":"854 243762335 1547490697258581"} 2015-08-03T20:43:05.333Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4190000 2015-08-03T20:43:05.661Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3880000 QueuedCount 4850 2015-08-03T20:43:05.822Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6677956, 3838978, 0 committed:false) SnapCreateDur 11.981µs SnapOpenDur 70.254181ms 2015-08-03T20:43:06.377Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4200000 2015-08-03T20:43:06.616Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.484µs CommitTime 683.151355ms TotalFlushTime 4m5.494051205s TotalCommitTime 1m12.97123101s 2015-08-03T20:43:06.616Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6681538, 3840769, 110 committed:true) SnapCreateDur 709.218184ms SnapOpenDur 34.507µs 2015-08-03T20:43:06.762Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3890000 QueuedCount 8274 2015-08-03T20:43:07.091Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6704380, 3852190, 0 committed:false) SnapCreateDur 20.605µs SnapOpenDur 61.780633ms 2015-08-03T20:43:07.172Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3900000 QueuedCount 3991 2015-08-03T20:43:07.427Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4210000 2015-08-03T20:43:08.363Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6727020, 3863510, 0 committed:false) SnapCreateDur 21.264µs SnapOpenDur 70.441635ms 2015-08-03T20:43:08.393Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3910000 QueuedCount 1672 2015-08-03T20:43:08.466Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4220000 2015-08-03T20:43:08.739Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3920000 QueuedCount 1247 2015-08-03T20:43:08.97Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6749570, 3874785, 0 committed:false) SnapCreateDur 30.344µs SnapOpenDur 186.023583ms 2015-08-03T20:43:09.51Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4230000 2015-08-03T20:43:10.187Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 15.85µs CommitTime 1.095856577s TotalFlushTime 4m7.481818275s TotalCommitTime 1m14.067087587s 2015-08-03T20:43:10.187Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6753364, 3876682, 111 committed:true) SnapCreateDur 1.121858139s SnapOpenDur 35.321µs 2015-08-03T20:43:10.328Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3930000 QueuedCount 4646 2015-08-03T20:43:10.483Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6776148, 3888074, 0 committed:false) SnapCreateDur 23.62µs SnapOpenDur 74.654488ms 2015-08-03T20:43:10.541Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4240000 2015-08-03T20:43:10.684Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3940000 QueuedCount 6221 2015-08-03T20:43:11.586Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4250000 2015-08-03T20:43:12.266Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6798760, 3899380, 0 committed:false) SnapCreateDur 28.633µs SnapOpenDur 119.509208ms 2015-08-03T20:43:12.625Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4260000 2015-08-03T20:43:13.106Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.65µs CommitTime 719.828732ms TotalFlushTime 4m9.365317185s TotalCommitTime 1m14.786916319s 2015-08-03T20:43:13.106Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6802476, 3901238, 112 committed:true) SnapCreateDur 746.53365ms SnapOpenDur 48.077µs 2015-08-03T20:43:13.202Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3950000 QueuedCount 5845 2015-08-03T20:43:13.591Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6825260, 3912630, 0 committed:false) SnapCreateDur 32.254µs SnapOpenDur 71.213788ms 2015-08-03T20:43:13.623Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3960000 QueuedCount 1628 2015-08-03T20:43:13.661Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4270000 2015-08-03T20:43:14.708Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4280000 2015-08-03T20:43:14.839Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3970000 QueuedCount 1215 2015-08-03T20:43:14.981Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6847940, 3923970, 0 committed:false) SnapCreateDur 28.724µs SnapOpenDur 118.047554ms 2015-08-03T20:43:15.306Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3980000 QueuedCount 2755 2015-08-03T20:43:15.592Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6870624, 3935312, 0 committed:false) SnapCreateDur 20.38µs SnapOpenDur 189.059378ms 2015-08-03T20:43:15.739Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4290000 2015-08-03T20:43:16.787Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4300000 2015-08-03T20:43:16.941Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.087µs CommitTime 267.666774ms TotalFlushTime 4m12.381566046s TotalCommitTime 1m15.054583093s 2015-08-03T20:43:16.941Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6874436, 3937218, 113 committed:true) SnapCreateDur 294.960481ms SnapOpenDur 37.084µs 2015-08-03T20:43:17.781Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 3990000 QueuedCount 6240 2015-08-03T20:43:17.837Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4310000 2015-08-03T20:43:18.072Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6897180, 3948590, 0 committed:false) SnapCreateDur 31.963µs SnapOpenDur 68.916822ms 2015-08-03T20:43:18.879Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4320000 2015-08-03T20:43:19.323Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4000000 QueuedCount 7758 2015-08-03T20:43:19.915Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4330000 2015-08-03T20:43:20.471Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6919818, 3959909, 0 committed:false) SnapCreateDur 20.699µs SnapOpenDur 116.983461ms 2015-08-03T20:43:20.957Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4340000 2015-08-03T20:43:21.013Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 5.566µs CommitTime 432.977622ms TotalFlushTime 4m15.708949547s TotalCommitTime 1m15.487560715s 2015-08-03T20:43:21.013Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6923572, 3961786, 114 committed:true) SnapCreateDur 447.309356ms SnapOpenDur 27.368µs 2015-08-03T20:43:21.042Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4010000 QueuedCount 1576 2015-08-03T20:43:21.246Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4020000 QueuedCount 1107 2015-08-03T20:43:21.333Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6946084, 3973042, 0 committed:false) SnapCreateDur 21.63µs SnapOpenDur 67.440781ms 2015-08-03T20:43:21.99Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4350000 2015-08-03T20:43:22.196Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4030000 QueuedCount 2585 2015-08-03T20:43:22.366Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6968676, 3984338, 0 committed:false) SnapCreateDur 24.684µs SnapOpenDur 120.907491ms 2015-08-03T20:43:22.513Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4040000 QueuedCount 4123 2015-08-03T20:43:22.78Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 6991368, 3995684, 0 committed:false) SnapCreateDur 20.833µs SnapOpenDur 191.471307ms 2015-08-03T20:43:23.034Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4360000 2015-08-03T20:43:23.812Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4050000 QueuedCount 5652 2015-08-03T20:43:24.077Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4370000 2015-08-03T20:43:24.082Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7014072, 4007036, 0 committed:false) SnapCreateDur 35.356µs SnapOpenDur 169.921643ms 2015-08-03T20:43:25.028Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.386µs CommitTime 812.750884ms TotalFlushTime 4m18.168624289s TotalCommitTime 1m16.300311599s 2015-08-03T20:43:25.028Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7017858, 4008929, 115 committed:true) SnapCreateDur 841.415008ms SnapOpenDur 34.95µs 2015-08-03T20:43:25.115Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4380000 2015-08-03T20:43:25.138Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4060000 QueuedCount 5243 2015-08-03T20:43:25.528Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7040594, 4020297, 0 committed:false) SnapCreateDur 19.919µs SnapOpenDur 48.973358ms 2015-08-03T20:43:25.594Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4070000 QueuedCount 2939 2015-08-03T20:43:26.162Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4390000 2015-08-03T20:43:26.911Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4080000 QueuedCount 623 2015-08-03T20:43:27.001Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7063220, 4031610, 0 committed:false) SnapCreateDur 12.523µs SnapOpenDur 72.170075ms 2015-08-03T20:43:27.212Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4400000 2015-08-03T20:43:27.346Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4090000 QueuedCount 240 2015-08-03T20:43:28.228Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 17.85µs CommitTime 842.911189ms TotalFlushTime 4m20.257426115s TotalCommitTime 1m17.143222788s 2015-08-03T20:43:28.259Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4410000 2015-08-03T20:43:29.296Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4420000 2015-08-03T20:43:29.874Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7082124, 4041062, 116 committed:true) SnapCreateDur 868.905903ms SnapOpenDur 1.64675878s 2015-08-03T20:43:30.246Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4100000 QueuedCount 1775 2015-08-03T20:43:30.337Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4430000 2015-08-03T20:43:30.381Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7104848, 4052424, 0 committed:false) SnapCreateDur 22.757µs SnapOpenDur 73.396912ms 2015-08-03T20:43:31.38Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4440000 2015-08-03T20:43:31.521Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 3.713µs CommitTime 1.019307355s TotalFlushTime 4m20.713754181s TotalCommitTime 1m18.162530143s 2015-08-03T20:43:31.521Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7108604, 4054302, 117 committed:true) SnapCreateDur 1.046174247s SnapOpenDur 33.976µs 2015-08-03T20:43:31.675Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4110000 QueuedCount 5114 2015-08-03T20:43:31.842Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7131088, 4065544, 0 committed:false) SnapCreateDur 16.814µs SnapOpenDur 48.91497ms 2015-08-03T20:43:32.028Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4120000 QueuedCount 6639 2015-08-03T20:43:32.425Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4450000 2015-08-03T20:43:33.294Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7153740, 4076870, 0 committed:false) SnapCreateDur 18.726µs SnapOpenDur 120.42083ms 2015-08-03T20:43:33.436Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4130000 QueuedCount 6285 2015-08-03T20:43:33.461Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4460000 2015-08-03T20:43:34.509Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4470000 2015-08-03T20:43:34.681Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 15.437µs CommitTime 1.000899047s TotalFlushTime 4m22.565938569s TotalCommitTime 1m19.16342919s 2015-08-03T20:43:34.681Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7172726, 4086363, 118 committed:true) SnapCreateDur 1.028679426s SnapOpenDur 36.258µs 2015-08-03T20:43:34.777Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4140000 QueuedCount 3930 2015-08-03T20:43:35.009Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7195458, 4097729, 0 committed:false) SnapCreateDur 28.773µs SnapOpenDur 65.974458ms 2015-08-03T20:43:35.107Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4150000 QueuedCount 5472 2015-08-03T20:43:35.541Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4480000 2015-08-03T20:43:36.588Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4490000 2015-08-03T20:43:37.416Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 2015-08-03T20:43:37.61Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7218044, 4109022, 0 committed:false) SnapCreateDur 37.557µs SnapOpenDur 29.516622ms 2015-08-03T20:43:37.636Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4500000 2015-08-03T20:43:37.657Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4160000 QueuedCount 1237 2015-08-03T20:43:38.677Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4510000 2015-08-03T20:43:39.265Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.377µs CommitTime 398.967239ms TotalFlushTime 4m25.390077252s TotalCommitTime 1m19.562396429s 2015-08-03T20:43:39.265Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7221824, 4110912, 119 committed:true) SnapCreateDur 1.560629633s SnapOpenDur 39.811µs 2015-08-03T20:43:39.588Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4170000 QueuedCount 2736 2015-08-03T20:43:39.714Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4520000 2015-08-03T20:43:39.768Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7244452, 4122226, 0 committed:false) SnapCreateDur 37.546µs SnapOpenDur 80.92129ms 2015-08-03T20:43:40.755Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4530000 2015-08-03T20:43:41.047Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4180000 QueuedCount 3010 2015-08-03T20:43:41.247Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7264684, 4132342, 0 committed:false) SnapCreateDur 27.153µs SnapOpenDur 136.045396ms 2015-08-03T20:43:41.789Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4540000 2015-08-03T20:43:42.035Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.94µs CommitTime 657.210394ms TotalFlushTime 4m27.154468066s TotalCommitTime 1m20.219606823s 2015-08-03T20:43:42.036Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7268532, 4134266, 120 committed:true) SnapCreateDur 683.810783ms SnapOpenDur 64.109µs 2015-08-03T20:43:42.242Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4190000 QueuedCount 6453 2015-08-03T20:43:42.544Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7291106, 4145553, 0 committed:false) SnapCreateDur 28.703µs SnapOpenDur 73.402358ms 2015-08-03T20:43:42.707Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4200000 QueuedCount 7896 2015-08-03T20:43:42.834Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4550000 2015-08-03T20:43:43.878Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4560000 2015-08-03T20:43:43.898Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7313646, 4156823, 0 committed:false) SnapCreateDur 14.754µs SnapOpenDur 78.780512ms 2015-08-03T20:43:44.002Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4210000 QueuedCount 5539 2015-08-03T20:43:44.563Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7336212, 4168106, 0 committed:false) SnapCreateDur 28.748µs SnapOpenDur 224.304173ms 2015-08-03T20:43:44.61Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4220000 QueuedCount 1303 2015-08-03T20:43:44.914Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4570000 2015-08-03T20:43:45.801Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 5.069µs CommitTime 157.784526ms TotalFlushTime 4m30.222699713s TotalCommitTime 1m20.377391349s 2015-08-03T20:43:45.801Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7340026, 4170013, 121 committed:true) SnapCreateDur 173.082462ms SnapOpenDur 33.69µs 2015-08-03T20:43:45.951Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4580000 2015-08-03T20:43:45.98Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4230000 QueuedCount 2850 2015-08-03T20:43:46.07Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7362746, 4181373, 0 committed:false) SnapCreateDur 10.581µs SnapOpenDur 37.530663ms 2015-08-03T20:43:46.913Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4240000 QueuedCount 4380 2015-08-03T20:43:47.007Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4590000 2015-08-03T20:43:47.121Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7385482, 4192741, 0 committed:false) SnapCreateDur 34.34µs SnapOpenDur 121.762676ms 2015-08-03T20:43:47.284Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4250000 QueuedCount 5949 2015-08-03T20:43:47.675Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7408226, 4204113, 0 committed:false) SnapCreateDur 35.367µs SnapOpenDur 239.543432ms 2015-08-03T20:43:48.052Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4600000 2015-08-03T20:43:48.986Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.27µs CommitTime 269.324025ms TotalFlushTime 4m32.583376743s TotalCommitTime 1m20.646715374s 2015-08-03T20:43:48.986Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7411968, 4205984, 122 committed:true) SnapCreateDur 296.763983ms SnapOpenDur 48.19µs 2015-08-03T20:43:49.087Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4260000 QueuedCount 5563 2015-08-03T20:43:49.088Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4610000 2015-08-03T20:43:49.497Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7434660, 4217330, 0 committed:false) SnapCreateDur 26.9µs SnapOpenDur 77.691204ms 2015-08-03T20:43:49.542Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4270000 QueuedCount 1390 2015-08-03T20:43:50.132Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4620000 2015-08-03T20:43:50.792Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4280000 QueuedCount 927 2015-08-03T20:43:50.943Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7457406, 4228703, 0 committed:false) SnapCreateDur 26.73µs SnapOpenDur 132.417411ms 2015-08-03T20:43:51.18Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4630000 2015-08-03T20:43:51.791Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.44µs CommitTime 716.065865ms TotalFlushTime 4m34.323161776s TotalCommitTime 1m21.362781239s 2015-08-03T20:43:51.791Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7461322, 4230661, 123 committed:true) SnapCreateDur 743.139938ms SnapOpenDur 38.559µs 2015-08-03T20:43:52.084Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4290000 QueuedCount 4386 2015-08-03T20:43:52.211Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4640000 2015-08-03T20:43:52.316Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7483924, 4241962, 0 committed:false) SnapCreateDur 23.206µs SnapOpenDur 72.760084ms 2015-08-03T20:43:52.554Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4300000 QueuedCount 5829 2015-08-03T20:43:53.258Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4650000 2015-08-03T20:43:53.836Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7506492, 4253246, 0 committed:false) SnapCreateDur 25.617µs SnapOpenDur 126.247375ms 2015-08-03T20:43:53.944Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4310000 QueuedCount 3518 2015-08-03T20:43:54.291Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4660000 2015-08-03T20:43:54.929Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 3.663µs CommitTime 836.900546ms TotalFlushTime 4m36.298875273s TotalCommitTime 1m22.199681785s 2015-08-03T20:43:54.929Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7529294, 4264647, 124 committed:true) SnapCreateDur 848.810771ms SnapOpenDur 28.868µs 2015-08-03T20:43:54.99Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4320000 QueuedCount 1208 2015-08-03T20:43:55.197Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7551934, 4275967, 0 committed:false) SnapCreateDur 27.47µs SnapOpenDur 67.857216ms 2015-08-03T20:43:55.239Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4330000 QueuedCount 838 2015-08-03T20:43:55.336Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4670000 2015-08-03T20:43:56.269Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4340000 QueuedCount 499 2015-08-03T20:43:56.384Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4680000 2015-08-03T20:43:56.441Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7574732, 4287366, 0 committed:false) SnapCreateDur 43.68µs SnapOpenDur 146.065536ms 2015-08-03T20:43:56.757Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4350000 QueuedCount 1988 2015-08-03T20:43:57.051Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7597344, 4298672, 0 committed:false) SnapCreateDur 25.403µs SnapOpenDur 217.015609ms 2015-08-03T20:43:57.424Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4690000 2015-08-03T20:43:58.3Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.635µs CommitTime 158.894455ms TotalFlushTime 4m38.921103692s TotalCommitTime 1m22.35857624s 2015-08-03T20:43:58.3Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7601122, 4300561, 125 committed:true) SnapCreateDur 174.101511ms SnapOpenDur 34.292µs 2015-08-03T20:43:58.432Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4360000 QueuedCount 3529 2015-08-03T20:43:58.464Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4700000 2015-08-03T20:43:58.612Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7623824, 4311912, 0 committed:false) SnapCreateDur 31.15µs SnapOpenDur 79.002396ms 2015-08-03T20:43:58.725Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4370000 QueuedCount 3149 2015-08-03T20:43:59.51Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4710000 2015-08-03T20:43:59.742Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7646486, 4323243, 0 committed:false) SnapCreateDur 29.667µs SnapOpenDur 136.34428ms 2015-08-03T20:43:59.883Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4380000 QueuedCount 7220 2015-08-03T20:44:00.426Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7669172, 4334586, 0 committed:false) SnapCreateDur 31.057µs SnapOpenDur 219.629528ms 2015-08-03T20:44:00.511Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4390000 QueuedCount 415 2015-08-03T20:44:00.553Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4720000 2015-08-03T20:44:01.587Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4730000 2015-08-03T20:44:01.74Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.574µs CommitTime 1.179509701s TotalFlushTime 4m40.577954842s TotalCommitTime 1m23.538085941s 2015-08-03T20:44:01.74Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7672906, 4336453, 126 committed:true) SnapCreateDur 1.204014439s SnapOpenDur 35.113µs 2015-08-03T20:44:01.936Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4400000 QueuedCount 1847 2015-08-03T20:44:02.011Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7695446, 4347723, 0 committed:false) SnapCreateDur 28.581µs SnapOpenDur 37.827816ms 2015-08-03T20:44:02.63Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4740000 2015-08-03T20:44:03.246Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4410000 QueuedCount 3301 2015-08-03T20:44:03.447Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7717968, 4358984, 0 committed:false) SnapCreateDur 32.543µs SnapOpenDur 136.095033ms 2015-08-03T20:44:03.724Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4750000 2015-08-03T20:44:04.249Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.277µs CommitTime 701.461746ms TotalFlushTime 4m42.083969967s TotalCommitTime 1m24.239547687s 2015-08-03T20:44:04.25Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7721854, 4360927, 127 committed:true) SnapCreateDur 727.790859ms SnapOpenDur 33.335µs 2015-08-03T20:44:04.377Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4420000 QueuedCount 4849 2015-08-03T20:44:04.597Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7744500, 4372250, 0 committed:false) SnapCreateDur 33.77µs SnapOpenDur 71.460051ms 2015-08-03T20:44:04.696Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4430000 QueuedCount 4460 2015-08-03T20:44:04.841Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4760000 2015-08-03T20:44:05.213Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":11228901848,"bucket-1:by_email:avg_scan_wait_latency":11227988984,"bucket-1:by_email:avg_ts_interval":636723087,"bucket-1:by_email:avg_ts_items_count":8796,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":443462230,"bucket-1:by_email:delete_bytes":144519861,"bucket-1:by_email:disk_size":2044813312,"bucket-1:by_email:flush_queue_size":4639,"bucket-1:by_email:frag_percent":78,"bucket-1:by_email:get_bytes":147964419,"bucket-1:by_email:insert_bytes":457897335,"bucket-1:by_email:items_count":1000000,"bucket-1:by_email:num_commits":127,"bucket-1:by_email:num_compactions":6,"bucket-1:by_email:num_completed_requests":12675,"bucket-1:by_email:num_docs_indexed":4415686,"bucket-1:by_email:num_docs_pending":205,"bucket-1:by_email:num_docs_queued":340363,"bucket-1:by_email:num_flush_queued":4438298,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":13059,"bucket-1:by_email:num_rows_returned":205,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":561,"bucket-1:by_email:scan_bytes_read":8200,"bucket-1:by_email:scan_wait_duration":146626308154347,"bucket-1:by_email:since_last_snapshot":347946711,"bucket-1:by_email:timings/storage_clone_handle":"12675 754866727 2735529243541683","bucket-1:by_email:timings/storage_commit":"127 84239547687 -7487783183039277581","bucket-1:by_email:timings/storage_del":"3379009 49484373161 8620213406579245727","bucket-1:by_email:timings/storage_get":"4433660 58932114399 -8502433644093904917","bucket-1:by_email:timings/storage_handle_open":"127 8486612075 -6164505885685086175","bucket-1:by_email:timings/storage_iterator_next":"205 3579244 78929959592","bucket-1:by_email:timings/storage_new_iterator":"12675 1078932049 2833869971243653","bucket-1:by_email:timings/storage_set":"8758017 168426239435 4195826442424777617","bucket-1:by_email:timings/storage_snapshot_close":"433 6811265292 749984161560293846","bucket-1:by_email:timings/storage_snapshot_create":"434 38756179222 6351759410040756304","bucket-1:by_email:total_scan_duration":146638229239131,"bucket-1:mutation_queue_size":325222,"bucket-1:num_mutations_queued":4763521,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":169,"memory_quota":104857600000,"memory_used":600330240,"needs_restart":false,"num_connections":384,"timings/stats_response":"926 268638321 1556420199042113"} 2015-08-03T20:44:05.89Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4770000 2015-08-03T20:44:05.947Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7767144, 4383572, 0 committed:false) SnapCreateDur 27.327µs SnapOpenDur 136.495441ms 2015-08-03T20:44:06.007Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4440000 QueuedCount 2141 2015-08-03T20:44:06.403Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7789806, 4394903, 0 committed:false) SnapCreateDur 24.79µs SnapOpenDur 212.145314ms 2015-08-03T20:44:06.432Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4450000 QueuedCount 1741 2015-08-03T20:44:06.935Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4780000 2015-08-03T20:44:07.417Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:448876462, Disk:2047844352, ExtraSnapshotData:0, Fragmentation:78.08053812480374% 2015-08-03T20:44:07.417Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:44:07.439Z-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 7461322. Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T20:44:07.987Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4790000 2015-08-03T20:44:08.027Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 16.46µs CommitTime 732.505µs TotalFlushTime 4m44.714209756s TotalCommitTime 1m24.240280192s 2015-08-03T20:44:08.027Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7801194, 4400597, 128 committed:true) SnapCreateDur 590.259863ms SnapOpenDur 32.434µs 2015-08-03T20:44:08.224Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4460000 QueuedCount 7079 2015-08-03T20:44:08.566Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7823778, 4411889, 0 committed:false) SnapCreateDur 71.474µs SnapOpenDur 82.884998ms 2015-08-03T20:44:08.705Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4470000 QueuedCount 6692 2015-08-03T20:44:09.04Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4800000 2015-08-03T20:44:09.915Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7846602, 4423301, 0 committed:false) SnapCreateDur 12.318µs SnapOpenDur 74.740905ms 2015-08-03T20:44:09.963Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4480000 QueuedCount 534 2015-08-03T20:44:10.077Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4810000 2015-08-03T20:44:10.721Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.64µs CommitTime 715.893747ms TotalFlushTime 4m46.403838933s TotalCommitTime 1m24.956173939s 2015-08-03T20:44:10.721Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7850264, 4425132, 129 committed:true) SnapCreateDur 741.858309ms SnapOpenDur 37.016µs 2015-08-03T20:44:11.059Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4490000 QueuedCount 2053 2015-08-03T20:44:11.116Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4820000 2015-08-03T20:44:11.201Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7872930, 4436465, 0 committed:false) SnapCreateDur 19.91µs SnapOpenDur 67.436564ms 2015-08-03T20:44:12.164Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4830000 2015-08-03T20:44:12.375Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4500000 QueuedCount 3501 2015-08-03T20:44:12.514Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7895504, 4447752, 0 committed:false) SnapCreateDur 11.904µs SnapOpenDur 70.529498ms 2015-08-03T20:44:12.781Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4510000 QueuedCount 5037 2015-08-03T20:44:13.161Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7918196, 4459098, 0 committed:false) SnapCreateDur 25.844µs SnapOpenDur 194.998687ms 2015-08-03T20:44:13.207Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4840000 2015-08-03T20:44:14.253Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4850000 2015-08-03T20:44:14.654Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.123µs CommitTime 273.142833ms TotalFlushTime 4m49.557815027s TotalCommitTime 1m25.229316772s 2015-08-03T20:44:14.654Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7922032, 4461016, 130 committed:true) SnapCreateDur 299.48885ms SnapOpenDur 58.47µs 2015-08-03T20:44:14.785Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4520000 QueuedCount 6642 2015-08-03T20:44:15.152Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7944756, 4472378, 0 committed:false) SnapCreateDur 22.41µs SnapOpenDur 73.950951ms 2015-08-03T20:44:15.229Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4530000 QueuedCount 4314 2015-08-03T20:44:15.287Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4860000 2015-08-03T20:44:16.331Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4870000 2015-08-03T20:44:16.349Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4540000 QueuedCount 119 2015-08-03T20:44:16.477Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7967602, 4483801, 0 committed:false) SnapCreateDur 21.48µs SnapOpenDur 123.648271ms 2015-08-03T20:44:17.279Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 15.8µs CommitTime 690.724975ms TotalFlushTime 4m51.161159045s TotalCommitTime 1m25.920041747s 2015-08-03T20:44:17.279Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7971314, 4485657, 131 committed:true) SnapCreateDur 716.986651ms SnapOpenDur 35.057µs 2015-08-03T20:44:17.379Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4880000 2015-08-03T20:44:17.479Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4550000 QueuedCount 3571 2015-08-03T20:44:17.645Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 7994090, 4497045, 0 committed:false) SnapCreateDur 18.27µs SnapOpenDur 74.889991ms 2015-08-03T20:44:18.429Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4890000 2015-08-03T20:44:18.742Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4560000 QueuedCount 5109 2015-08-03T20:44:18.961Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8016782, 4508391, 0 committed:false) SnapCreateDur 21.371µs SnapOpenDur 124.509136ms 2015-08-03T20:44:19.161Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4570000 QueuedCount 6708 2015-08-03T20:44:19.459Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4900000 2015-08-03T20:44:19.607Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8039628, 4519814, 0 committed:false) SnapCreateDur 23.343µs SnapOpenDur 204.815991ms 2015-08-03T20:44:20.498Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4910000 2015-08-03T20:44:20.914Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.13µs CommitTime 266.927031ms TotalFlushTime 4m53.958163725s TotalCommitTime 1m26.186968778s 2015-08-03T20:44:20.914Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8043308, 4521654, 132 committed:true) SnapCreateDur 293.288731ms SnapOpenDur 35.694µs 2015-08-03T20:44:20.985Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4580000 QueuedCount 2998 2015-08-03T20:44:21.4Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8065748, 4532874, 0 committed:false) SnapCreateDur 20.457µs SnapOpenDur 72.6939ms 2015-08-03T20:44:21.421Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4590000 QueuedCount 46 2015-08-03T20:44:21.538Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4920000 2015-08-03T20:44:22.565Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4930000 2015-08-03T20:44:23.609Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4940000 2015-08-03T20:44:23.852Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4600000 QueuedCount 1462 2015-08-03T20:44:24.029Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8088334, 4544167, 0 committed:false) SnapCreateDur 21.647µs SnapOpenDur 126.218345ms 2015-08-03T20:44:24.64Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4950000 2015-08-03T20:44:24.816Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.9µs CommitTime 666.501458ms TotalFlushTime 4m56.86033591s TotalCommitTime 1m26.853470236s 2015-08-03T20:44:24.816Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8091970, 4545985, 133 committed:true) SnapCreateDur 692.701364ms SnapOpenDur 26.685µs 2015-08-03T20:44:24.986Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4610000 QueuedCount 4880 2015-08-03T20:44:25.217Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8114686, 4557343, 0 committed:false) SnapCreateDur 18.207µs SnapOpenDur 78.455178ms 2015-08-03T20:44:25.396Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4620000 QueuedCount 6446 2015-08-03T20:44:25.684Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4960000 2015-08-03T20:44:26.735Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4970000 2015-08-03T20:44:27.674Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8137446, 4568723, 0 committed:false) SnapCreateDur 25.917µs SnapOpenDur 615.734435ms 2015-08-03T20:44:27.778Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4980000 2015-08-03T20:44:28.566Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.237µs CommitTime 759.066365ms TotalFlushTime 4m59.02850849s TotalCommitTime 1m27.612536601s 2015-08-03T20:44:28.566Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8141192, 4570596, 134 committed:true) SnapCreateDur 787.372292ms SnapOpenDur 31.803µs 2015-08-03T20:44:28.642Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4630000 QueuedCount 4079 2015-08-03T20:44:28.814Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4990000 2015-08-03T20:44:29.049Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8163938, 4581969, 0 committed:false) SnapCreateDur 18.942µs SnapOpenDur 70.7011ms 2015-08-03T20:44:29.076Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4640000 QueuedCount 1789 2015-08-03T20:44:29.858Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5000000 2015-08-03T20:44:30.292Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4650000 QueuedCount 1432 2015-08-03T20:44:30.4Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8186650, 4593325, 0 committed:false) SnapCreateDur 19.827µs SnapOpenDur 78.33572ms 2015-08-03T20:44:30.606Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4660000 QueuedCount 2940 2015-08-03T20:44:30.876Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8209276, 4604638, 0 committed:false) SnapCreateDur 27.027µs SnapOpenDur 201.197364ms 2015-08-03T20:44:30.905Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5010000 2015-08-03T20:44:31.925Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.6µs CommitTime 289.654917ms TotalFlushTime 5m1.581216672s TotalCommitTime 1m27.902191518s 2015-08-03T20:44:31.926Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8213056, 4606528, 135 committed:true) SnapCreateDur 315.823862ms SnapOpenDur 51.67µs 2015-08-03T20:44:31.942Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5020000 2015-08-03T20:44:32.126Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4670000 QueuedCount 6402 2015-08-03T20:44:32.426Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8235750, 4617875, 0 committed:false) SnapCreateDur 28.18µs SnapOpenDur 68.747713ms 2015-08-03T20:44:32.582Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4680000 QueuedCount 7873 2015-08-03T20:44:32.988Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5030000 2015-08-03T20:44:33.817Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8258354, 4629177, 0 committed:false) SnapCreateDur 10.599µs SnapOpenDur 79.081848ms 2015-08-03T20:44:33.919Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4690000 QueuedCount 5495 2015-08-03T20:44:34.031Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5040000 2015-08-03T20:44:34.895Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 5.658µs CommitTime 771.231602ms TotalFlushTime 5m3.507795942s TotalCommitTime 1m28.67342312s 2015-08-03T20:44:34.895Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8273320, 4636660, 136 committed:true) SnapCreateDur 784.261848ms SnapOpenDur 30.402µs 2015-08-03T20:44:35.065Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5050000 2015-08-03T20:44:35.086Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4700000 QueuedCount 7104 2015-08-03T20:44:35.409Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8296226, 4648113, 0 committed:false) SnapCreateDur 21.553µs SnapOpenDur 72.263837ms 2015-08-03T20:44:35.533Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4710000 QueuedCount 6746 2015-08-03T20:44:36.112Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5060000 2015-08-03T20:44:36.846Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8318956, 4659478, 0 committed:false) SnapCreateDur 21.017µs SnapOpenDur 124.967804ms 2015-08-03T20:44:36.918Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4720000 QueuedCount 4370 2015-08-03T20:44:37.16Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5070000 2015-08-03T20:44:37.906Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.38µs CommitTime 826.479136ms TotalFlushTime 5m5.380010318s TotalCommitTime 1m29.499902256s 2015-08-03T20:44:37.906Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8330248, 4665124, 137 committed:true) SnapCreateDur 839.442335ms SnapOpenDur 43.368µs 2015-08-03T20:44:38.152Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4730000 QueuedCount 5954 2015-08-03T20:44:38.214Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5080000 2015-08-03T20:44:38.433Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8353036, 4676518, 0 committed:false) SnapCreateDur 24.914µs SnapOpenDur 75.76329ms 2015-08-03T20:44:38.616Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4740000 QueuedCount 7476 2015-08-03T20:44:39.252Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5090000 2015-08-03T20:44:39.906Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8375686, 4687843, 0 committed:false) SnapCreateDur 28.117µs SnapOpenDur 127.312134ms 2015-08-03T20:44:40.287Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5100000 2015-08-03T20:44:40.734Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.987µs CommitTime 706.773973ms TotalFlushTime 5m7.155273496s TotalCommitTime 1m30.206676229s 2015-08-03T20:44:40.734Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8379346, 4689673, 138 committed:true) SnapCreateDur 733.732158ms SnapOpenDur 32.528µs 2015-08-03T20:44:40.769Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4750000 QueuedCount 864 2015-08-03T20:44:41.035Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8399538, 4699769, 0 committed:false) SnapCreateDur 28.175µs SnapOpenDur 59.493809ms 2015-08-03T20:44:41.072Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4760000 QueuedCount 1496 2015-08-03T20:44:41.331Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5110000 2015-08-03T20:44:42.255Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4770000 QueuedCount 1117 2015-08-03T20:44:42.344Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8422256, 4711128, 0 committed:false) SnapCreateDur 17.691µs SnapOpenDur 67.317278ms 2015-08-03T20:44:42.377Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5120000 2015-08-03T20:44:42.67Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4780000 QueuedCount 2555 2015-08-03T20:44:42.942Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8444730, 4722365, 0 committed:false) SnapCreateDur 22.016µs SnapOpenDur 186.887862ms 2015-08-03T20:44:43.414Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5130000 2015-08-03T20:44:44.147Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 13.753µs CommitTime 1.084844403s TotalFlushTime 5m9.013879178s TotalCommitTime 1m31.291520632s 2015-08-03T20:44:44.147Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8448344, 4724172, 139 committed:true) SnapCreateDur 1.110910991s SnapOpenDur 31.759µs 2015-08-03T20:44:44.372Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4790000 QueuedCount 5870 2015-08-03T20:44:44.447Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5140000 2015-08-03T20:44:44.648Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8470874, 4735437, 0 committed:false) SnapCreateDur 23.69µs SnapOpenDur 68.738086ms 2015-08-03T20:44:44.816Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4800000 QueuedCount 7379 2015-08-03T20:44:45.483Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5150000 2015-08-03T20:44:45.991Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8493498, 4746749, 0 committed:false) SnapCreateDur 21.98µs SnapOpenDur 141.209106ms 2015-08-03T20:44:46.532Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5160000 2015-08-03T20:44:46.827Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.047µs CommitTime 745.801677ms TotalFlushTime 5m10.612005918s TotalCommitTime 1m32.037322309s 2015-08-03T20:44:46.827Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8497244, 4748622, 140 committed:true) SnapCreateDur 772.014891ms SnapOpenDur 33.626µs 2015-08-03T20:44:46.863Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4810000 QueuedCount 1197 2015-08-03T20:44:47.065Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4820000 QueuedCount 802 2015-08-03T20:44:47.161Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8519914, 4759957, 0 committed:false) SnapCreateDur 21.37µs SnapOpenDur 72.361156ms 2015-08-03T20:44:47.583Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5170000 2015-08-03T20:44:48.085Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4830000 QueuedCount 2325 2015-08-03T20:44:48.263Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8542546, 4771273, 0 committed:false) SnapCreateDur 25.757µs SnapOpenDur 131.092618ms 2015-08-03T20:44:48.616Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5180000 2015-08-03T20:44:49.369Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4840000 QueuedCount 3802 2015-08-03T20:44:49.662Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5190000 2015-08-03T20:44:49.704Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8565140, 4782570, 0 committed:false) SnapCreateDur 21.7µs SnapOpenDur 201.087138ms 2015-08-03T20:44:50.709Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5200000 2015-08-03T20:44:51.015Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 14.473µs CommitTime 193.832688ms TotalFlushTime 5m14.056218318s TotalCommitTime 1m32.231154997s 2015-08-03T20:44:51.015Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8568948, 4784474, 141 committed:true) SnapCreateDur 207.217571ms SnapOpenDur 25.813µs 2015-08-03T20:44:51.163Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4850000 QueuedCount 7256 2015-08-03T20:44:51.45Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8591618, 4795809, 0 committed:false) SnapCreateDur 19.981µs SnapOpenDur 44.529084ms 2015-08-03T20:44:51.531Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4860000 QueuedCount 2976 2015-08-03T20:44:51.74Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5210000 2015-08-03T20:44:52.671Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8614246, 4807123, 0 committed:false) SnapCreateDur 23.086µs SnapOpenDur 139.936495ms 2015-08-03T20:44:52.726Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4870000 QueuedCount 2549 2015-08-03T20:44:52.79Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5220000 2015-08-03T20:44:53.827Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5230000 2015-08-03T20:44:53.861Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 3.596µs CommitTime 1.001623636s TotalFlushTime 5m15.581402393s TotalCommitTime 1m33.232778633s 2015-08-03T20:44:53.861Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8625488, 4812744, 142 committed:true) SnapCreateDur 1.028520127s SnapOpenDur 25.993µs 2015-08-03T20:44:54.103Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4880000 QueuedCount 5988 2015-08-03T20:44:54.403Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8648220, 4824110, 0 committed:false) SnapCreateDur 21.95µs SnapOpenDur 69.033347ms 2015-08-03T20:44:54.629Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4890000 QueuedCount 7542 2015-08-03T20:44:54.865Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5240000 2015-08-03T20:44:55.907Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5250000 2015-08-03T20:44:56.956Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5260000 2015-08-03T20:44:56.999Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 2015-08-03T20:44:57.204Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8670960, 4835480, 0 committed:false) SnapCreateDur 37.093µs SnapOpenDur 33.227322ms 2015-08-03T20:44:57.998Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5270000 2015-08-03T20:44:59.017Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.783µs CommitTime 392.940559ms TotalFlushTime 5m18.808066651s TotalCommitTime 1m33.625719192s 2015-08-03T20:44:59.017Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8674704, 4837352, 143 committed:true) SnapCreateDur 1.718325651s SnapOpenDur 30.982µs 2015-08-03T20:44:59.041Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5280000 2015-08-03T20:44:59.057Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4900000 QueuedCount 1382 2015-08-03T20:44:59.399Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4910000 QueuedCount 1073 2015-08-03T20:44:59.514Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8697630, 4848815, 0 committed:false) SnapCreateDur 25.35µs SnapOpenDur 75.089645ms 2015-08-03T20:45:00.085Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5290000 2015-08-03T20:45:00.762Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4920000 QueuedCount 2603 2015-08-03T20:45:00.959Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8720312, 4860156, 0 committed:false) SnapCreateDur 22.726µs SnapOpenDur 140.646916ms 2015-08-03T20:45:01.114Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5300000 2015-08-03T20:45:01.839Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 15.61µs CommitTime 758.169044ms TotalFlushTime 5m20.525760097s TotalCommitTime 1m34.383888236s 2015-08-03T20:45:01.839Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8724094, 4862047, 144 committed:true) SnapCreateDur 785.8682ms SnapOpenDur 40.409µs 2015-08-03T20:45:02.056Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4930000 QueuedCount 6123 2015-08-03T20:45:02.159Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5310000 2015-08-03T20:45:02.36Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8746914, 4873457, 0 committed:false) SnapCreateDur 38.217µs SnapOpenDur 79.754427ms 2015-08-03T20:45:02.53Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4940000 QueuedCount 7666 2015-08-03T20:45:03.191Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5320000 2015-08-03T20:45:03.892Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8769596, 4884798, 0 committed:false) SnapCreateDur 34.17µs SnapOpenDur 158.411407ms 2015-08-03T20:45:03.971Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4950000 QueuedCount 3444 2015-08-03T20:45:04.247Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5330000 2015-08-03T20:45:05.075Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 3.57µs CommitTime 970.499799ms TotalFlushTime 5m22.419730855s TotalCommitTime 1m35.354388035s 2015-08-03T20:45:05.075Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8784792, 4892396, 145 committed:true) SnapCreateDur 996.471811ms SnapOpenDur 29.555µs 2015-08-03T20:45:05.215Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":11978018291,"bucket-1:by_email:avg_scan_wait_latency":11977093766,"bucket-1:by_email:avg_ts_interval":1175740484,"bucket-1:by_email:avg_ts_items_count":8956,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":425176259,"bucket-1:by_email:delete_bytes":167373028,"bucket-1:by_email:disk_size":1722179584,"bucket-1:by_email:flush_queue_size":7381,"bucket-1:by_email:frag_percent":75,"bucket-1:by_email:get_bytes":171222990,"bucket-1:by_email:insert_bytes":513701580,"bucket-1:by_email:items_count":1000000,"bucket-1:by_email:num_commits":145,"bucket-1:by_email:num_compactions":7,"bucket-1:by_email:num_completed_requests":13059,"bucket-1:by_email:num_docs_indexed":4955948,"bucket-1:by_email:num_docs_pending":234,"bucket-1:by_email:num_docs_queued":380393,"bucket-1:by_email:num_flush_queued":4966040,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":13443,"bucket-1:by_email:num_rows_returned":213,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":620,"bucket-1:by_email:scan_bytes_read":8520,"bucket-1:by_email:scan_wait_duration":161008071504062,"bucket-1:by_email:since_last_snapshot":1182699340,"bucket-1:by_email:timings/storage_clone_handle":"13059 773379491 2736506580269375","bucket-1:by_email:timings/storage_commit":"145 95354388035 974401536027953831","bucket-1:by_email:timings/storage_del":"3895599 59394806723 -3733929726497047411","bucket-1:by_email:timings/storage_get":"4958654 66778774866 -6807301342554992634","bucket-1:by_email:timings/storage_handle_open":"145 8487180491 -6164505866624149621","bucket-1:by_email:timings/storage_iterator_next":"213 3809094 86398763126","bucket-1:by_email:timings/storage_new_iterator":"13059 1108922539 2836597894387989","bucket-1:by_email:timings/storage_set":"9791196 189997725423 1023208650956151305","bucket-1:by_email:timings/storage_snapshot_close":"475 7488698456 763679960952996834","bucket-1:by_email:timings/storage_snapshot_create":"475 43657286224 7285873664866366424","bucket-1:by_email:total_scan_duration":161020499893236,"bucket-1:mutation_queue_size":372538,"bucket-1:num_mutations_queued":5338610,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":193,"memory_quota":104857600000,"memory_used":557879296,"needs_restart":false,"num_connections":384,"timings/stats_response":"998 349232046 4719937979655350"} 2015-08-03T20:45:05.262Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4960000 QueuedCount 6902 2015-08-03T20:45:05.356Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5340000 2015-08-03T20:45:05.584Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8807426, 4903713, 0 committed:false) SnapCreateDur 29.423µs SnapOpenDur 77.663625ms 2015-08-03T20:45:05.721Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4970000 QueuedCount 6412 2015-08-03T20:45:06.391Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5350000 2015-08-03T20:45:07.302Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8829942, 4914971, 0 committed:false) SnapCreateDur 26.72µs SnapOpenDur 145.830558ms 2015-08-03T20:45:07.385Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4980000 QueuedCount 268 2015-08-03T20:45:07.447Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5360000 2015-08-03T20:45:08.224Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.923µs CommitTime 801.371769ms TotalFlushTime 5m24.40512882s TotalCommitTime 1m36.155759804s 2015-08-03T20:45:08.224Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8833748, 4916874, 146 committed:true) SnapCreateDur 828.086536ms SnapOpenDur 40.213µs 2015-08-03T20:45:08.457Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 4990000 QueuedCount 1755 2015-08-03T20:45:08.49Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5370000 2015-08-03T20:45:08.542Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8856332, 4928166, 0 committed:false) SnapCreateDur 15.254µs SnapOpenDur 45.833702ms 2015-08-03T20:45:09.404Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5000000 QueuedCount 3289 2015-08-03T20:45:09.536Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5380000 2015-08-03T20:45:09.604Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8879008, 4939504, 0 committed:false) SnapCreateDur 21.65µs SnapOpenDur 140.057791ms 2015-08-03T20:45:09.889Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5010000 QueuedCount 4780 2015-08-03T20:45:10.284Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8901620, 4950810, 0 committed:false) SnapCreateDur 25.067µs SnapOpenDur 225.218599ms 2015-08-03T20:45:10.577Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5390000 2015-08-03T20:45:11.612Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5400000 2015-08-03T20:45:12.189Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 18.397µs CommitTime 519.456967ms TotalFlushTime 5m27.284004997s TotalCommitTime 1m36.675216771s 2015-08-03T20:45:12.189Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8909186, 4954593, 147 committed:true) SnapCreateDur 546.038873ms SnapOpenDur 57.943µs 2015-08-03T20:45:12.263Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5020000 QueuedCount 4318 2015-08-03T20:45:12.617Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5030000 QueuedCount 157 2015-08-03T20:45:12.657Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5410000 2015-08-03T20:45:12.707Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8931880, 4965940, 0 committed:false) SnapCreateDur 26.55µs SnapOpenDur 82.762789ms 2015-08-03T20:45:13.697Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5420000 2015-08-03T20:45:13.998Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5040000 QueuedCount 1676 2015-08-03T20:45:14.167Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8954520, 4977260, 0 committed:false) SnapCreateDur 22.191µs SnapOpenDur 135.013907ms 2015-08-03T20:45:14.739Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5430000 2015-08-03T20:45:15.074Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.88µs CommitTime 775.667817ms TotalFlushTime 5m29.043497019s TotalCommitTime 1m37.450884588s 2015-08-03T20:45:15.074Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8958308, 4979154, 148 committed:true) SnapCreateDur 802.399689ms SnapOpenDur 54.609µs 2015-08-03T20:45:15.324Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5050000 QueuedCount 5113 2015-08-03T20:45:15.576Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 8980956, 4990478, 0 committed:false) SnapCreateDur 22.877µs SnapOpenDur 67.964851ms 2015-08-03T20:45:15.777Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5440000 2015-08-03T20:45:15.784Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5060000 QueuedCount 6547 2015-08-03T20:45:16.831Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5450000 2015-08-03T20:45:16.908Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9003430, 5001715, 0 committed:false) SnapCreateDur 22.911µs SnapOpenDur 78.906758ms 2015-08-03T20:45:16.998Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5070000 QueuedCount 2303 2015-08-03T20:45:17.513Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9025756, 5012878, 0 committed:false) SnapCreateDur 40.72µs SnapOpenDur 220.294499ms 2015-08-03T20:45:17.884Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5460000 2015-08-03T20:45:18.916Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5470000 2015-08-03T20:45:18.991Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 5.727µs CommitTime 350.127286ms TotalFlushTime 5m32.059105789s TotalCommitTime 1m37.801011874s 2015-08-03T20:45:18.992Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9029680, 5014840, 149 committed:true) SnapCreateDur 378.140344ms SnapOpenDur 44.083µs 2015-08-03T20:45:19.017Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5080000 QueuedCount 1813 2015-08-03T20:45:19.366Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5090000 QueuedCount 1462 2015-08-03T20:45:19.492Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9052408, 5026204, 0 committed:false) SnapCreateDur 36.414µs SnapOpenDur 76.160879ms 2015-08-03T20:45:19.961Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5480000 2015-08-03T20:45:20.755Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5100000 QueuedCount 2969 2015-08-03T20:45:20.887Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9075052, 5037526, 0 committed:false) SnapCreateDur 14.712µs SnapOpenDur 79.293791ms 2015-08-03T20:45:20.995Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5490000 2015-08-03T20:45:21.769Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.75µs CommitTime 791.293834ms TotalFlushTime 5m33.768013821s TotalCommitTime 1m38.592305708s 2015-08-03T20:45:21.769Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9078904, 5039452, 150 committed:true) SnapCreateDur 817.781448ms SnapOpenDur 34.055µs 2015-08-03T20:45:21.906Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5110000 QueuedCount 4579 2015-08-03T20:45:22.036Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5500000 2015-08-03T20:45:22.089Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9101622, 5050811, 0 committed:false) SnapCreateDur 13.277µs SnapOpenDur 46.022683ms 2015-08-03T20:45:22.236Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5120000 QueuedCount 8008 2015-08-03T20:45:23.077Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5510000 2015-08-03T20:45:23.542Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9124346, 5062173, 0 committed:false) SnapCreateDur 35.126µs SnapOpenDur 150.520804ms 2015-08-03T20:45:23.647Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5130000 QueuedCount 5725 2015-08-03T20:45:24.114Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5520000 2015-08-03T20:45:24.237Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9147064, 5073532, 0 committed:false) SnapCreateDur 37.89µs SnapOpenDur 238.78423ms 2015-08-03T20:45:25.152Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5530000 2015-08-03T20:45:25.353Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5140000 QueuedCount 1492 2015-08-03T20:45:25.826Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.29µs CommitTime 415.338948ms TotalFlushTime 5m36.808834738s TotalCommitTime 1m39.007644656s 2015-08-03T20:45:25.826Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9150882, 5075441, 151 committed:true) SnapCreateDur 443.413014ms SnapOpenDur 57.177µs 2015-08-03T20:45:26.191Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5540000 2015-08-03T20:45:26.25Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5150000 QueuedCount 3042 2015-08-03T20:45:26.468Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9173652, 5086826, 0 committed:false) SnapCreateDur 23.08µs SnapOpenDur 87.897632ms 2015-08-03T20:45:26.999Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:459172940, Disk:2225041408, ExtraSnapshotData:0, Fragmentation:79.3633979867039% 2015-08-03T20:45:26.999Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:45:27.021Z-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 8909186. Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T20:45:27.24Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5550000 2015-08-03T20:45:27.968Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5160000 QueuedCount 4500 2015-08-03T20:45:28.088Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9196216, 5098108, 0 committed:false) SnapCreateDur 14.925µs SnapOpenDur 20.304987ms 2015-08-03T20:45:28.286Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5560000 2015-08-03T20:45:28.949Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.51µs CommitTime 739.695758ms TotalFlushTime 5m38.952157015s TotalCommitTime 1m39.747340414s 2015-08-03T20:45:28.949Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9199898, 5099949, 152 committed:true) SnapCreateDur 767.050428ms SnapOpenDur 37.157µs 2015-08-03T20:45:29.049Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5170000 QueuedCount 4057 2015-08-03T20:45:29.272Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9222590, 5111295, 0 committed:false) SnapCreateDur 25.236µs SnapOpenDur 59.907463ms 2015-08-03T20:45:29.328Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5570000 2015-08-03T20:45:29.361Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5180000 QueuedCount 3715 2015-08-03T20:45:30.366Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5580000 2015-08-03T20:45:30.582Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9245278, 5122639, 0 committed:false) SnapCreateDur 19.886µs SnapOpenDur 80.948829ms 2015-08-03T20:45:30.617Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5190000 QueuedCount 1398 2015-08-03T20:45:31.414Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5590000 2015-08-03T20:45:31.63Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5200000 QueuedCount 995 2015-08-03T20:45:31.878Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9268028, 5134014, 0 committed:false) SnapCreateDur 26.257µs SnapOpenDur 203.509714ms 2015-08-03T20:45:32.456Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5600000 2015-08-03T20:45:33.339Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.417µs CommitTime 341.260387ms TotalFlushTime 5m42.488384842s TotalCommitTime 1m40.088600801s 2015-08-03T20:45:33.339Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9271742, 5135871, 153 committed:true) SnapCreateDur 367.357623ms SnapOpenDur 44.727µs 2015-08-03T20:45:33.49Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5610000 2015-08-03T20:45:33.618Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5210000 QueuedCount 4432 2015-08-03T20:45:33.863Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9294462, 5147231, 0 committed:false) SnapCreateDur 31.034µs SnapOpenDur 77.242175ms 2015-08-03T20:45:34.088Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5220000 QueuedCount 5936 2015-08-03T20:45:34.532Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5620000 2015-08-03T20:45:35.335Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9317114, 5158557, 0 committed:false) SnapCreateDur 23.29µs SnapOpenDur 129.575305ms 2015-08-03T20:45:35.575Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5630000 2015-08-03T20:45:36.197Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.57µs CommitTime 770.307674ms TotalFlushTime 5m44.237345013s TotalCommitTime 1m40.858908475s 2015-08-03T20:45:36.197Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9320796, 5160398, 154 committed:true) SnapCreateDur 797.666373ms SnapOpenDur 50.383µs 2015-08-03T20:45:36.289Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5230000 QueuedCount 4142 2015-08-03T20:45:36.623Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5640000 2015-08-03T20:45:36.686Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9343696, 5171848, 0 committed:false) SnapCreateDur 28.117µs SnapOpenDur 67.633895ms 2015-08-03T20:45:36.729Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5240000 QueuedCount 1357 2015-08-03T20:45:37.683Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5650000 2015-08-03T20:45:38.717Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5660000 2015-08-03T20:45:38.979Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5250000 QueuedCount 928 2015-08-03T20:45:39.156Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9366292, 5183146, 0 committed:false) SnapCreateDur 34.94µs SnapOpenDur 138.319278ms 2015-08-03T20:45:39.761Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5670000 2015-08-03T20:45:40.13Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.03µs CommitTime 849.762417ms TotalFlushTime 5m46.973264845s TotalCommitTime 1m41.708670892s 2015-08-03T20:45:40.13Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9370052, 5185026, 155 committed:true) SnapCreateDur 879.483444ms SnapOpenDur 50.6µs 2015-08-03T20:45:40.402Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5260000 QueuedCount 4271 2015-08-03T20:45:40.627Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9392534, 5196267, 0 committed:false) SnapCreateDur 27.457µs SnapOpenDur 74.20155ms 2015-08-03T20:45:40.803Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5680000 2015-08-03T20:45:40.863Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5270000 QueuedCount 5792 2015-08-03T20:45:41.841Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5690000 2015-08-03T20:45:42.028Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9415234, 5207617, 0 committed:false) SnapCreateDur 21.893µs SnapOpenDur 129.783698ms 2015-08-03T20:45:42.139Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5280000 QueuedCount 3484 2015-08-03T20:45:42.436Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9437892, 5218946, 0 committed:false) SnapCreateDur 13.651µs SnapOpenDur 135.933109ms 2015-08-03T20:45:42.885Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5700000 2015-08-03T20:45:43.922Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5710000 2015-08-03T20:45:44.074Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 13.893µs CommitTime 377.25258ms TotalFlushTime 5m50.043866262s TotalCommitTime 1m42.085923472s 2015-08-03T20:45:44.074Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9441776, 5220888, 156 committed:true) SnapCreateDur 404.917159ms SnapOpenDur 48.14µs 2015-08-03T20:45:44.125Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5290000 QueuedCount 3093 2015-08-03T20:45:44.428Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5300000 QueuedCount 809 2015-08-03T20:45:44.535Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9464442, 5232221, 0 committed:false) SnapCreateDur 24.596µs SnapOpenDur 78.361911ms 2015-08-03T20:45:44.965Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5720000 2015-08-03T20:45:45.649Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5310000 QueuedCount 2388 2015-08-03T20:45:45.777Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9487236, 5243618, 0 committed:false) SnapCreateDur 18.067µs SnapOpenDur 80.858786ms 2015-08-03T20:45:46.007Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5730000 2015-08-03T20:45:46.732Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.4µs CommitTime 861.990696ms TotalFlushTime 5m51.556207065s TotalCommitTime 1m42.947914168s 2015-08-03T20:45:46.732Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9491026, 5245513, 157 committed:true) SnapCreateDur 890.845428ms SnapOpenDur 41.133µs 2015-08-03T20:45:46.95Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5320000 QueuedCount 5642 2015-08-03T20:45:47.053Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5740000 2015-08-03T20:45:47.2Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9513356, 5256678, 0 committed:false) SnapCreateDur 14.72µs SnapOpenDur 46.737836ms 2015-08-03T20:45:47.38Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5330000 QueuedCount 6677 2015-08-03T20:45:48.109Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5750000 2015-08-03T20:45:48.523Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9535096, 5267548, 0 committed:false) SnapCreateDur 21.21µs SnapOpenDur 123.10513ms 2015-08-03T20:45:48.673Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5340000 QueuedCount 8068 2015-08-03T20:45:49.137Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5760000 2015-08-03T20:45:49.868Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9557528, 5278764, 0 committed:false) SnapCreateDur 22.493µs SnapOpenDur 194.615565ms 2015-08-03T20:45:50.187Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5770000 2015-08-03T20:45:50.99Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5350000 QueuedCount 109 2015-08-03T20:45:51.229Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5780000 2015-08-03T20:45:51.365Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 3.58µs CommitTime 336.209118ms TotalFlushTime 5m55.317935297s TotalCommitTime 1m43.284123286s 2015-08-03T20:45:51.365Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9561556, 5280778, 158 committed:true) SnapCreateDur 362.780183ms SnapOpenDur 49.6µs 2015-08-03T20:45:51.72Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5360000 QueuedCount 1453 2015-08-03T20:45:51.854Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9583884, 5291942, 0 committed:false) SnapCreateDur 23.216µs SnapOpenDur 75.528306ms 2015-08-03T20:45:52.264Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5790000 2015-08-03T20:45:53.132Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5370000 QueuedCount 2966 2015-08-03T20:45:53.275Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9606568, 5303284, 0 committed:false) SnapCreateDur 23.197µs SnapOpenDur 79.38062ms 2015-08-03T20:45:53.309Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5800000 2015-08-03T20:45:54.139Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.923µs CommitTime 773.780616ms TotalFlushTime 5m57.03420851s TotalCommitTime 1m44.057903902s 2015-08-03T20:45:54.139Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9610376, 5305188, 159 committed:true) SnapCreateDur 799.901791ms SnapOpenDur 45.403µs 2015-08-03T20:45:54.339Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5380000 QueuedCount 6474 2015-08-03T20:45:54.353Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5810000 2015-08-03T20:45:54.647Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9633154, 5316577, 0 committed:false) SnapCreateDur 27.637µs SnapOpenDur 85.927482ms 2015-08-03T20:45:54.794Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5390000 QueuedCount 7906 2015-08-03T20:45:55.39Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5820000 2015-08-03T20:45:55.972Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9655610, 5327805, 0 committed:false) SnapCreateDur 33.423µs SnapOpenDur 142.487485ms 2015-08-03T20:45:56.071Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5400000 QueuedCount 5703 2015-08-03T20:45:56.441Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5830000 2015-08-03T20:45:56.64Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9678380, 5339190, 0 committed:false) SnapCreateDur 21.604µs SnapOpenDur 214.737752ms 2015-08-03T20:45:57.51Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5840000 2015-08-03T20:45:57.619Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5410000 QueuedCount 1429 2015-08-03T20:45:58.081Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.487µs CommitTime 399.802112ms TotalFlushTime 5m59.966558089s TotalCommitTime 1m44.457706014s 2015-08-03T20:45:58.081Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9682216, 5341108, 160 committed:true) SnapCreateDur 425.991963ms SnapOpenDur 47.103µs 2015-08-03T20:45:58.319Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5420000 QueuedCount 2974 2015-08-03T20:45:58.507Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9704922, 5352461, 0 committed:false) SnapCreateDur 23.11µs SnapOpenDur 81.177543ms 2015-08-03T20:45:58.539Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5850000 2015-08-03T20:45:59.585Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5860000 2015-08-03T20:45:59.686Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5430000 QueuedCount 4509 2015-08-03T20:45:59.907Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9727600, 5363800, 0 committed:false) SnapCreateDur 24.1µs SnapOpenDur 135.588332ms 2015-08-03T20:46:00.616Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5870000 2015-08-03T20:46:00.85Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 13.98µs CommitTime 763.365708ms TotalFlushTime 6m1.622695604s TotalCommitTime 1m45.221071722s 2015-08-03T20:46:00.85Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9731186, 5365593, 161 committed:true) SnapCreateDur 798.123475ms SnapOpenDur 34.521µs 2015-08-03T20:46:00.948Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5440000 QueuedCount 4058 2015-08-03T20:46:01.182Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9753868, 5376934, 0 committed:false) SnapCreateDur 11.162µs SnapOpenDur 50.264054ms 2015-08-03T20:46:01.25Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5450000 QueuedCount 1609 2015-08-03T20:46:01.657Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5880000 2015-08-03T20:46:02.184Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9776336, 5388168, 0 committed:false) SnapCreateDur 15.034µs SnapOpenDur 80.285858ms 2015-08-03T20:46:02.225Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5460000 QueuedCount 1270 2015-08-03T20:46:02.403Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5470000 QueuedCount 820 2015-08-03T20:46:02.539Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9799076, 5399538, 0 committed:false) SnapCreateDur 10.471µs SnapOpenDur 118.093395ms 2015-08-03T20:46:02.698Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5890000 2015-08-03T20:46:03.32Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5480000 QueuedCount 2362 2015-08-03T20:46:03.463Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9821756, 5410878, 0 committed:false) SnapCreateDur 11.723µs SnapOpenDur 102.966642ms 2015-08-03T20:46:03.743Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5900000 2015-08-03T20:46:04.146Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.155µs CommitTime 607.222619ms TotalFlushTime 6m3.787297644s TotalCommitTime 1m45.828294341s 2015-08-03T20:46:04.817Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5910000 2015-08-03T20:46:05.215Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":13611603141,"bucket-1:by_email:avg_scan_wait_latency":13610665525,"bucket-1:by_email:avg_ts_interval":760280122,"bucket-1:by_email:avg_ts_items_count":11181,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":508984387,"bucket-1:by_email:delete_bytes":187720499,"bucket-1:by_email:disk_size":2935533568,"bucket-1:by_email:flush_queue_size":0,"bucket-1:by_email:frag_percent":82,"bucket-1:by_email:get_bytes":191940519,"bucket-1:by_email:insert_bytes":563387265,"bucket-1:by_email:items_count":1000001,"bucket-1:by_email:num_commits":162,"bucket-1:by_email:num_compactions":8,"bucket-1:by_email:num_completed_requests":13827,"bucket-1:by_email:num_docs_indexed":5437110,"bucket-1:by_email:num_docs_pending":406,"bucket-1:by_email:num_docs_queued":431522,"bucket-1:by_email:num_flush_queued":5484292,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":14211,"bucket-1:by_email:num_rows_returned":226,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":680,"bucket-1:by_email:scan_bytes_read":9040,"bucket-1:by_email:scan_wait_duration":193421167782229,"bucket-1:by_email:since_last_snapshot":924184253,"bucket-1:by_email:timings/storage_clone_handle":"13827 810662379 2739176705233485","bucket-1:by_email:timings/storage_commit":"162 105828294341 8096823108800095489","bucket-1:by_email:timings/storage_del":"4412775 68445370651 1670666327785505843","bucket-1:by_email:timings/storage_get":"5484292 74365003784 -6148887066534595988","bucket-1:by_email:timings/storage_handle_open":"161 8487848654 -6164505838001784044","bucket-1:by_email:timings/storage_iterator_next":"226 4414920 121935040876","bucket-1:by_email:timings/storage_new_iterator":"13827 1170046298 2844882623532142","bucket-1:by_email:timings/storage_set":"10825550 213928891088 802968560688702472","bucket-1:by_email:timings/storage_snapshot_close":"517 8275177724 782413544595922512","bucket-1:by_email:timings/storage_snapshot_create":"518 48362735645 7923065992283918121","bucket-1:by_email:total_scan_duration":193434492248502,"bucket-1:mutation_queue_size":429311,"bucket-1:num_mutations_queued":5913603,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":223,"memory_quota":104857600000,"memory_used":1880686592,"needs_restart":false,"num_connections":384,"timings/stats_response":"1069 374637848 4729466180487846"} 2015-08-03T20:46:05.28Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9825550, 5412775, 162 committed:true) SnapCreateDur 618.430306ms SnapOpenDur 1.134114398s 2015-08-03T20:46:05.501Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5490000 QueuedCount 5778 2015-08-03T20:46:05.877Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5920000 2015-08-03T20:46:06.587Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9848176, 5424088, 0 committed:false) SnapCreateDur 25.87µs SnapOpenDur 75.801592ms 2015-08-03T20:46:06.925Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5930000 2015-08-03T20:46:07.651Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.5µs CommitTime 933.424759ms TotalFlushTime 6m5.049283782s TotalCommitTime 1m46.7617191s 2015-08-03T20:46:07.651Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9852180, 5426090, 163 committed:true) SnapCreateDur 960.245904ms SnapOpenDur 29.002µs 2015-08-03T20:46:07.749Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5500000 QueuedCount 5449 2015-08-03T20:46:07.965Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5940000 2015-08-03T20:46:08.14Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9874788, 5437394, 0 committed:false) SnapCreateDur 26.633µs SnapOpenDur 75.034968ms 2015-08-03T20:46:08.19Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5510000 QueuedCount 3191 2015-08-03T20:46:09.012Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5950000 2015-08-03T20:46:10.052Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5960000 2015-08-03T20:46:10.108Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 2015-08-03T20:46:10.277Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5520000 QueuedCount 826 2015-08-03T20:46:10.353Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9897458, 5448729, 0 committed:false) SnapCreateDur 22.93µs SnapOpenDur 40.762047ms 2015-08-03T20:46:11.091Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5970000 2015-08-03T20:46:12.136Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5980000 2015-08-03T20:46:12.14Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 13.004µs CommitTime 437.080885ms TotalFlushTime 6m7.62459976s TotalCommitTime 1m47.198799985s 2015-08-03T20:46:12.14Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9901282, 5450641, 164 committed:true) SnapCreateDur 1.692878884s SnapOpenDur 28.404µs 2015-08-03T20:46:12.287Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5530000 QueuedCount 4307 2015-08-03T20:46:12.44Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9924040, 5462020, 0 committed:false) SnapCreateDur 35.341µs SnapOpenDur 77.647456ms 2015-08-03T20:46:13.181Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5990000 2015-08-03T20:46:13.27Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5540000 QueuedCount 5771 2015-08-03T20:46:13.489Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9946630, 5473315, 0 committed:false) SnapCreateDur 22.314µs SnapOpenDur 123.285235ms 2015-08-03T20:46:14.215Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6000000 2015-08-03T20:46:14.268Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 2.624µs CommitTime 672.805887ms TotalFlushTime 6m8.772631624s TotalCommitTime 1m47.871605872s 2015-08-03T20:46:14.268Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9950430, 5475215, 165 committed:true) SnapCreateDur 684.818461ms SnapOpenDur 31.602µs 2015-08-03T20:46:14.338Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5550000 QueuedCount 2576 2015-08-03T20:46:14.61Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9973024, 5486512, 0 committed:false) SnapCreateDur 44.733µs SnapOpenDur 80.144931ms 2015-08-03T20:46:14.653Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5560000 QueuedCount 1100 2015-08-03T20:46:15.261Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6010000 2015-08-03T20:46:15.623Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5570000 QueuedCount 660 2015-08-03T20:46:15.787Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 9995632, 5497816, 0 committed:false) SnapCreateDur 20.437µs SnapOpenDur 143.774122ms 2015-08-03T20:46:15.999Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5580000 QueuedCount 2224 2015-08-03T20:46:16.246Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10018330, 5509165, 0 committed:false) SnapCreateDur 38.927µs SnapOpenDur 204.354816ms 2015-08-03T20:46:16.311Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6020000 2015-08-03T20:46:17.358Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6030000 2015-08-03T20:46:17.366Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5590000 QueuedCount 3756 2015-08-03T20:46:17.629Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10040990, 5520495, 0 committed:false) SnapCreateDur 22.547µs SnapOpenDur 188.91075ms 2015-08-03T20:46:18.39Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6040000 2015-08-03T20:46:18.681Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.53µs CommitTime 950.806672ms TotalFlushTime 6m11.429203574s TotalCommitTime 1m48.822412544s 2015-08-03T20:46:18.681Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10044744, 5522372, 166 committed:true) SnapCreateDur 977.392179ms SnapOpenDur 29.518µs 2015-08-03T20:46:18.863Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5600000 QueuedCount 7199 2015-08-03T20:46:19.206Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10067500, 5533750, 0 committed:false) SnapCreateDur 26.72µs SnapOpenDur 81.118893ms 2015-08-03T20:46:19.338Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5610000 QueuedCount 6805 2015-08-03T20:46:19.436Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6050000 2015-08-03T20:46:20.474Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6060000 2015-08-03T20:46:20.697Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10090152, 5545076, 0 committed:false) SnapCreateDur 26.303µs SnapOpenDur 145.315484ms 2015-08-03T20:46:20.768Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5620000 QueuedCount 645 2015-08-03T20:46:21.503Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 13.525µs CommitTime 694.763508ms TotalFlushTime 6m13.206285808s TotalCommitTime 1m49.517176052s 2015-08-03T20:46:21.503Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10093918, 5546959, 167 committed:true) SnapCreateDur 711.146145ms SnapOpenDur 68.96µs 2015-08-03T20:46:21.512Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6070000 2015-08-03T20:46:21.867Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5630000 QueuedCount 2122 2015-08-03T20:46:22.03Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10116518, 5558259, 0 committed:false) SnapCreateDur 25.47µs SnapOpenDur 80.936745ms 2015-08-03T20:46:22.556Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6080000 2015-08-03T20:46:23.246Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5640000 QueuedCount 3544 2015-08-03T20:46:23.394Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10139054, 5569527, 0 committed:false) SnapCreateDur 13.277µs SnapOpenDur 78.655194ms 2015-08-03T20:46:23.59Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6090000 2015-08-03T20:46:23.687Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5650000 QueuedCount 5015 2015-08-03T20:46:24.109Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10161640, 5580820, 0 committed:false) SnapCreateDur 27.829µs SnapOpenDur 202.932269ms 2015-08-03T20:46:24.636Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6100000 2015-08-03T20:46:25.555Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.776µs CommitTime 376.06258ms TotalFlushTime 6m16.347621719s TotalCommitTime 1m49.893238632s 2015-08-03T20:46:25.555Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10165434, 5582717, 168 committed:true) SnapCreateDur 401.969455ms SnapOpenDur 60.323µs 2015-08-03T20:46:25.645Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5660000 QueuedCount 2705 2015-08-03T20:46:25.662Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6110000 2015-08-03T20:46:25.976Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10188120, 5594060, 0 committed:false) SnapCreateDur 25.15µs SnapOpenDur 76.876039ms 2015-08-03T20:46:26.057Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5670000 QueuedCount 4177 2015-08-03T20:46:26.712Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6120000 2015-08-03T20:46:27.28Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5680000 QueuedCount 2 2015-08-03T20:46:27.428Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10210768, 5605384, 0 committed:false) SnapCreateDur 33.473µs SnapOpenDur 137.961237ms 2015-08-03T20:46:27.76Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6130000 2015-08-03T20:46:28.319Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 15.748µs CommitTime 770.137728ms TotalFlushTime 6m17.992551893s TotalCommitTime 1m50.66337636s 2015-08-03T20:46:28.32Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10214412, 5607206, 169 committed:true) SnapCreateDur 796.669601ms SnapOpenDur 39.192µs 2015-08-03T20:46:28.517Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5690000 QueuedCount 3460 2015-08-03T20:46:28.641Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10237270, 5618635, 0 committed:false) SnapCreateDur 19.678µs SnapOpenDur 47.962226ms 2015-08-03T20:46:28.808Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6140000 2015-08-03T20:46:29.468Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5700000 QueuedCount 4987 2015-08-03T20:46:29.649Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10259978, 5629989, 0 committed:false) SnapCreateDur 12.386µs SnapOpenDur 84.06519ms 2015-08-03T20:46:29.769Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5710000 QueuedCount 4598 2015-08-03T20:46:29.839Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6150000 2015-08-03T20:46:30.566Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10282654, 5641327, 0 committed:false) SnapCreateDur 13.464µs SnapOpenDur 63.101151ms 2015-08-03T20:46:30.884Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6160000 2015-08-03T20:46:31.131Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.01µs CommitTime 443.62996ms TotalFlushTime 6m20.005906434s TotalCommitTime 1m51.10700632s 2015-08-03T20:46:31.131Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10286448, 5643224, 170 committed:true) SnapCreateDur 470.726005ms SnapOpenDur 44.52µs 2015-08-03T20:46:31.208Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5720000 QueuedCount 4249 2015-08-03T20:46:31.755Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10309206, 5654603, 0 committed:false) SnapCreateDur 31.825µs SnapOpenDur 94.56033ms 2015-08-03T20:46:31.777Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5730000 QueuedCount 371 2015-08-03T20:46:31.915Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6170000 2015-08-03T20:46:32.962Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6180000 2015-08-03T20:46:33.169Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5740000 QueuedCount 1413 2015-08-03T20:46:33.346Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10331766, 5665883, 0 committed:false) SnapCreateDur 23.913µs SnapOpenDur 145.730918ms 2015-08-03T20:46:33.548Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5750000 QueuedCount 2945 2015-08-03T20:46:34.002Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6190000 2015-08-03T20:46:34.357Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10354478, 5677239, 0 committed:false) SnapCreateDur 21.463µs SnapOpenDur 67.156248ms 2015-08-03T20:46:34.908Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.81µs CommitTime 430.540954ms TotalFlushTime 6m22.884125477s TotalCommitTime 1m51.537547274s 2015-08-03T20:46:34.908Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10358142, 5679071, 171 committed:true) SnapCreateDur 456.933703ms SnapOpenDur 62.17µs 2015-08-03T20:46:35.039Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6200000 2015-08-03T20:46:35.122Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5760000 QueuedCount 6352 2015-08-03T20:46:35.429Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10380890, 5690445, 0 committed:false) SnapCreateDur 34.536µs SnapOpenDur 76.562888ms 2015-08-03T20:46:35.602Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5770000 QueuedCount 7843 2015-08-03T20:46:36.082Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6210000 2015-08-03T20:46:36.927Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10403508, 5701754, 0 committed:false) SnapCreateDur 20.884µs SnapOpenDur 141.688215ms 2015-08-03T20:46:37.037Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5780000 QueuedCount 5543 2015-08-03T20:46:37.138Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6220000 2015-08-03T20:46:37.621Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10426252, 5713126, 0 committed:false) SnapCreateDur 26.43µs SnapOpenDur 231.355971ms 2015-08-03T20:46:38.186Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6230000 2015-08-03T20:46:38.698Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5790000 QueuedCount 1311 2015-08-03T20:46:39.189Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.544µs CommitTime 426.422227ms TotalFlushTime 6m26.110706843s TotalCommitTime 1m51.963969501s 2015-08-03T20:46:39.189Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10430044, 5715022, 172 committed:true) SnapCreateDur 453.490157ms SnapOpenDur 64.886µs 2015-08-03T20:46:39.227Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6240000 2015-08-03T20:46:39.515Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5800000 QueuedCount 2822 2015-08-03T20:46:39.697Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10452696, 5726348, 0 committed:false) SnapCreateDur 28.287µs SnapOpenDur 84.035963ms 2015-08-03T20:46:40.108Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:446299975, Disk:2050154496, ExtraSnapshotData:0, Fragmentation:78.23091011576135% 2015-08-03T20:46:40.108Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:46:40.13Z-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 10165434. Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T20:46:40.262Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6250000 2015-08-03T20:46:41.069Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5810000 QueuedCount 4365 2015-08-03T20:46:41.203Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10475454, 5737727, 0 committed:false) SnapCreateDur 29.253µs SnapOpenDur 42.46966ms 2015-08-03T20:46:41.314Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6260000 2015-08-03T20:46:42.167Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.207µs CommitTime 830.877433ms TotalFlushTime 6m27.997393135s TotalCommitTime 1m52.794846934s 2015-08-03T20:46:42.167Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10479242, 5739621, 173 committed:true) SnapCreateDur 859.464739ms SnapOpenDur 31.991µs 2015-08-03T20:46:42.309Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5820000 QueuedCount 7038 2015-08-03T20:46:42.349Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6270000 2015-08-03T20:46:42.639Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10501758, 5750879, 0 committed:false) SnapCreateDur 30.853µs SnapOpenDur 78.329219ms 2015-08-03T20:46:42.734Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5830000 QueuedCount 4339 2015-08-03T20:46:43.39Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6280000 2015-08-03T20:46:43.952Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10523974, 5761987, 0 committed:false) SnapCreateDur 12.057µs SnapOpenDur 76.820698ms 2015-08-03T20:46:44.01Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5840000 QueuedCount 2857 2015-08-03T20:46:44.361Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5850000 QueuedCount 554 2015-08-03T20:46:44.436Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6290000 2015-08-03T20:46:44.538Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10546680, 5773340, 0 committed:false) SnapCreateDur 16.58µs SnapOpenDur 155.080596ms 2015-08-03T20:46:45.471Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6300000 2015-08-03T20:46:46.032Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.393µs CommitTime 362.814381ms TotalFlushTime 6m31.025051158s TotalCommitTime 1m53.157661315s 2015-08-03T20:46:46.032Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10550510, 5775255, 174 committed:true) SnapCreateDur 390.294709ms SnapOpenDur 38.144µs 2015-08-03T20:46:46.31Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5860000 QueuedCount 4057 2015-08-03T20:46:46.514Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6310000 2015-08-03T20:46:46.515Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10573284, 5786642, 0 committed:false) SnapCreateDur 23.383µs SnapOpenDur 70.499682ms 2015-08-03T20:46:46.767Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5870000 QueuedCount 5611 2015-08-03T20:46:47.56Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6320000 2015-08-03T20:46:47.903Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10595988, 5797994, 0 committed:false) SnapCreateDur 24.713µs SnapOpenDur 121.927144ms 2015-08-03T20:46:48.463Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.496µs CommitTime 452.212982ms TotalFlushTime 6m32.692414856s TotalCommitTime 1m53.609874297s 2015-08-03T20:46:48.463Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10599814, 5799907, 175 committed:true) SnapCreateDur 465.723354ms SnapOpenDur 38.528µs 2015-08-03T20:46:48.544Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5880000 QueuedCount 3334 2015-08-03T20:46:48.609Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6330000 2015-08-03T20:46:48.858Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10622500, 5811250, 0 committed:false) SnapCreateDur 19.894µs SnapOpenDur 70.638873ms 2015-08-03T20:46:48.921Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5890000 QueuedCount 2922 2015-08-03T20:46:49.641Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6340000 2015-08-03T20:46:50.689Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6350000 2015-08-03T20:46:51.053Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5900000 QueuedCount 683 2015-08-03T20:46:51.202Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10645378, 5822689, 0 committed:false) SnapCreateDur 27.864µs SnapOpenDur 125.920668ms 2015-08-03T20:46:51.726Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6360000 2015-08-03T20:46:52.094Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 14.943µs CommitTime 780.593551ms TotalFlushTime 6m35.217531014s TotalCommitTime 1m54.390467848s 2015-08-03T20:46:52.094Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10648850, 5824425, 176 committed:true) SnapCreateDur 806.55933ms SnapOpenDur 34.623µs 2015-08-03T20:46:52.368Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5910000 QueuedCount 3544 2015-08-03T20:46:52.557Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10670708, 5835354, 0 committed:false) SnapCreateDur 28.972µs SnapOpenDur 68.824627ms 2015-08-03T20:46:52.766Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6370000 2015-08-03T20:46:53.694Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5920000 QueuedCount 5108 2015-08-03T20:46:53.809Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6380000 2015-08-03T20:46:53.918Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10693434, 5846717, 0 committed:false) SnapCreateDur 30.957µs SnapOpenDur 125.670771ms 2015-08-03T20:46:54.113Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5930000 QueuedCount 6521 2015-08-03T20:46:54.838Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6390000 2015-08-03T20:46:55.256Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10715910, 5857955, 0 committed:false) SnapCreateDur 22.17µs SnapOpenDur 105.315093ms 2015-08-03T20:46:55.789Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.827µs CommitTime 401.003138ms TotalFlushTime 6m38.036316912s TotalCommitTime 1m54.791470986s 2015-08-03T20:46:55.789Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10719596, 5859798, 177 committed:true) SnapCreateDur 428.497688ms SnapOpenDur 71.21µs 2015-08-03T20:46:55.865Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5940000 QueuedCount 4177 2015-08-03T20:46:55.883Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6400000 2015-08-03T20:46:56.272Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10742386, 5871193, 0 committed:false) SnapCreateDur 26.234µs SnapOpenDur 68.702607ms 2015-08-03T20:46:56.294Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5950000 QueuedCount 1312 2015-08-03T20:46:56.933Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6410000 2015-08-03T20:46:57.594Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5960000 QueuedCount 1473 2015-08-03T20:46:57.707Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10765042, 5882521, 0 committed:false) SnapCreateDur 22.849µs SnapOpenDur 82.122753ms 2015-08-03T20:46:57.888Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5970000 QueuedCount 2929 2015-08-03T20:46:57.986Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6420000 2015-08-03T20:46:58.14Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10787578, 5893789, 0 committed:false) SnapCreateDur 21.904µs SnapOpenDur 199.227423ms 2015-08-03T20:46:59.016Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6430000 2015-08-03T20:46:59.25Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5980000 QueuedCount 4501 2015-08-03T20:47:00.064Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6440000 2015-08-03T20:47:00.336Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 13.233µs CommitTime 977.740197ms TotalFlushTime 6m41.099330897s TotalCommitTime 1m55.769211183s 2015-08-03T20:47:00.337Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10810360, 5905180, 178 committed:true) SnapCreateDur 1.003906829s SnapOpenDur 34.144µs 2015-08-03T20:47:00.547Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 5990000 QueuedCount 5977 2015-08-03T20:47:00.843Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10832942, 5916471, 0 committed:false) SnapCreateDur 26.05µs SnapOpenDur 73.395176ms 2015-08-03T20:47:01.011Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6000000 QueuedCount 7524 2015-08-03T20:47:01.11Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6450000 2015-08-03T20:47:02.14Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6460000 2015-08-03T20:47:02.315Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10855654, 5927827, 0 committed:false) SnapCreateDur 35.01µs SnapOpenDur 128.40165ms 2015-08-03T20:47:03.191Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6470000 2015-08-03T20:47:03.269Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.353µs CommitTime 833.759447ms TotalFlushTime 6m42.862072222s TotalCommitTime 1m56.60297063s 2015-08-03T20:47:03.269Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10859424, 5929712, 179 committed:true) SnapCreateDur 859.55356ms SnapOpenDur 25.194µs 2015-08-03T20:47:03.302Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6010000 QueuedCount 1414 2015-08-03T20:47:03.69Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6020000 QueuedCount 907 2015-08-03T20:47:03.799Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10881956, 5940978, 0 committed:false) SnapCreateDur 29.86µs SnapOpenDur 75.969566ms 2015-08-03T20:47:04.241Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6480000 2015-08-03T20:47:05.085Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6030000 QueuedCount 2392 2015-08-03T20:47:05.216Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":14480874658,"bucket-1:by_email:avg_scan_wait_latency":14479933456,"bucket-1:by_email:avg_ts_interval":811421204,"bucket-1:by_email:avg_ts_items_count":9072,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":479791655,"bucket-1:by_email:delete_bytes":211408253,"bucket-1:by_email:disk_size":2535788544,"bucket-1:by_email:flush_queue_size":0,"bucket-1:by_email:frag_percent":81,"bucket-1:by_email:get_bytes":216069023,"bucket-1:by_email:insert_bytes":621229455,"bucket-1:by_email:items_count":1000001,"bucket-1:by_email:num_commits":179,"bucket-1:by_email:num_compactions":9,"bucket-1:by_email:num_completed_requests":14211,"bucket-1:by_email:num_docs_indexed":5996702,"bucket-1:by_email:num_docs_pending":223,"bucket-1:by_email:num_docs_queued":452799,"bucket-1:by_email:num_flush_queued":6032392,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":14595,"bucket-1:by_email:num_rows_returned":231,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":741,"bucket-1:by_email:scan_bytes_read":9240,"bucket-1:by_email:scan_wait_duration":211334628793295,"bucket-1:by_email:since_last_snapshot":530952316,"bucket-1:by_email:timings/storage_clone_handle":"14211 830443536 2741463994131538","bucket-1:by_email:timings/storage_commit":"179 116602970630 -2701909227028700538","bucket-1:by_email:timings/storage_del":"4952272 79183840320 8659747177794501830","bucket-1:by_email:timings/storage_get":"6032392 81962107799 -5431246106564650567","bucket-1:by_email:timings/storage_handle_open":"179 9622620464 -4878300604686073754","bucket-1:by_email:timings/storage_iterator_next":"231 4688424 137630603530","bucket-1:by_email:timings/storage_new_iterator":"14211 1199305178 2849028350542010","bucket-1:by_email:timings/storage_set":"11904544 235689129965 -2736529144555456787","bucket-1:by_email:timings/storage_snapshot_close":"560 9062279931 801187126961821071","bucket-1:by_email:timings/storage_snapshot_create":"561 52881540348 8491939032634206586","bucket-1:by_email:total_scan_duration":211348365640941,"bucket-1:mutation_queue_size":456185,"bucket-1:num_mutations_queued":6488577,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":237,"memory_quota":104857600000,"memory_used":1110564864,"needs_restart":false,"num_connections":384,"timings/stats_response":"1141 400231252 4738987650437200"} 2015-08-03T20:47:05.263Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10904544, 5952272, 0 committed:false) SnapCreateDur 20.837µs SnapOpenDur 129.309963ms 2015-08-03T20:47:05.365Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6490000 2015-08-03T20:47:05.438Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6040000 QueuedCount 87 2015-08-03T20:47:06.127Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.037µs CommitTime 674.169256ms TotalFlushTime 6m44.734159846s TotalCommitTime 1m57.277139886s 2015-08-03T20:47:06.127Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10919660, 5959830, 180 committed:true) SnapCreateDur 687.396548ms SnapOpenDur 32.207µs 2015-08-03T20:47:06.33Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6050000 QueuedCount 1593 2015-08-03T20:47:06.415Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6500000 2015-08-03T20:47:06.432Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10942296, 5971148, 0 committed:false) SnapCreateDur 21.61µs SnapOpenDur 72.611365ms 2015-08-03T20:47:07.46Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6510000 2015-08-03T20:47:07.667Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6060000 QueuedCount 3154 2015-08-03T20:47:07.857Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10965060, 5982530, 0 committed:false) SnapCreateDur 20.99µs SnapOpenDur 121.308872ms 2015-08-03T20:47:08.15Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6070000 QueuedCount 4741 2015-08-03T20:47:08.509Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6520000 2015-08-03T20:47:09.201Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10987866, 5993933, 0 committed:false) SnapCreateDur 18.152µs SnapOpenDur 71.780668ms 2015-08-03T20:47:09.541Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6530000 2015-08-03T20:47:09.741Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.833µs CommitTime 420.814533ms TotalFlushTime 6m47.497421454s TotalCommitTime 1m57.697954419s 2015-08-03T20:47:09.741Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 10991552, 5995776, 181 committed:true) SnapCreateDur 446.823066ms SnapOpenDur 38.015µs 2015-08-03T20:47:09.881Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6080000 QueuedCount 7077 2015-08-03T20:47:10.244Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11014374, 6007187, 0 committed:false) SnapCreateDur 29.573µs SnapOpenDur 71.097854ms 2015-08-03T20:47:10.331Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6090000 QueuedCount 4003 2015-08-03T20:47:10.585Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6540000 2015-08-03T20:47:11.625Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6550000 2015-08-03T20:47:11.709Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11037100, 6018550, 0 committed:false) SnapCreateDur 24.594µs SnapOpenDur 132.986358ms 2015-08-03T20:47:11.741Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6100000 QueuedCount 1718 2015-08-03T20:47:12.107Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6110000 QueuedCount 1192 2015-08-03T20:47:12.663Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6560000 2015-08-03T20:47:13.094Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11059594, 6029797, 0 committed:false) SnapCreateDur 20.605µs SnapOpenDur 71.262023ms 2015-08-03T20:47:13.684Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.55µs CommitTime 488.989864ms TotalFlushTime 6m50.50876458s TotalCommitTime 1m58.186944283s 2015-08-03T20:47:13.684Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11063576, 6031788, 182 committed:true) SnapCreateDur 515.194087ms SnapOpenDur 42.723µs 2015-08-03T20:47:13.702Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6570000 2015-08-03T20:47:13.939Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6120000 QueuedCount 4638 2015-08-03T20:47:14.184Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11086052, 6043026, 0 committed:false) SnapCreateDur 21.577µs SnapOpenDur 75.725056ms 2015-08-03T20:47:14.406Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6130000 QueuedCount 6164 2015-08-03T20:47:14.737Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6580000 2015-08-03T20:47:15.492Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11108742, 6054371, 0 committed:false) SnapCreateDur 29.337µs SnapOpenDur 133.539366ms 2015-08-03T20:47:15.66Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6140000 QueuedCount 7686 2015-08-03T20:47:15.782Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6590000 2015-08-03T20:47:16.831Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6600000 2015-08-03T20:47:16.867Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11131442, 6065721, 0 committed:false) SnapCreateDur 20.087µs SnapOpenDur 113.113797ms 2015-08-03T20:47:17.446Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.88µs CommitTime 449.374943ms TotalFlushTime 6m53.332230351s TotalCommitTime 1m58.636319226s 2015-08-03T20:47:17.446Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11135246, 6067623, 183 committed:true) SnapCreateDur 475.136697ms SnapOpenDur 52.74µs 2015-08-03T20:47:17.479Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6150000 QueuedCount 1576 2015-08-03T20:47:17.77Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6160000 QueuedCount 1160 2015-08-03T20:47:17.843Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11157972, 6078986, 0 committed:false) SnapCreateDur 23.079µs SnapOpenDur 43.300289ms 2015-08-03T20:47:17.883Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6610000 2015-08-03T20:47:18.924Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6620000 2015-08-03T20:47:19.119Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6170000 QueuedCount 2711 2015-08-03T20:47:19.258Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11180714, 6090357, 0 committed:false) SnapCreateDur 12.882µs SnapOpenDur 83.856046ms 2015-08-03T20:47:19.427Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6180000 QueuedCount 4244 2015-08-03T20:47:19.964Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6630000 2015-08-03T20:47:21.01Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6640000 2015-08-03T20:47:22.04Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6650000 2015-08-03T20:47:22.06Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 2015-08-03T20:47:22.097Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11203414, 6101707, 0 committed:false) SnapCreateDur 21.403µs SnapOpenDur 4.204532ms 2015-08-03T20:47:23.087Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6660000 2015-08-03T20:47:23.5Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.86µs CommitTime 192.131594ms TotalFlushTime 6m57.813077716s TotalCommitTime 1m58.82845082s 2015-08-03T20:47:23.5Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11207196, 6103598, 184 committed:true) SnapCreateDur 1.308729504s SnapOpenDur 38.538µs 2015-08-03T20:47:23.589Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6190000 QueuedCount 1958 2015-08-03T20:47:23.794Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11229812, 6114906, 0 committed:false) SnapCreateDur 21.254µs SnapOpenDur 71.212604ms 2015-08-03T20:47:23.906Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6200000 QueuedCount 5389 2015-08-03T20:47:24.122Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6670000 2015-08-03T20:47:25.158Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6680000 2015-08-03T20:47:25.342Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11252632, 6126316, 0 committed:false) SnapCreateDur 22.45µs SnapOpenDur 83.567066ms 2015-08-03T20:47:25.391Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6210000 QueuedCount 1150 2015-08-03T20:47:26.212Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6690000 2015-08-03T20:47:26.289Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.354µs CommitTime 826.193916ms TotalFlushTime 6m59.492362809s TotalCommitTime 1m59.654644736s 2015-08-03T20:47:26.289Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11256346, 6128173, 185 committed:true) SnapCreateDur 852.687008ms SnapOpenDur 30.309µs 2015-08-03T20:47:26.624Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6220000 QueuedCount 2510 2015-08-03T20:47:26.792Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11278698, 6139349, 0 committed:false) SnapCreateDur 19.446µs SnapOpenDur 79.65999ms 2015-08-03T20:47:27.266Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6700000 2015-08-03T20:47:28.111Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6230000 QueuedCount 4002 2015-08-03T20:47:28.279Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11301302, 6150651, 0 committed:false) SnapCreateDur 26.52µs SnapOpenDur 83.401726ms 2015-08-03T20:47:28.31Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6710000 2015-08-03T20:47:28.523Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6240000 QueuedCount 1671 2015-08-03T20:47:29.348Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6720000 2015-08-03T20:47:29.71Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 15.497µs CommitTime 1.099215063s TotalFlushTime 7m1.522837054s TotalCommitTime 2m0.753859799s 2015-08-03T20:47:29.71Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11316420, 6158210, 186 committed:true) SnapCreateDur 1.12573354s SnapOpenDur 43.802µs 2015-08-03T20:47:29.887Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6250000 QueuedCount 3175 2015-08-03T20:47:29.983Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11339088, 6169544, 0 committed:false) SnapCreateDur 20.584µs SnapOpenDur 38.952604ms 2015-08-03T20:47:30.385Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6730000 2015-08-03T20:47:30.788Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6260000 QueuedCount 4739 2015-08-03T20:47:30.99Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11361824, 6180912, 0 committed:false) SnapCreateDur 22.687µs SnapOpenDur 124.056552ms 2015-08-03T20:47:31.207Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6270000 QueuedCount 6259 2015-08-03T20:47:31.423Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6740000 2015-08-03T20:47:32.348Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11384478, 6192239, 0 committed:false) SnapCreateDur 26.647µs SnapOpenDur 66.538416ms 2015-08-03T20:47:32.461Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6750000 2015-08-03T20:47:32.943Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.607µs CommitTime 474.806177ms TotalFlushTime 7m3.899054494s TotalCommitTime 2m1.228665976s 2015-08-03T20:47:32.943Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11388238, 6194119, 187 committed:true) SnapCreateDur 501.29932ms SnapOpenDur 52.44µs 2015-08-03T20:47:33.033Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6280000 QueuedCount 3952 2015-08-03T20:47:33.472Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11410974, 6205487, 0 committed:false) SnapCreateDur 34.79µs SnapOpenDur 82.992126ms 2015-08-03T20:47:33.498Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6760000 2015-08-03T20:47:33.506Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6290000 QueuedCount 1636 2015-08-03T20:47:34.535Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6770000 2015-08-03T20:47:34.777Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6300000 QueuedCount 1234 2015-08-03T20:47:34.888Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11433622, 6216811, 0 committed:false) SnapCreateDur 11.912µs SnapOpenDur 81.342994ms 2015-08-03T20:47:35.244Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6310000 QueuedCount 2749 2015-08-03T20:47:35.579Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6780000 2015-08-03T20:47:36.362Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11456238, 6228119, 0 committed:false) SnapCreateDur 28.79µs SnapOpenDur 120.586664ms 2015-08-03T20:47:36.623Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6790000 2015-08-03T20:47:36.964Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.273µs CommitTime 471.114131ms TotalFlushTime 7m6.98856592s TotalCommitTime 2m1.699780107s 2015-08-03T20:47:36.964Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11460102, 6230051, 188 committed:true) SnapCreateDur 497.392741ms SnapOpenDur 42.367µs 2015-08-03T20:47:37.178Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6320000 QueuedCount 6227 2015-08-03T20:47:37.478Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11482770, 6241385, 0 committed:false) SnapCreateDur 34.054µs SnapOpenDur 79.100863ms 2015-08-03T20:47:37.676Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6800000 2015-08-03T20:47:37.726Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6330000 QueuedCount 7664 2015-08-03T20:47:38.715Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6810000 2015-08-03T20:47:39.038Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11505284, 6252642, 0 committed:false) SnapCreateDur 14.165µs SnapOpenDur 84.698053ms 2015-08-03T20:47:39.154Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6340000 QueuedCount 5369 2015-08-03T20:47:39.762Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6820000 2015-08-03T20:47:40.395Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11528050, 6264025, 0 committed:false) SnapCreateDur 13.929µs SnapOpenDur 69.761393ms 2015-08-03T20:47:40.431Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6350000 QueuedCount 1126 2015-08-03T20:47:40.809Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6830000 2015-08-03T20:47:40.932Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.37µs CommitTime 446.656912ms TotalFlushTime 7m10.043827607s TotalCommitTime 2m2.146437019s 2015-08-03T20:47:40.932Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11531792, 6265896, 189 committed:true) SnapCreateDur 472.749427ms SnapOpenDur 51.636µs 2015-08-03T20:47:41.286Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6360000 QueuedCount 2623 2015-08-03T20:47:41.466Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11554424, 6277212, 0 committed:false) SnapCreateDur 24.383µs SnapOpenDur 78.597066ms 2015-08-03T20:47:41.847Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6840000 2015-08-03T20:47:42.709Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6370000 QueuedCount 4247 2015-08-03T20:47:42.887Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6850000 2015-08-03T20:47:42.936Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11577318, 6288659, 0 committed:false) SnapCreateDur 34.124µs SnapOpenDur 144.42656ms 2015-08-03T20:47:43.188Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6380000 QueuedCount 5787 2015-08-03T20:47:43.926Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6860000 2015-08-03T20:47:44.339Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11600014, 6300007, 0 committed:false) SnapCreateDur 14.539µs SnapOpenDur 68.447575ms 2015-08-03T20:47:44.963Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6870000 2015-08-03T20:47:44.966Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.297µs CommitTime 506.404984ms TotalFlushTime 7m13.105764721s TotalCommitTime 2m2.652842003s 2015-08-03T20:47:44.966Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11603804, 6301902, 190 committed:true) SnapCreateDur 532.84824ms SnapOpenDur 44.767µs 2015-08-03T20:47:45.074Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6390000 QueuedCount 5309 2015-08-03T20:47:45.496Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11626458, 6313229, 0 committed:false) SnapCreateDur 33.503µs SnapOpenDur 85.762863ms 2015-08-03T20:47:45.549Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6400000 QueuedCount 3048 2015-08-03T20:47:46.009Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6880000 2015-08-03T20:47:46.79Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6410000 QueuedCount 647 2015-08-03T20:47:46.955Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11648972, 6324486, 0 committed:false) SnapCreateDur 21.017µs SnapOpenDur 144.695926ms 2015-08-03T20:47:47.06Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6890000 2015-08-03T20:47:47.178Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6420000 QueuedCount 2057 2015-08-03T20:47:48.111Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6900000 2015-08-03T20:47:48.124Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11671440, 6335720, 0 committed:false) SnapCreateDur 32.977µs SnapOpenDur 115.461197ms 2015-08-03T20:47:48.694Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.598µs CommitTime 477.685119ms TotalFlushTime 7m15.840159341s TotalCommitTime 2m3.130527122s 2015-08-03T20:47:48.694Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11675330, 6337665, 191 committed:true) SnapCreateDur 505.079408ms SnapOpenDur 53.03µs 2015-08-03T20:47:48.933Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6430000 QueuedCount 5557 2015-08-03T20:47:49.14Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6910000 2015-08-03T20:47:49.213Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11698028, 6349014, 0 committed:false) SnapCreateDur 43.823µs SnapOpenDur 84.573837ms 2015-08-03T20:47:49.408Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6440000 QueuedCount 7095 2015-08-03T20:47:50.187Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6920000 2015-08-03T20:47:50.727Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11720732, 6360366, 0 committed:false) SnapCreateDur 24.007µs SnapOpenDur 149.200204ms 2015-08-03T20:47:50.869Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6450000 QueuedCount 7156 2015-08-03T20:47:51.22Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6930000 2015-08-03T20:47:52.06Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:447409492, Disk:1939030016, ExtraSnapshotData:0, Fragmentation:76.92611830099695% 2015-08-03T20:47:52.06Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:47:52.089Z-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 11388238. Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T20:47:52.262Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6940000 2015-08-03T20:47:52.452Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11743410, 6371705, 0 committed:false) SnapCreateDur 30.744µs SnapOpenDur 421.177µs 2015-08-03T20:47:52.527Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6460000 QueuedCount 578 2015-08-03T20:47:52.884Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.316µs CommitTime 299.112056ms TotalFlushTime 7m19.320091161s TotalCommitTime 2m3.429639178s 2015-08-03T20:47:52.884Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11747292, 6373646, 192 committed:true) SnapCreateDur 327.211829ms SnapOpenDur 49.997µs 2015-08-03T20:47:53.28Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6470000 QueuedCount 2053 2015-08-03T20:47:53.311Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6950000 2015-08-03T20:47:53.439Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11769916, 6384958, 0 committed:false) SnapCreateDur 21.623µs SnapOpenDur 79.901521ms 2015-08-03T20:47:54.34Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6960000 2015-08-03T20:47:54.667Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6480000 QueuedCount 3434 2015-08-03T20:47:54.818Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11792302, 6396151, 0 committed:false) SnapCreateDur 13.629µs SnapOpenDur 76.886609ms 2015-08-03T20:47:54.97Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6490000 QueuedCount 4146 2015-08-03T20:47:55.387Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6970000 2015-08-03T20:47:56.428Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6980000 2015-08-03T20:47:57.015Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11813388, 6406694, 0 committed:false) SnapCreateDur 22.82µs SnapOpenDur 105.60998ms 2015-08-03T20:47:57.477Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6990000 2015-08-03T20:47:57.536Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.017µs CommitTime 399.202165ms TotalFlushTime 7m23.143164673s TotalCommitTime 2m3.828841343s 2015-08-03T20:47:57.536Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11817200, 6408600, 193 committed:true) SnapCreateDur 425.981053ms SnapOpenDur 36.32µs 2015-08-03T20:47:57.695Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6500000 QueuedCount 7542 2015-08-03T20:47:58.023Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11839764, 6419882, 0 committed:false) SnapCreateDur 18.303µs SnapOpenDur 75.159112ms 2015-08-03T20:47:58.142Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6510000 QueuedCount 7138 2015-08-03T20:47:58.512Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7000000 2015-08-03T20:47:59.409Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11862332, 6431166, 0 committed:false) SnapCreateDur 23.62µs SnapOpenDur 134.097274ms 2015-08-03T20:47:59.468Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6520000 QueuedCount 2851 2015-08-03T20:47:59.56Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7010000 2015-08-03T20:48:00.602Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7020000 2015-08-03T20:48:00.694Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6530000 QueuedCount 545 2015-08-03T20:48:00.818Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11885056, 6442528, 0 committed:false) SnapCreateDur 36.17µs SnapOpenDur 107.429723ms 2015-08-03T20:48:01.371Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.174µs CommitTime 462.571955ms TotalFlushTime 7m26.038310509s TotalCommitTime 2m4.291413298s 2015-08-03T20:48:01.371Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11888840, 6444420, 194 committed:true) SnapCreateDur 488.762175ms SnapOpenDur 49.117µs 2015-08-03T20:48:01.64Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7030000 2015-08-03T20:48:01.649Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6540000 QueuedCount 3989 2015-08-03T20:48:01.865Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11911530, 6455765, 0 committed:false) SnapCreateDur 29.108µs SnapOpenDur 71.083051ms 2015-08-03T20:48:02.684Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7040000 2015-08-03T20:48:03.003Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6550000 QueuedCount 5573 2015-08-03T20:48:03.259Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11934330, 6467165, 0 committed:false) SnapCreateDur 29.164µs SnapOpenDur 139.26653ms 2015-08-03T20:48:03.379Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6560000 QueuedCount 5188 2015-08-03T20:48:03.725Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7050000 2015-08-03T20:48:04.279Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11957048, 6478524, 0 committed:false) SnapCreateDur 12.418µs SnapOpenDur 68.155048ms 2015-08-03T20:48:04.637Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.655µs CommitTime 281.24302ms TotalFlushTime 7m28.592590836s TotalCommitTime 2m4.572656318s 2015-08-03T20:48:04.637Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11960786, 6480393, 195 committed:true) SnapCreateDur 293.909388ms SnapOpenDur 27.663µs 2015-08-03T20:48:04.674Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6570000 QueuedCount 968 2015-08-03T20:48:04.79Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7060000 2015-08-03T20:48:04.839Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6580000 QueuedCount 589 2015-08-03T20:48:04.897Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 11983550, 6491775, 0 committed:false) SnapCreateDur 17.796µs SnapOpenDur 38.1416ms 2015-08-03T20:48:05.216Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":15394823789,"bucket-1:by_email:avg_scan_wait_latency":15393871631,"bucket-1:by_email:avg_ts_interval":477409212,"bucket-1:by_email:avg_ts_items_count":8987,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":461283731,"bucket-1:by_email:delete_bytes":234110060,"bucket-1:by_email:disk_size":2186145792,"bucket-1:by_email:flush_queue_size":6292,"bucket-1:by_email:frag_percent":78,"bucket-1:by_email:get_bytes":239171934,"bucket-1:by_email:insert_bytes":676664100,"bucket-1:by_email:items_count":1000001,"bucket-1:by_email:num_commits":195,"bucket-1:by_email:num_compactions":10,"bucket-1:by_email:num_completed_requests":14595,"bucket-1:by_email:num_docs_indexed":6533320,"bucket-1:by_email:num_docs_pending":269,"bucket-1:by_email:num_docs_queued":496435,"bucket-1:by_email:num_flush_queued":6592128,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":14979,"bucket-1:by_email:num_rows_returned":238,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":801,"bucket-1:by_email:scan_bytes_read":9520,"bucket-1:by_email:scan_wait_duration":230584803164554,"bucket-1:by_email:since_last_snapshot":259434884,"bucket-1:by_email:timings/storage_clone_handle":"14595 848169594 2742458695006576","bucket-1:by_email:timings/storage_commit":"195 124572656318 1971935195709560130","bucket-1:by_email:timings/storage_del":"5496942 89547021516 -3217590937959255012","bucket-1:by_email:timings/storage_get":"6585837 90544164720 -3844254120600049728","bucket-1:by_email:timings/storage_handle_open":"195 9623233229 -4878300580420692545","bucket-1:by_email:timings/storage_iterator_next":"238 4780185 138929031691","bucket-1:by_email:timings/storage_new_iterator":"14595 1224516415 2851075516158179","bucket-1:by_email:timings/storage_set":"12993884 260408356545 -1510503739368723003","bucket-1:by_email:timings/storage_snapshot_close":"605 9784994777 815242432329845189","bucket-1:by_email:timings/storage_snapshot_create":"606 56838582637 8891823189309216433","bucket-1:by_email:total_scan_duration":230599065546404,"bucket-1:mutation_queue_size":471158,"bucket-1:num_mutations_queued":7063286,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":245,"memory_quota":104857600000,"memory_used":821223424,"needs_restart":false,"num_connections":384,"timings/stats_response":"1213 425364068 4748069105187772"} 2015-08-03T20:48:05.937Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7070000 2015-08-03T20:48:06.165Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6590000 QueuedCount 2128 2015-08-03T20:48:06.353Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12006286, 6503143, 0 committed:false) SnapCreateDur 18.223µs SnapOpenDur 135.803313ms 2015-08-03T20:48:06.52Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6600000 QueuedCount 3505 2015-08-03T20:48:06.991Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7080000 2015-08-03T20:48:07.219Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12028676, 6514338, 0 committed:false) SnapCreateDur 12.098µs SnapOpenDur 62.95754ms 2015-08-03T20:48:07.47Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6610000 QueuedCount 1169 2015-08-03T20:48:08.036Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7090000 2015-08-03T20:48:08.374Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.367µs CommitTime 825.542143ms TotalFlushTime 7m31.097806105s TotalCommitTime 2m5.398198461s 2015-08-03T20:48:08.374Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12043752, 6521876, 196 committed:true) SnapCreateDur 851.617393ms SnapOpenDur 28.482µs 2015-08-03T20:48:08.581Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6620000 QueuedCount 2691 2015-08-03T20:48:08.693Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12066434, 6533217, 0 committed:false) SnapCreateDur 20.191µs SnapOpenDur 45.318892ms 2015-08-03T20:48:09.076Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7100000 2015-08-03T20:48:10.067Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6630000 QueuedCount 4226 2015-08-03T20:48:10.114Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7110000 2015-08-03T20:48:10.352Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12089114, 6544557, 0 committed:false) SnapCreateDur 22.04µs SnapOpenDur 136.241947ms 2015-08-03T20:48:11.146Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7120000 2015-08-03T20:48:11.284Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.804µs CommitTime 811.890321ms TotalFlushTime 7m32.891873332s TotalCommitTime 2m6.210088782s 2015-08-03T20:48:11.284Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12092932, 6546466, 197 committed:true) SnapCreateDur 837.241379ms SnapOpenDur 30.373µs 2015-08-03T20:48:11.437Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6640000 QueuedCount 7682 2015-08-03T20:48:11.79Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12115572, 6557786, 0 committed:false) SnapCreateDur 21.932µs SnapOpenDur 83.522976ms 2015-08-03T20:48:11.898Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6650000 QueuedCount 5365 2015-08-03T20:48:12.186Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7130000 2015-08-03T20:48:13.231Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7140000 2015-08-03T20:48:14.264Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7150000 2015-08-03T20:48:14.276Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12138206, 6569103, 0 committed:false) SnapCreateDur 37.668µs SnapOpenDur 143.350007ms 2015-08-03T20:48:14.328Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6660000 QueuedCount 1100 2015-08-03T20:48:15.233Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.447µs CommitTime 835.704718ms TotalFlushTime 7m35.641416269s TotalCommitTime 2m7.0457935s 2015-08-03T20:48:15.233Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12141968, 6570984, 198 committed:true) SnapCreateDur 861.979356ms SnapOpenDur 37.373µs 2015-08-03T20:48:15.308Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7160000 2015-08-03T20:48:15.567Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6670000 QueuedCount 2668 2015-08-03T20:48:15.729Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12164774, 6582387, 0 committed:false) SnapCreateDur 22.107µs SnapOpenDur 72.839928ms 2015-08-03T20:48:16.353Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7170000 2015-08-03T20:48:16.97Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6680000 QueuedCount 4181 2015-08-03T20:48:17.192Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12187462, 6593731, 0 committed:false) SnapCreateDur 31.427µs SnapOpenDur 140.364688ms 2015-08-03T20:48:17.411Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7180000 2015-08-03T20:48:17.427Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6690000 QueuedCount 3707 2015-08-03T20:48:18.453Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7190000 2015-08-03T20:48:18.686Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.163µs CommitTime 1.099191887s TotalFlushTime 7m37.646337454s TotalCommitTime 2m8.144985387s 2015-08-03T20:48:19.491Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7200000 2015-08-03T20:48:20.012Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12206210, 6603105, 199 committed:true) SnapCreateDur 1.125108796s SnapOpenDur 1.325985045s 2015-08-03T20:48:20.273Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6700000 QueuedCount 5074 2015-08-03T20:48:20.514Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12228580, 6614290, 0 committed:false) SnapCreateDur 21.047µs SnapOpenDur 69.72447ms 2015-08-03T20:48:20.535Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7210000 2015-08-03T20:48:21.571Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7220000 2015-08-03T20:48:21.668Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.887µs CommitTime 1.032618312s TotalFlushTime 7m38.104368971s TotalCommitTime 2m9.177603699s 2015-08-03T20:48:21.668Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12232450, 6616225, 200 committed:true) SnapCreateDur 1.059710497s SnapOpenDur 36.478µs 2015-08-03T20:48:21.793Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6710000 QueuedCount 6681 2015-08-03T20:48:22.122Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12255264, 6627632, 0 committed:false) SnapCreateDur 30.313µs SnapOpenDur 40.076082ms 2015-08-03T20:48:22.193Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6720000 QueuedCount 4316 2015-08-03T20:48:22.613Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7230000 2015-08-03T20:48:23.335Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6730000 QueuedCount 161 2015-08-03T20:48:23.476Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12277946, 6638973, 0 committed:false) SnapCreateDur 22.14µs SnapOpenDur 131.855249ms 2015-08-03T20:48:23.659Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7240000 2015-08-03T20:48:24.573Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.703µs CommitTime 929.74629ms TotalFlushTime 7m39.775390085s TotalCommitTime 2m10.107349989s 2015-08-03T20:48:24.573Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12289358, 6644679, 201 committed:true) SnapCreateDur 955.786775ms SnapOpenDur 37.568µs 2015-08-03T20:48:24.681Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6740000 QueuedCount 3641 2015-08-03T20:48:24.698Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7250000 2015-08-03T20:48:24.923Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12312214, 6656107, 0 committed:false) SnapCreateDur 29.066µs SnapOpenDur 78.240477ms 2015-08-03T20:48:25.036Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6750000 QueuedCount 5219 2015-08-03T20:48:25.738Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7260000 2015-08-03T20:48:26.789Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7270000 2015-08-03T20:48:27.539Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12334938, 6667469, 0 committed:false) SnapCreateDur 31.294µs SnapOpenDur 152.516612ms 2015-08-03T20:48:27.595Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6760000 QueuedCount 982 2015-08-03T20:48:27.832Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7280000 2015-08-03T20:48:28.421Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 3.686µs CommitTime 760.831493ms TotalFlushTime 7m42.506953289s TotalCommitTime 2m10.868181482s 2015-08-03T20:48:28.421Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12338650, 6669325, 202 committed:true) SnapCreateDur 786.883443ms SnapOpenDur 30.319µs 2015-08-03T20:48:28.768Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6770000 QueuedCount 2553 2015-08-03T20:48:28.866Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7290000 2015-08-03T20:48:28.92Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12361440, 6680720, 0 committed:false) SnapCreateDur 22.24µs SnapOpenDur 66.201707ms 2015-08-03T20:48:29.91Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7300000 2015-08-03T20:48:30.144Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6780000 QueuedCount 4118 2015-08-03T20:48:30.318Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12384170, 6692085, 0 committed:false) SnapCreateDur 10.416µs SnapOpenDur 83.958199ms 2015-08-03T20:48:30.461Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6790000 QueuedCount 5427 2015-08-03T20:48:30.944Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7310000 2015-08-03T20:48:31.317Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12406650, 6703325, 0 committed:false) SnapCreateDur 26.43µs SnapOpenDur 116.21592ms 2015-08-03T20:48:31.961Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.026µs CommitTime 523.435292ms TotalFlushTime 7m45.099883834s TotalCommitTime 2m11.391616774s 2015-08-03T20:48:31.961Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12410288, 6705144, 203 committed:true) SnapCreateDur 549.376692ms SnapOpenDur 36.58µs 2015-08-03T20:48:31.988Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7320000 2015-08-03T20:48:32.071Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6800000 QueuedCount 5040 2015-08-03T20:48:32.44Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12432872, 6716436, 0 committed:false) SnapCreateDur 25.713µs SnapOpenDur 76.682626ms 2015-08-03T20:48:32.501Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6810000 QueuedCount 2711 2015-08-03T20:48:33.028Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7330000 2015-08-03T20:48:33.734Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6820000 QueuedCount 342 2015-08-03T20:48:33.875Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12455388, 6727694, 0 committed:false) SnapCreateDur 21.473µs SnapOpenDur 130.619381ms 2015-08-03T20:48:34.064Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7340000 2015-08-03T20:48:34.252Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6830000 QueuedCount 1891 2015-08-03T20:48:35.104Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7350000 2015-08-03T20:48:35.318Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12478140, 6739070, 0 committed:false) SnapCreateDur 22.107µs SnapOpenDur 114.193919ms 2015-08-03T20:48:35.668Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.466µs CommitTime 272.367012ms TotalFlushTime 7m48.049062167s TotalCommitTime 2m11.663983786s 2015-08-03T20:48:35.669Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12481900, 6740950, 204 committed:true) SnapCreateDur 286.124473ms SnapOpenDur 33.44µs 2015-08-03T20:48:35.896Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6840000 QueuedCount 5332 2015-08-03T20:48:36.112Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12504572, 6752286, 0 committed:false) SnapCreateDur 10.885µs SnapOpenDur 41.972499ms 2015-08-03T20:48:36.158Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7360000 2015-08-03T20:48:36.302Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6850000 QueuedCount 6915 2015-08-03T20:48:37.191Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7370000 2015-08-03T20:48:37.598Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12527378, 6763689, 0 committed:false) SnapCreateDur 22.16µs SnapOpenDur 134.855266ms 2015-08-03T20:48:37.683Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6860000 QueuedCount 2637 2015-08-03T20:48:38.237Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7380000 2015-08-03T20:48:38.665Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12550012, 6775006, 0 committed:false) SnapCreateDur 20.59µs SnapOpenDur 101.471642ms 2015-08-03T20:48:38.718Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6870000 QueuedCount 346 2015-08-03T20:48:39.019Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 5.733µs CommitTime 276.034564ms TotalFlushTime 7m50.695455359s TotalCommitTime 2m11.94001835s 2015-08-03T20:48:39.019Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12553816, 6776908, 205 committed:true) SnapCreateDur 289.937032ms SnapOpenDur 32.443µs 2015-08-03T20:48:39.277Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7390000 2015-08-03T20:48:39.363Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6880000 QueuedCount 1838 2015-08-03T20:48:39.476Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12576412, 6788206, 0 committed:false) SnapCreateDur 22.02µs SnapOpenDur 46.581147ms 2015-08-03T20:48:40.316Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7400000 2015-08-03T20:48:41.361Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7410000 2015-08-03T20:48:41.849Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 2015-08-03T20:48:41.873Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6890000 QueuedCount 3270 2015-08-03T20:48:42.022Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12598888, 6799444, 0 committed:false) SnapCreateDur 28.987µs SnapOpenDur 24.354649ms 2015-08-03T20:48:42.402Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7420000 2015-08-03T20:48:43.437Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7430000 2015-08-03T20:48:43.642Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.256µs CommitTime 368.598732ms TotalFlushTime 7m53.614773002s TotalCommitTime 2m12.308617082s 2015-08-03T20:48:43.642Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12602676, 6801338, 206 committed:true) SnapCreateDur 1.525973155s SnapOpenDur 49.457µs 2015-08-03T20:48:43.819Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6900000 QueuedCount 6704 2015-08-03T20:48:44.13Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12625344, 6812672, 0 committed:false) SnapCreateDur 23.217µs SnapOpenDur 77.729733ms 2015-08-03T20:48:44.283Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6910000 QueuedCount 8280 2015-08-03T20:48:44.474Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7440000 2015-08-03T20:48:45.497Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12648142, 6824071, 0 committed:false) SnapCreateDur 19.194µs SnapOpenDur 82.494475ms 2015-08-03T20:48:45.512Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7450000 2015-08-03T20:48:45.581Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6920000 QueuedCount 192 2015-08-03T20:48:46.434Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.734µs CommitTime 814.939298ms TotalFlushTime 7m55.307028147s TotalCommitTime 2m13.12355638s 2015-08-03T20:48:46.434Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12651902, 6825951, 207 committed:true) SnapCreateDur 842.918056ms SnapOpenDur 29.765µs 2015-08-03T20:48:46.561Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7460000 2015-08-03T20:48:46.675Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6930000 QueuedCount 1673 2015-08-03T20:48:46.801Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12674464, 6837232, 0 committed:false) SnapCreateDur 25.103µs SnapOpenDur 82.116056ms 2015-08-03T20:48:47.611Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7470000 2015-08-03T20:48:48.13Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6940000 QueuedCount 3191 2015-08-03T20:48:48.362Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12697128, 6848564, 0 committed:false) SnapCreateDur 33.633µs SnapOpenDur 156.45011ms 2015-08-03T20:48:48.528Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6950000 QueuedCount 4770 2015-08-03T20:48:48.647Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7480000 2015-08-03T20:48:49.383Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12719904, 6859952, 0 committed:false) SnapCreateDur 19.193µs SnapOpenDur 67.178119ms 2015-08-03T20:48:49.69Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7490000 2015-08-03T20:48:49.94Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.87µs CommitTime 436.943139ms TotalFlushTime 7m57.909050794s TotalCommitTime 2m13.560499519s 2015-08-03T20:48:49.94Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12723714, 6861857, 208 committed:true) SnapCreateDur 463.606833ms SnapOpenDur 45.224µs 2015-08-03T20:48:50.092Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6960000 QueuedCount 8245 2015-08-03T20:48:50.481Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12746416, 6873208, 0 committed:false) SnapCreateDur 38.843µs SnapOpenDur 83.904322ms 2015-08-03T20:48:50.575Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6970000 QueuedCount 3977 2015-08-03T20:48:50.735Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7500000 2015-08-03T20:48:51.776Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7510000 2015-08-03T20:48:51.991Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12769144, 6884572, 0 committed:false) SnapCreateDur 24.97µs SnapOpenDur 155.940968ms 2015-08-03T20:48:52.024Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6980000 QueuedCount 1652 2015-08-03T20:48:52.405Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 6990000 QueuedCount 1198 2015-08-03T20:48:52.811Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7520000 2015-08-03T20:48:53.426Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12791652, 6895826, 0 committed:false) SnapCreateDur 31.707µs SnapOpenDur 120.436757ms 2015-08-03T20:48:53.85Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7530000 2015-08-03T20:48:54.052Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.264µs CommitTime 534.250558ms TotalFlushTime 8m0.955271201s TotalCommitTime 2m14.094750077s 2015-08-03T20:48:54.052Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12795456, 6897728, 209 committed:true) SnapCreateDur 560.542213ms SnapOpenDur 47.127µs 2015-08-03T20:48:54.323Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7000000 QueuedCount 4649 2015-08-03T20:48:54.569Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12818142, 6909071, 0 committed:false) SnapCreateDur 25.563µs SnapOpenDur 80.046598ms 2015-08-03T20:48:54.886Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7540000 2015-08-03T20:48:55.735Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7010000 QueuedCount 6177 2015-08-03T20:48:55.934Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7550000 2015-08-03T20:48:55.939Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12840816, 6920408, 0 committed:false) SnapCreateDur 18.421µs SnapOpenDur 85.347474ms 2015-08-03T20:48:56.111Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7020000 QueuedCount 7697 2015-08-03T20:48:56.984Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7560000 2015-08-03T20:48:57.402Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12863532, 6931766, 0 committed:false) SnapCreateDur 24.36µs SnapOpenDur 120.481764ms 2015-08-03T20:48:58.028Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7570000 2015-08-03T20:48:58.063Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.003µs CommitTime 540.12035ms TotalFlushTime 8m3.975031088s TotalCommitTime 2m14.634870427s 2015-08-03T20:48:58.063Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12867324, 6933662, 210 committed:true) SnapCreateDur 566.728547ms SnapOpenDur 44.847µs 2015-08-03T20:48:58.099Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7030000 QueuedCount 1544 2015-08-03T20:48:58.466Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7040000 QueuedCount 1160 2015-08-03T20:48:58.588Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12890034, 6945017, 0 committed:false) SnapCreateDur 18.37µs SnapOpenDur 80.254421ms 2015-08-03T20:48:59.062Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7580000 2015-08-03T20:49:00.034Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7050000 QueuedCount 2639 2015-08-03T20:49:00.11Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7590000 2015-08-03T20:49:00.193Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12912642, 6956321, 0 committed:false) SnapCreateDur 33.157µs SnapOpenDur 99.322351ms 2015-08-03T20:49:00.502Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7060000 QueuedCount 3187 2015-08-03T20:49:01.141Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7600000 2015-08-03T20:49:01.68Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12933424, 6966712, 0 committed:false) SnapCreateDur 23.917µs SnapOpenDur 123.950153ms 2015-08-03T20:49:02.026Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.401µs CommitTime 266.306121ms TotalFlushTime 8m7.218482005s TotalCommitTime 2m14.901176548s 2015-08-03T20:49:02.026Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12936764, 6968382, 211 committed:true) SnapCreateDur 281.711298ms SnapOpenDur 34.411µs 2015-08-03T20:49:02.163Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7070000 QueuedCount 4402 2015-08-03T20:49:02.185Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7610000 2015-08-03T20:49:02.33Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12959226, 6979613, 0 committed:false) SnapCreateDur 20.141µs SnapOpenDur 40.387183ms 2015-08-03T20:49:02.427Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7080000 QueuedCount 3918 2015-08-03T20:49:03.225Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7620000 2015-08-03T20:49:03.391Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 12981784, 6990892, 0 committed:false) SnapCreateDur 20.583µs SnapOpenDur 148.027706ms 2015-08-03T20:49:03.508Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7090000 QueuedCount 5592 2015-08-03T20:49:04.263Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7630000 2015-08-03T20:49:04.987Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13004566, 7002283, 0 committed:false) SnapCreateDur 34.967µs SnapOpenDur 131.678186ms 2015-08-03T20:49:05.036Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7100000 QueuedCount 1248 2015-08-03T20:49:05.217Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":16380544893,"bucket-1:by_email:avg_scan_wait_latency":16379593346,"bucket-1:by_email:avg_ts_interval":1206339148,"bucket-1:by_email:avg_ts_items_count":10830,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":446449049,"bucket-1:by_email:delete_bytes":256640426,"bucket-1:by_email:disk_size":2006122496,"bucket-1:by_email:flush_queue_size":0,"bucket-1:by_email:frag_percent":77,"bucket-1:by_email:get_bytes":262099362,"bucket-1:by_email:insert_bytes":731680110,"bucket-1:by_email:items_count":1000001,"bucket-1:by_email:num_commits":211,"bucket-1:by_email:num_compactions":10,"bucket-1:by_email:num_completed_requests":14979,"bucket-1:by_email:num_docs_indexed":7066023,"bucket-1:by_email:num_docs_pending":147,"bucket-1:by_email:num_docs_queued":543937,"bucket-1:by_email:num_flush_queued":7101248,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":15363,"bucket-1:by_email:num_rows_returned":248,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":858,"bucket-1:by_email:scan_bytes_read":9920,"bucket-1:by_email:scan_wait_duration":251639692588022,"bucket-1:by_email:since_last_snapshot":1596532594,"bucket-1:by_email:timings/storage_clone_handle":"14979 865262915 2743326520987891","bucket-1:by_email:timings/storage_commit":"211 134901176548 -8654558304551275252","bucket-1:by_email:timings/storage_del":"6004179 101910547523 5768749081087027913","bucket-1:by_email:timings/storage_get":"7101248 97134504982 -3844147729152244118","bucket-1:by_email:timings/storage_handle_open":"211 10949712800 -3120077504749595782","bucket-1:by_email:timings/storage_iterator_next":"248 4973901 143328852127","bucket-1:by_email:timings/storage_new_iterator":"14979 1250648186 2853337235358506","bucket-1:by_email:timings/storage_set":"14008358 281652232762 -4539671208410451674","bucket-1:by_email:timings/storage_snapshot_close":"646 10442925234 827816997838407032","bucket-1:by_email:timings/storage_snapshot_create":"647 60804028911 -9117046108899257621","bucket-1:by_email:total_scan_duration":251654311205295,"bucket-1:mutation_queue_size":537379,"bucket-1:num_mutations_queued":7638627,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":279,"memory_quota":104857600000,"memory_used":606035968,"needs_restart":false,"num_connections":384,"timings/stats_response":"1285 449531702 4756574257196790"} 2015-08-03T20:49:05.352Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7640000 2015-08-03T20:49:05.673Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.78µs CommitTime 554.235381ms TotalFlushTime 8m9.830384045s TotalCommitTime 2m15.455411929s 2015-08-03T20:49:05.673Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13008358, 7004179, 212 committed:true) SnapCreateDur 580.807475ms SnapOpenDur 42.25µs 2015-08-03T20:49:06.019Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7110000 QueuedCount 2736 2015-08-03T20:49:06.197Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13030970, 7015485, 0 committed:false) SnapCreateDur 26.132µs SnapOpenDur 70.078621ms 2015-08-03T20:49:06.41Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7650000 2015-08-03T20:49:07.458Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7660000 2015-08-03T20:49:07.615Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7120000 QueuedCount 4332 2015-08-03T20:49:07.866Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13053804, 7026902, 0 committed:false) SnapCreateDur 22.577µs SnapOpenDur 153.878253ms 2015-08-03T20:49:08.059Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7130000 QueuedCount 5907 2015-08-03T20:49:08.51Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7670000 2015-08-03T20:49:09.318Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13076612, 7038306, 0 committed:false) SnapCreateDur 35.844µs SnapOpenDur 137.044353ms 2015-08-03T20:49:09.537Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7680000 2015-08-03T20:49:09.995Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.893µs CommitTime 584.218088ms TotalFlushTime 8m13.037880564s TotalCommitTime 2m16.039630017s 2015-08-03T20:49:09.996Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13080424, 7040212, 213 committed:true) SnapCreateDur 612.889035ms SnapOpenDur 71.717µs 2015-08-03T20:49:10.101Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7140000 QueuedCount 5553 2015-08-03T20:49:10.522Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13103136, 7051568, 0 committed:false) SnapCreateDur 24.564µs SnapOpenDur 81.302937ms 2015-08-03T20:49:10.571Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7150000 QueuedCount 3230 2015-08-03T20:49:10.587Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7690000 2015-08-03T20:49:11.622Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7700000 2015-08-03T20:49:11.822Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7160000 QueuedCount 956 2015-08-03T20:49:11.849Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:463348927, Disk:2158256128, ExtraSnapshotData:0, Fragmentation:78.53132809453096% 2015-08-03T20:49:11.849Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:49:11.86Z-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 12795456. Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T20:49:12.307Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13125880, 7062940, 0 committed:false) SnapCreateDur 25.267µs SnapOpenDur 459.904163ms 2015-08-03T20:49:12.662Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7710000 2015-08-03T20:49:12.979Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.913µs CommitTime 488.262426ms TotalFlushTime 8m14.859877866s TotalCommitTime 2m16.527892443s 2015-08-03T20:49:12.979Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13133472, 7066736, 214 committed:true) SnapCreateDur 514.44896ms SnapOpenDur 46.18µs 2015-08-03T20:49:13.187Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7170000 QueuedCount 6216 2015-08-03T20:49:13.515Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13155928, 7077964, 0 committed:false) SnapCreateDur 27.317µs SnapOpenDur 75.367119ms 2015-08-03T20:49:13.686Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7180000 QueuedCount 7647 2015-08-03T20:49:13.704Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7720000 2015-08-03T20:49:14.741Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7730000 2015-08-03T20:49:14.958Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13178436, 7089218, 0 committed:false) SnapCreateDur 21.214µs SnapOpenDur 141.356747ms 2015-08-03T20:49:15.071Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7190000 QueuedCount 5292 2015-08-03T20:49:15.784Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7740000 2015-08-03T20:49:16.337Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13201036, 7100518, 0 committed:false) SnapCreateDur 13.353µs SnapOpenDur 67.891092ms 2015-08-03T20:49:16.375Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7200000 QueuedCount 1053 2015-08-03T20:49:16.666Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.463µs CommitTime 251.393076ms TotalFlushTime 8m17.845183963s TotalCommitTime 2m16.779285519s 2015-08-03T20:49:16.666Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13204796, 7102398, 215 committed:true) SnapCreateDur 263.99465ms SnapOpenDur 24.944µs 2015-08-03T20:49:16.835Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7750000 2015-08-03T20:49:16.836Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7210000 QueuedCount 2621 2015-08-03T20:49:16.924Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13227568, 7113784, 0 committed:false) SnapCreateDur 18.137µs SnapOpenDur 37.845708ms 2015-08-03T20:49:17.674Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7220000 QueuedCount 4132 2015-08-03T20:49:17.823Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13250212, 7125106, 0 committed:false) SnapCreateDur 10.887µs SnapOpenDur 76.591526ms 2015-08-03T20:49:17.882Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7760000 2015-08-03T20:49:17.953Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7230000 QueuedCount 3711 2015-08-03T20:49:18.765Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13272808, 7136404, 0 committed:false) SnapCreateDur 25.051µs SnapOpenDur 111.727152ms 2015-08-03T20:49:18.923Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7770000 2015-08-03T20:49:18.971Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7240000 QueuedCount 7140 2015-08-03T20:49:19.445Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13295462, 7147731, 0 committed:false) SnapCreateDur 32.591µs SnapOpenDur 205.781523ms 2015-08-03T20:49:19.961Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7780000 2015-08-03T20:49:20.644Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.533µs CommitTime 1.068196277s TotalFlushTime 8m20.139102604s TotalCommitTime 2m17.847481796s 2015-08-03T20:49:20.644Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13299254, 7149627, 216 committed:true) SnapCreateDur 1.09529875s SnapOpenDur 36.501µs 2015-08-03T20:49:20.698Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7250000 QueuedCount 2928 2015-08-03T20:49:21.001Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7790000 2015-08-03T20:49:21.046Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7260000 QueuedCount 606 2015-08-03T20:49:21.154Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13321960, 7160980, 0 committed:false) SnapCreateDur 21.927µs SnapOpenDur 79.22363ms 2015-08-03T20:49:22.04Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7800000 2015-08-03T20:49:22.361Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7270000 QueuedCount 2044 2015-08-03T20:49:22.488Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13344460, 7172230, 0 committed:false) SnapCreateDur 19.687µs SnapOpenDur 81.703677ms 2015-08-03T20:49:23.084Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7810000 2015-08-03T20:49:23.379Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 16.186µs CommitTime 789.10523ms TotalFlushTime 8m21.785397904s TotalCommitTime 2m18.636587026s 2015-08-03T20:49:23.379Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13348256, 7174128, 217 committed:true) SnapCreateDur 816.763197ms SnapOpenDur 26.761µs 2015-08-03T20:49:23.614Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7280000 QueuedCount 5522 2015-08-03T20:49:23.902Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13370968, 7185484, 0 committed:false) SnapCreateDur 22.79µs SnapOpenDur 90.899779ms 2015-08-03T20:49:24.098Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7290000 QueuedCount 7028 2015-08-03T20:49:24.112Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7820000 2015-08-03T20:49:25.157Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7830000 2015-08-03T20:49:26.19Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7840000 2015-08-03T20:49:26.815Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13393620, 7196810, 0 committed:false) SnapCreateDur 21.893µs SnapOpenDur 141.461218ms 2015-08-03T20:49:27.24Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7850000 2015-08-03T20:49:27.799Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.43µs CommitTime 853.024025ms TotalFlushTime 8m24.976954176s TotalCommitTime 2m19.489611051s 2015-08-03T20:49:27.799Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13397434, 7198717, 218 committed:true) SnapCreateDur 879.094604ms SnapOpenDur 41.996µs 2015-08-03T20:49:27.857Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7300000 QueuedCount 2816 2015-08-03T20:49:28.204Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7310000 QueuedCount 527 2015-08-03T20:49:28.284Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7860000 2015-08-03T20:49:28.315Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13420164, 7210082, 0 committed:false) SnapCreateDur 24.513µs SnapOpenDur 84.27268ms 2015-08-03T20:49:29.332Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7870000 2015-08-03T20:49:29.641Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7320000 QueuedCount 2003 2015-08-03T20:49:29.773Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13442744, 7221372, 0 committed:false) SnapCreateDur 23.471µs SnapOpenDur 85.361158ms 2015-08-03T20:49:30.366Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7880000 2015-08-03T20:49:31.029Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.79µs CommitTime 978.418161ms TotalFlushTime 8m26.91945113s TotalCommitTime 2m20.468029212s 2015-08-03T20:49:31.029Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13454244, 7227122, 219 committed:true) SnapCreateDur 1.004611628s SnapOpenDur 39.106µs 2015-08-03T20:49:31.127Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7330000 QueuedCount 5561 2015-08-03T20:49:31.409Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7890000 2015-08-03T20:49:31.538Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13476958, 7238479, 0 committed:false) SnapCreateDur 25.02µs SnapOpenDur 77.948981ms 2015-08-03T20:49:31.588Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7340000 QueuedCount 1290 2015-08-03T20:49:32.45Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7900000 2015-08-03T20:49:32.783Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7350000 QueuedCount 1003 2015-08-03T20:49:32.946Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13499842, 7249921, 0 committed:false) SnapCreateDur 30.78µs SnapOpenDur 139.696555ms 2015-08-03T20:49:33.485Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7910000 2015-08-03T20:49:33.884Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.387µs CommitTime 815.992954ms TotalFlushTime 8m28.605639189s TotalCommitTime 2m21.284022166s 2015-08-03T20:49:33.884Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13503566, 7251783, 220 committed:true) SnapCreateDur 843.55109ms SnapOpenDur 30.653µs 2015-08-03T20:49:34.156Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7360000 QueuedCount 4387 2015-08-03T20:49:34.398Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13526196, 7263098, 0 committed:false) SnapCreateDur 35.677µs SnapOpenDur 82.007799ms 2015-08-03T20:49:34.531Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7920000 2015-08-03T20:49:35.564Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7930000 2015-08-03T20:49:35.6Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7370000 QueuedCount 5888 2015-08-03T20:49:35.868Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13548850, 7274425, 0 committed:false) SnapCreateDur 30.184µs SnapOpenDur 138.078758ms 2015-08-03T20:49:36.047Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7380000 QueuedCount 5449 2015-08-03T20:49:36.61Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7940000 2015-08-03T20:49:37.51Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.77µs CommitTime 1.22680481s TotalFlushTime 8m30.636485078s TotalCommitTime 2m22.510826976s 2015-08-03T20:49:37.66Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7950000 2015-08-03T20:49:38.703Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7960000 2015-08-03T20:49:39.157Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13567594, 7283797, 221 committed:true) SnapCreateDur 1.254063236s SnapOpenDur 1.646848042s 2015-08-03T20:49:39.545Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.04µs CommitTime 267.574837ms TotalFlushTime 8m30.700753805s TotalCommitTime 2m22.778401813s 2015-08-03T20:49:39.545Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13571388, 7285694, 222 committed:true) SnapCreateDur 293.489116ms SnapOpenDur 47.074µs 2015-08-03T20:49:39.655Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7390000 QueuedCount 5078 2015-08-03T20:49:39.741Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7970000 2015-08-03T20:49:40.046Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13594088, 7297044, 0 committed:false) SnapCreateDur 28.723µs SnapOpenDur 79.085644ms 2015-08-03T20:49:40.111Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7400000 QueuedCount 2754 2015-08-03T20:49:40.783Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7980000 2015-08-03T20:49:41.311Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7410000 QueuedCount 470 2015-08-03T20:49:41.466Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13616898, 7308449, 0 committed:false) SnapCreateDur 30.433µs SnapOpenDur 137.126691ms 2015-08-03T20:49:41.789Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7420000 QueuedCount 2003 2015-08-03T20:49:41.817Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7990000 2015-08-03T20:49:42.85Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13639596, 7319798, 0 committed:false) SnapCreateDur 20.357µs SnapOpenDur 113.576046ms 2015-08-03T20:49:42.859Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8000000 2015-08-03T20:49:43.511Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.283µs CommitTime 531.909513ms TotalFlushTime 8m33.646267216s TotalCommitTime 2m23.310311326s 2015-08-03T20:49:43.512Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13643370, 7321685, 223 committed:true) SnapCreateDur 558.154509ms SnapOpenDur 52.837µs 2015-08-03T20:49:43.759Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7430000 QueuedCount 5409 2015-08-03T20:49:43.902Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8010000 2015-08-03T20:49:44.039Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13665958, 7332979, 0 committed:false) SnapCreateDur 22.813µs SnapOpenDur 84.378123ms 2015-08-03T20:49:44.941Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8020000 2015-08-03T20:49:45.445Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7440000 QueuedCount 7055 2015-08-03T20:49:45.981Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8030000 2015-08-03T20:49:46.795Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13688864, 7344432, 0 committed:false) SnapCreateDur 42.016µs SnapOpenDur 144.483659ms 2015-08-03T20:49:47.027Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8040000 2015-08-03T20:49:47.741Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.876µs CommitTime 825.503901ms TotalFlushTime 8m36.684532435s TotalCommitTime 2m24.135815227s 2015-08-03T20:49:47.741Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13692576, 7346288, 224 committed:true) SnapCreateDur 851.28284ms SnapOpenDur 28.877µs 2015-08-03T20:49:47.785Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7450000 QueuedCount 847 2015-08-03T20:49:47.985Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7460000 QueuedCount 402 2015-08-03T20:49:48.066Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8050000 2015-08-03T20:49:48.084Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13715160, 7357580, 0 committed:false) SnapCreateDur 28.274µs SnapOpenDur 81.34388ms 2015-08-03T20:49:49.111Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8060000 2015-08-03T20:49:49.155Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7470000 QueuedCount 1873 2015-08-03T20:49:49.312Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13737720, 7368860, 0 committed:false) SnapCreateDur 10.907µs SnapOpenDur 87.12242ms 2015-08-03T20:49:49.511Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7480000 QueuedCount 3392 2015-08-03T20:49:50.156Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8070000 2015-08-03T20:49:50.362Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13760392, 7380196, 0 committed:false) SnapCreateDur 28.183µs SnapOpenDur 120.661051ms 2015-08-03T20:49:51.01Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.55µs CommitTime 557.218708ms TotalFlushTime 8m38.942685017s TotalCommitTime 2m24.693033935s 2015-08-03T20:49:51.011Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13764266, 7382133, 225 committed:true) SnapCreateDur 584.398878ms SnapOpenDur 43.237µs 2015-08-03T20:49:51.191Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8080000 2015-08-03T20:49:51.193Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7490000 QueuedCount 6844 2015-08-03T20:49:51.517Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13786876, 7393438, 0 committed:false) SnapCreateDur 21.663µs SnapOpenDur 85.834622ms 2015-08-03T20:49:51.671Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7500000 QueuedCount 8409 2015-08-03T20:49:52.236Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8090000 2015-08-03T20:49:53.191Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13809644, 7404822, 0 committed:false) SnapCreateDur 34.07µs SnapOpenDur 151.235019ms 2015-08-03T20:49:53.266Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8100000 2015-08-03T20:49:53.276Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7510000 QueuedCount 4182 2015-08-03T20:49:54.309Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8110000 2015-08-03T20:49:55.348Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8120000 2015-08-03T20:49:56.119Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13832348, 7416174, 0 committed:false) SnapCreateDur 23.063µs SnapOpenDur 131.031084ms 2015-08-03T20:49:56.144Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7520000 QueuedCount 1869 2015-08-03T20:49:56.389Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8130000 2015-08-03T20:49:56.803Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.47µs CommitTime 552.185ms TotalFlushTime 8m43.644501278s TotalCommitTime 2m25.245218935s 2015-08-03T20:49:57.441Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8140000 2015-08-03T20:49:58.487Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8150000 2015-08-03T20:49:58.649Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13836172, 7418086, 226 committed:true) SnapCreateDur 579.345947ms SnapOpenDur 1.846363621s 2015-08-03T20:49:58.968Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7530000 QueuedCount 3397 2015-08-03T20:49:59.175Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13858844, 7429422, 0 committed:false) SnapCreateDur 23.927µs SnapOpenDur 79.9221ms 2015-08-03T20:49:59.525Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8160000 2015-08-03T20:50:00.302Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.303µs CommitTime 1.009211592s TotalFlushTime 8m44.117841579s TotalCommitTime 2m26.254430527s 2015-08-03T20:50:00.302Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13862644, 7431322, 227 committed:true) SnapCreateDur 1.033348868s SnapOpenDur 29.298µs 2015-08-03T20:50:00.431Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7540000 QueuedCount 4954 2015-08-03T20:50:00.565Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8170000 2015-08-03T20:50:00.655Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13885374, 7442687, 0 committed:false) SnapCreateDur 26.89µs SnapOpenDur 81.235927ms 2015-08-03T20:50:00.801Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7550000 QueuedCount 8306 2015-08-03T20:50:01.607Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8180000 2015-08-03T20:50:02.637Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8190000 2015-08-03T20:50:03.192Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 2015-08-03T20:50:03.472Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13907912, 7453956, 0 committed:false) SnapCreateDur 30.85µs SnapOpenDur 47.163526ms 2015-08-03T20:50:03.557Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7560000 QueuedCount 213 2015-08-03T20:50:03.683Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8200000 2015-08-03T20:50:04.752Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8210000 2015-08-03T20:50:05.217Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":17388604705,"bucket-1:by_email:avg_scan_wait_latency":17387662982,"bucket-1:by_email:avg_ts_interval":1813936562,"bucket-1:by_email:avg_ts_items_count":9968,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":535723844,"bucket-1:by_email:delete_bytes":277250713,"bucket-1:by_email:disk_size":3194904576,"bucket-1:by_email:flush_queue_size":0,"bucket-1:by_email:frag_percent":83,"bucket-1:by_email:get_bytes":283077471,"bucket-1:by_email:insert_bytes":782007555,"bucket-1:by_email:items_count":1000001,"bucket-1:by_email:num_commits":227,"bucket-1:by_email:num_compactions":11,"bucket-1:by_email:num_completed_requests":15363,"bucket-1:by_email:num_docs_indexed":7559242,"bucket-1:by_email:num_docs_pending":239,"bucket-1:by_email:num_docs_queued":631766,"bucket-1:by_email:num_flush_queued":7560213,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":15747,"bucket-1:by_email:num_rows_returned":255,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":910,"bucket-1:by_email:scan_bytes_read":10200,"bucket-1:by_email:scan_wait_duration":273803528986106,"bucket-1:by_email:since_last_snapshot":2817740824,"bucket-1:by_email:timings/storage_clone_handle":"15363 884848100 2744996868313508","bucket-1:by_email:timings/storage_commit":"227 146254430527 584994694859136563","bucket-1:by_email:timings/storage_del":"6455834 106716538262 8045678664679714536","bucket-1:by_email:timings/storage_get":"7560213 107133710268 1336137679377887850","bucket-1:by_email:timings/storage_handle_open":"227 14443422360 3001051262681081328","bucket-1:by_email:timings/storage_iterator_next":"255 5210189 152601680809","bucket-1:by_email:timings/storage_new_iterator":"15363 1279019752 2856787411365804","bucket-1:by_email:timings/storage_set":"14911668 303572297607 -6425054191164743389","bucket-1:by_email:timings/storage_snapshot_close":"682 11069336831 840497968187718821","bucket-1:by_email:timings/storage_snapshot_create":"683 64847450815 -8494865186643367557","bucket-1:by_email:total_scan_duration":273818358292342,"bucket-1:mutation_queue_size":654186,"bucket-1:num_mutations_queued":8214399,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":340,"memory_quota":104857600000,"memory_used":2380263424,"needs_restart":false,"num_connections":384,"timings/stats_response":"1357 473997680 4765338122416932"} 2015-08-03T20:50:05.268Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.259µs CommitTime 471.473206ms TotalFlushTime 8m47.157535336s TotalCommitTime 2m26.725903733s 2015-08-03T20:50:05.268Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13911668, 7455834, 228 committed:true) SnapCreateDur 1.701113626s SnapOpenDur 25.84µs 2015-08-03T20:50:05.451Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7570000 QueuedCount 1718 2015-08-03T20:50:05.513Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13934274, 7467137, 0 committed:false) SnapCreateDur 11.377µs SnapOpenDur 33.956255ms 2015-08-03T20:50:05.839Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8220000 2015-08-03T20:50:06.744Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7580000 QueuedCount 3298 2015-08-03T20:50:06.889Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8230000 2015-08-03T20:50:06.948Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13957054, 7478527, 0 committed:false) SnapCreateDur 26.01µs SnapOpenDur 134.666027ms 2015-08-03T20:50:07.927Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8µs CommitTime 843.298116ms TotalFlushTime 8m48.67390225s TotalCommitTime 2m27.569201849s 2015-08-03T20:50:07.927Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13960816, 7480408, 229 committed:true) SnapCreateDur 873.975564ms SnapOpenDur 39.803µs 2015-08-03T20:50:07.937Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8240000 2015-08-03T20:50:08.127Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7590000 QueuedCount 6751 2015-08-03T20:50:08.459Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 13983506, 7491753, 0 committed:false) SnapCreateDur 29.014µs SnapOpenDur 87.774005ms 2015-08-03T20:50:08.615Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7600000 QueuedCount 8242 2015-08-03T20:50:08.977Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8250000 2015-08-03T20:50:09.952Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14006134, 7503067, 0 committed:false) SnapCreateDur 23.024µs SnapOpenDur 146.566313ms 2015-08-03T20:50:10.014Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8260000 2015-08-03T20:50:10.015Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7610000 QueuedCount 210 2015-08-03T20:50:10.544Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 2.204µs CommitTime 505.688797ms TotalFlushTime 8m50.425699423s TotalCommitTime 2m28.074890646s 2015-08-03T20:50:10.545Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14010004, 7505002, 230 committed:true) SnapCreateDur 517.625417ms SnapOpenDur 36.626µs 2015-08-03T20:50:10.931Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7620000 QueuedCount 1791 2015-08-03T20:50:11.058Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8270000 2015-08-03T20:50:11.086Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14032792, 7516396, 0 committed:false) SnapCreateDur 46.584µs SnapOpenDur 88.206215ms 2015-08-03T20:50:12.099Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8280000 2015-08-03T20:50:12.306Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7630000 QueuedCount 3117 2015-08-03T20:50:12.519Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14055102, 7527551, 0 committed:false) SnapCreateDur 43.695µs SnapOpenDur 148.698223ms 2015-08-03T20:50:12.831Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7640000 QueuedCount 4267 2015-08-03T20:50:13.139Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8290000 2015-08-03T20:50:14.164Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14077046, 7538523, 0 committed:false) SnapCreateDur 22.92µs SnapOpenDur 122.38606ms 2015-08-03T20:50:14.171Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8300000 2015-08-03T20:50:14.79Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.877µs CommitTime 502.99372ms TotalFlushTime 8m53.642021966s TotalCommitTime 2m28.577884366s 2015-08-03T20:50:14.79Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14080684, 7540342, 231 committed:true) SnapCreateDur 530.784019ms SnapOpenDur 61.232µs 2015-08-03T20:50:14.957Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7650000 QueuedCount 7550 2015-08-03T20:50:15.217Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8310000 2015-08-03T20:50:15.325Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14103180, 7551590, 0 committed:false) SnapCreateDur 26.224µs SnapOpenDur 91.329725ms 2015-08-03T20:50:15.449Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7660000 QueuedCount 7102 2015-08-03T20:50:16.26Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8320000 2015-08-03T20:50:16.885Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14125774, 7562887, 0 committed:false) SnapCreateDur 33.917µs SnapOpenDur 155.322802ms 2015-08-03T20:50:16.947Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7670000 QueuedCount 2883 2015-08-03T20:50:17.297Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8330000 2015-08-03T20:50:18.233Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7680000 QueuedCount 555 2015-08-03T20:50:18.322Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14148484, 7574242, 0 committed:false) SnapCreateDur 31.68µs SnapOpenDur 72.71528ms 2015-08-03T20:50:18.337Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8340000 2015-08-03T20:50:18.971Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 3.76µs CommitTime 554.063101ms TotalFlushTime 8m56.772092235s TotalCommitTime 2m29.131947467s 2015-08-03T20:50:18.971Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14152276, 7576138, 232 committed:true) SnapCreateDur 585.036947ms SnapOpenDur 63.064µs 2015-08-03T20:50:19.266Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7690000 QueuedCount 3998 2015-08-03T20:50:19.382Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8350000 2015-08-03T20:50:19.5Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14174956, 7587478, 0 committed:false) SnapCreateDur 28.23µs SnapOpenDur 83.223403ms 2015-08-03T20:50:20.423Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8360000 2015-08-03T20:50:20.745Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7700000 QueuedCount 5581 2015-08-03T20:50:20.942Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14197746, 7598873, 0 committed:false) SnapCreateDur 21.827µs SnapOpenDur 86.156754ms 2015-08-03T20:50:21.137Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7710000 QueuedCount 7154 2015-08-03T20:50:21.462Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8370000 2015-08-03T20:50:22.397Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14220516, 7610258, 0 committed:false) SnapCreateDur 22.46µs SnapOpenDur 124.042107ms 2015-08-03T20:50:22.509Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8380000 2015-08-03T20:50:23.084Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 17.551µs CommitTime 566.007301ms TotalFlushTime 8m59.857429867s TotalCommitTime 2m29.697954768s 2015-08-03T20:50:23.085Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14224200, 7612100, 233 committed:true) SnapCreateDur 592.601705ms SnapOpenDur 63.89µs 2015-08-03T20:50:23.144Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7720000 QueuedCount 2925 2015-08-03T20:50:23.513Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7730000 QueuedCount 569 2015-08-03T20:50:23.54Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8390000 2015-08-03T20:50:23.629Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14246942, 7623471, 0 committed:false) SnapCreateDur 28.377µs SnapOpenDur 86.32089ms 2015-08-03T20:50:24.587Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8400000 2015-08-03T20:50:24.829Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7740000 QueuedCount 2157 2015-08-03T20:50:25.02Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14269742, 7634871, 0 committed:false) SnapCreateDur 29.497µs SnapOpenDur 147.563308ms 2015-08-03T20:50:25.339Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7750000 QueuedCount 3524 2015-08-03T20:50:25.616Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8410000 2015-08-03T20:50:26.42Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14292090, 7646045, 0 committed:false) SnapCreateDur 21.823µs SnapOpenDur 117.415072ms 2015-08-03T20:50:26.662Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8420000 2015-08-03T20:50:27.017Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 15.974µs CommitTime 473.749633ms TotalFlushTime 9m2.793661035s TotalCommitTime 2m30.171704401s 2015-08-03T20:50:27.017Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14295852, 7647926, 234 committed:true) SnapCreateDur 502.036659ms SnapOpenDur 44.12µs 2015-08-03T20:50:27.206Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7760000 QueuedCount 6858 2015-08-03T20:50:27.531Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14318342, 7659171, 0 committed:false) SnapCreateDur 34.24µs SnapOpenDur 79.955641ms 2015-08-03T20:50:27.686Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7770000 QueuedCount 8476 2015-08-03T20:50:27.712Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8430000 2015-08-03T20:50:28.758Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8440000 2015-08-03T20:50:28.947Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14341220, 7670610, 0 committed:false) SnapCreateDur 19.455µs SnapOpenDur 82.039808ms 2015-08-03T20:50:29.001Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7780000 QueuedCount 2327 2015-08-03T20:50:29.791Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8450000 2015-08-03T20:50:29.89Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7790000 QueuedCount 33 2015-08-03T20:50:29.968Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14363950, 7681975, 0 committed:false) SnapCreateDur 10.981µs SnapOpenDur 69.471039ms 2015-08-03T20:50:30.311Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 5.052µs CommitTime 235.555115ms TotalFlushTime 9m5.465109935s TotalCommitTime 2m30.407259516s 2015-08-03T20:50:30.311Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14367768, 7683884, 235 committed:true) SnapCreateDur 248.821004ms SnapOpenDur 33.517µs 2015-08-03T20:50:30.482Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7800000 QueuedCount 3547 2015-08-03T20:50:30.593Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14390554, 7695277, 0 committed:false) SnapCreateDur 17.084µs SnapOpenDur 40.646219ms 2015-08-03T20:50:30.837Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8460000 2015-08-03T20:50:31.297Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7810000 QueuedCount 5047 2015-08-03T20:50:31.461Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14413204, 7706602, 0 committed:false) SnapCreateDur 18.836µs SnapOpenDur 76.458655ms 2015-08-03T20:50:31.704Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7820000 QueuedCount 6486 2015-08-03T20:50:31.881Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8470000 2015-08-03T20:50:32.845Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14435738, 7717869, 0 committed:false) SnapCreateDur 12.259µs SnapOpenDur 71.168925ms 2015-08-03T20:50:32.913Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8480000 2015-08-03T20:50:33.003Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7830000 QueuedCount 8188 2015-08-03T20:50:33.193Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:457414508, Disk:2095767552, ExtraSnapshotData:0, Fragmentation:78.1743682612374% 2015-08-03T20:50:33.193Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:50:33.219Z-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 14080684. Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T20:50:33.958Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8490000 2015-08-03T20:50:34.244Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14458778, 7729389, 0 committed:false) SnapCreateDur 35.02µs SnapOpenDur 17.53903ms 2015-08-03T20:50:34.333Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7840000 QueuedCount 84 2015-08-03T20:50:34.917Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.217µs CommitTime 552.410171ms TotalFlushTime 9m9.115106028s TotalCommitTime 2m30.959669687s 2015-08-03T20:50:34.917Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14462498, 7731249, 236 committed:true) SnapCreateDur 578.530204ms SnapOpenDur 45.353µs 2015-08-03T20:50:35Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8500000 2015-08-03T20:50:35.293Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7850000 QueuedCount 1543 2015-08-03T20:50:35.428Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14485060, 7742530, 0 committed:false) SnapCreateDur 17.057µs SnapOpenDur 79.101439ms 2015-08-03T20:50:36.041Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8510000 2015-08-03T20:50:36.665Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7860000 QueuedCount 3012 2015-08-03T20:50:36.873Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14507710, 7753855, 0 committed:false) SnapCreateDur 21.55µs SnapOpenDur 133.976349ms 2015-08-03T20:50:37.035Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7870000 QueuedCount 4560 2015-08-03T20:50:37.105Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8520000 2015-08-03T20:50:37.917Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14530430, 7765215, 0 committed:false) SnapCreateDur 41.973µs SnapOpenDur 112.244414ms 2015-08-03T20:50:38.14Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8530000 2015-08-03T20:50:38.589Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 15.742µs CommitTime 551.043521ms TotalFlushTime 9m11.747557062s TotalCommitTime 2m31.510713208s 2015-08-03T20:50:38.589Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14534264, 7767132, 237 committed:true) SnapCreateDur 577.776015ms SnapOpenDur 38.413µs 2015-08-03T20:50:38.739Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7880000 QueuedCount 8038 2015-08-03T20:50:39.1Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14556956, 7778478, 0 committed:false) SnapCreateDur 21.086µs SnapOpenDur 77.931207ms 2015-08-03T20:50:39.186Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8540000 2015-08-03T20:50:39.188Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7890000 QueuedCount 3847 2015-08-03T20:50:40.227Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8550000 2015-08-03T20:50:40.445Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14579672, 7789836, 0 committed:false) SnapCreateDur 37.864µs SnapOpenDur 141.581717ms 2015-08-03T20:50:40.483Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7900000 QueuedCount 1466 2015-08-03T20:50:41.262Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8560000 2015-08-03T20:50:41.691Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7910000 QueuedCount 1162 2015-08-03T20:50:41.834Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14602492, 7801246, 0 committed:false) SnapCreateDur 28.993µs SnapOpenDur 108.857452ms 2015-08-03T20:50:42.301Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8570000 2015-08-03T20:50:42.484Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.194µs CommitTime 530.32404ms TotalFlushTime 9m14.602072368s TotalCommitTime 2m32.041037248s 2015-08-03T20:50:42.484Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14606188, 7803094, 238 committed:true) SnapCreateDur 556.434322ms SnapOpenDur 42.354µs 2015-08-03T20:50:42.759Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7920000 QueuedCount 4595 2015-08-03T20:50:43.004Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14628926, 7814463, 0 committed:false) SnapCreateDur 23.024µs SnapOpenDur 76.454751ms 2015-08-03T20:50:43.339Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8580000 2015-08-03T20:50:44.189Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7930000 QueuedCount 6104 2015-08-03T20:50:44.378Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8590000 2015-08-03T20:50:44.398Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14651596, 7825798, 0 committed:false) SnapCreateDur 15.348µs SnapOpenDur 82.162671ms 2015-08-03T20:50:44.502Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7940000 QueuedCount 3686 2015-08-03T20:50:45.411Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8600000 2015-08-03T20:50:45.426Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14674008, 7837004, 0 committed:false) SnapCreateDur 23.35µs SnapOpenDur 97.884591ms 2015-08-03T20:50:45.803Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 5.645µs CommitTime 299.595801ms TotalFlushTime 9m17.21997111s TotalCommitTime 2m32.340633049s 2015-08-03T20:50:45.803Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14677816, 7838908, 239 committed:true) SnapCreateDur 312.751345ms SnapOpenDur 43.556µs 2015-08-03T20:50:45.843Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7950000 QueuedCount 1294 2015-08-03T20:50:46.188Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7960000 QueuedCount 903 2015-08-03T20:50:46.308Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14700362, 7850181, 0 committed:false) SnapCreateDur 30.437µs SnapOpenDur 84.377961ms 2015-08-03T20:50:46.461Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8610000 2015-08-03T20:50:47.506Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8620000 2015-08-03T20:50:47.627Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7970000 QueuedCount 2512 2015-08-03T20:50:47.849Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14723202, 7861601, 0 committed:false) SnapCreateDur 23.6µs SnapOpenDur 138.604913ms 2015-08-03T20:50:48.157Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7980000 QueuedCount 4074 2015-08-03T20:50:48.553Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8630000 2015-08-03T20:50:49.326Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14745992, 7872996, 0 committed:false) SnapCreateDur 20.983µs SnapOpenDur 115.919292ms 2015-08-03T20:50:49.588Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8640000 2015-08-03T20:50:50.014Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.517µs CommitTime 565.83662ms TotalFlushTime 9m20.356231833s TotalCommitTime 2m32.906469669s 2015-08-03T20:50:50.551Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14749750, 7874875, 240 committed:true) SnapCreateDur 594.117853ms SnapOpenDur 536.202217ms 2015-08-03T20:50:50.634Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8650000 2015-08-03T20:50:50.731Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 7990000 QueuedCount 7491 2015-08-03T20:50:51.09Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14772376, 7886188, 0 committed:false) SnapCreateDur 32.717µs SnapOpenDur 84.768347ms 2015-08-03T20:50:51.211Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8000000 QueuedCount 7132 2015-08-03T20:50:51.676Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8660000 2015-08-03T20:50:52.458Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14795098, 7897549, 0 committed:false) SnapCreateDur 13.928µs SnapOpenDur 85.620707ms 2015-08-03T20:50:52.499Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8010000 QueuedCount 962 2015-08-03T20:50:52.711Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8670000 2015-08-03T20:50:53.391Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.084µs CommitTime 841.635013ms TotalFlushTime 9m22.053530727s TotalCommitTime 2m33.748104682s 2015-08-03T20:50:53.756Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8680000 2015-08-03T20:50:53.826Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14798914, 7899457, 241 committed:true) SnapCreateDur 868.476173ms SnapOpenDur 435.724845ms 2015-08-03T20:50:54.614Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8020000 QueuedCount 2497 2015-08-03T20:50:54.797Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14821616, 7910808, 0 committed:false) SnapCreateDur 28.53µs SnapOpenDur 86.17679ms 2015-08-03T20:50:54.799Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8690000 2015-08-03T20:50:55.837Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8700000 2015-08-03T20:50:56.096Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8030000 QueuedCount 3990 2015-08-03T20:50:56.279Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14844234, 7922117, 0 committed:false) SnapCreateDur 23.33µs SnapOpenDur 89.628504ms 2015-08-03T20:50:56.89Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8710000 2015-08-03T20:50:57.312Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 13.78µs CommitTime 894.464971ms TotalFlushTime 9m24.336986702s TotalCommitTime 2m34.642569653s 2015-08-03T20:50:57.313Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14848122, 7924061, 242 committed:true) SnapCreateDur 920.137284ms SnapOpenDur 27.712µs 2015-08-03T20:50:57.477Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8040000 QueuedCount 7420 2015-08-03T20:50:57.836Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14870688, 7935344, 0 committed:false) SnapCreateDur 34.343µs SnapOpenDur 91.239629ms 2015-08-03T20:50:57.935Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8720000 2015-08-03T20:50:57.961Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8050000 QueuedCount 7029 2015-08-03T20:50:58.972Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8730000 2015-08-03T20:50:59.309Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14893374, 7946687, 0 committed:false) SnapCreateDur 11.837µs SnapOpenDur 87.216726ms 2015-08-03T20:50:59.372Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8060000 QueuedCount 870 2015-08-03T20:51:00.013Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8740000 2015-08-03T20:51:00.317Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.04µs CommitTime 878.078845ms TotalFlushTime 9m26.149275948s TotalCommitTime 2m35.520648498s 2015-08-03T20:51:00.317Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14897162, 7948581, 243 committed:true) SnapCreateDur 904.100705ms SnapOpenDur 35.076µs 2015-08-03T20:51:00.538Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8070000 QueuedCount 2403 2015-08-03T20:51:00.679Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14919838, 7959919, 0 committed:false) SnapCreateDur 22.11µs SnapOpenDur 80.542618ms 2015-08-03T20:51:01.058Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8750000 2015-08-03T20:51:01.816Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8080000 QueuedCount 3953 2015-08-03T20:51:01.973Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14942544, 7971272, 0 committed:false) SnapCreateDur 22.47µs SnapOpenDur 81.606724ms 2015-08-03T20:51:02.097Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8760000 2015-08-03T20:51:03.136Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8770000 2015-08-03T20:51:03.699Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8090000 QueuedCount 5495 2015-08-03T20:51:04.175Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8780000 2015-08-03T20:51:04.826Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14965294, 7982647, 0 committed:false) SnapCreateDur 15.214µs SnapOpenDur 72.072439ms 2015-08-03T20:51:05.218Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":18612864629,"bucket-1:by_email:avg_scan_wait_latency":18611924197,"bucket-1:by_email:avg_ts_interval":1930576018,"bucket-1:by_email:avg_ts_items_count":10854,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":503560342,"bucket-1:by_email:delete_bytes":299770931,"bucket-1:by_email:disk_size":2742915072,"bucket-1:by_email:flush_queue_size":0,"bucket-1:by_email:frag_percent":81,"bucket-1:by_email:get_bytes":305993504,"bucket-1:by_email:insert_bytes":836998785,"bucket-1:by_email:items_count":1000001,"bucket-1:by_email:num_commits":243,"bucket-1:by_email:num_compactions":12,"bucket-1:by_email:num_completed_requests":15747,"bucket-1:by_email:num_docs_indexed":8096471,"bucket-1:by_email:num_docs_pending":57,"bucket-1:by_email:num_docs_queued":675688,"bucket-1:by_email:num_flush_queued":8097445,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":16131,"bucket-1:by_email:num_rows_returned":266,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":970,"bucket-1:by_email:scan_bytes_read":10640,"bucket-1:by_email:scan_wait_duration":300228949228467,"bucket-1:by_email:since_last_snapshot":2853059685,"bucket-1:by_email:timings/storage_clone_handle":"15747 903726984 2746894935751994","bucket-1:by_email:timings/storage_commit":"243 155520648498 6513850153162468098","bucket-1:by_email:timings/storage_del":"6984559 121385422612 195311016108037174","bucket-1:by_email:timings/storage_get":"8097445 114221798582 1338952635314597994","bucket-1:by_email:timings/storage_handle_open":"243 15415882852 3478409543878850648","bucket-1:by_email:timings/storage_iterator_next":"266 5499339 160536457919","bucket-1:by_email:timings/storage_new_iterator":"15747 1306061050 2859194667549068","bucket-1:by_email:timings/storage_set":"15969118 324307133218 8081711459692850318","bucket-1:by_email:timings/storage_snapshot_close":"726 11805404604 854922528046841206","bucket-1:by_email:timings/storage_snapshot_create":"727 69018798397 -8059278751512932457","bucket-1:by_email:total_scan_duration":300244119344335,"bucket-1:mutation_queue_size":692215,"bucket-1:num_mutations_queued":8789660,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":360,"memory_quota":104857600000,"memory_used":1473605632,"needs_restart":false,"num_connections":384,"timings/stats_response":"1429 500027534 4775137019663236"} 2015-08-03T20:51:05.22Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.101µs CommitTime 317.978919ms TotalFlushTime 9m30.355496059s TotalCommitTime 2m35.838627417s 2015-08-03T20:51:05.221Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14969118, 7984559, 244 committed:true) SnapCreateDur 330.637794ms SnapOpenDur 24.175µs 2015-08-03T20:51:05.256Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8790000 2015-08-03T20:51:05.303Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8100000 QueuedCount 3218 2015-08-03T20:51:05.541Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 14991910, 7995955, 0 committed:false) SnapCreateDur 26.41µs SnapOpenDur 78.771482ms 2015-08-03T20:51:05.599Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8110000 QueuedCount 2813 2015-08-03T20:51:06.412Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8800000 2015-08-03T20:51:07.006Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8120000 QueuedCount 550 2015-08-03T20:51:07.178Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15014542, 8007271, 0 committed:false) SnapCreateDur 29.783µs SnapOpenDur 154.712843ms 2015-08-03T20:51:07.521Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8810000 2015-08-03T20:51:07.612Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8130000 QueuedCount 1960 2015-08-03T20:51:08.511Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8820000 2015-08-03T20:51:08.72Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15037038, 8018519, 0 committed:false) SnapCreateDur 12.224µs SnapOpenDur 79.536954ms 2015-08-03T20:51:09.362Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.776µs CommitTime 521.185659ms TotalFlushTime 9m33.499151553s TotalCommitTime 2m36.359813076s 2015-08-03T20:51:09.362Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15040750, 8020375, 245 committed:true) SnapCreateDur 547.432169ms SnapOpenDur 57.54µs 2015-08-03T20:51:09.559Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8830000 2015-08-03T20:51:09.594Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8140000 QueuedCount 5350 2015-08-03T20:51:09.87Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15063352, 8031676, 0 committed:false) SnapCreateDur 22.12µs SnapOpenDur 87.453901ms 2015-08-03T20:51:10.069Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8150000 QueuedCount 6882 2015-08-03T20:51:10.601Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8840000 2015-08-03T20:51:11.635Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8850000 2015-08-03T20:51:12.677Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8860000 2015-08-03T20:51:12.781Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15086100, 8043050, 0 committed:false) SnapCreateDur 25.823µs SnapOpenDur 149.288678ms 2015-08-03T20:51:13.714Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8870000 2015-08-03T20:51:13.781Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.66µs CommitTime 879.634596ms TotalFlushTime 9m36.674099408s TotalCommitTime 2m37.239447672s 2015-08-03T20:51:13.782Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15089894, 8044947, 246 committed:true) SnapCreateDur 905.878498ms SnapOpenDur 34.703µs 2015-08-03T20:51:13.83Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8160000 QueuedCount 2660 2015-08-03T20:51:14.037Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8170000 QueuedCount 354 2015-08-03T20:51:14.133Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15112626, 8056313, 0 committed:false) SnapCreateDur 23.3µs SnapOpenDur 78.901264ms 2015-08-03T20:51:14.76Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8880000 2015-08-03T20:51:15.339Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8180000 QueuedCount 1916 2015-08-03T20:51:15.531Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15135374, 8067687, 0 committed:false) SnapCreateDur 28.364µs SnapOpenDur 144.63955ms 2015-08-03T20:51:15.798Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8890000 2015-08-03T20:51:15.844Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8190000 QueuedCount 3458 2015-08-03T20:51:16.853Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8900000 2015-08-03T20:51:16.935Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15158100, 8079050, 0 committed:false) SnapCreateDur 26.164µs SnapOpenDur 131.236894ms 2015-08-03T20:51:17.662Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.25µs CommitTime 597.236509ms TotalFlushTime 9m39.431098015s TotalCommitTime 2m37.836684181s 2015-08-03T20:51:17.663Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15161854, 8080927, 247 committed:true) SnapCreateDur 623.347655ms SnapOpenDur 37.404µs 2015-08-03T20:51:17.856Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8200000 QueuedCount 6802 2015-08-03T20:51:17.897Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8910000 2015-08-03T20:51:18.177Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15184362, 8092181, 0 committed:false) SnapCreateDur 19.866µs SnapOpenDur 80.836849ms 2015-08-03T20:51:18.316Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8210000 QueuedCount 6157 2015-08-03T20:51:18.939Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8920000 2015-08-03T20:51:19.983Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8930000 2015-08-03T20:51:21.016Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8940000 2015-08-03T20:51:21.043Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15205952, 8102976, 0 committed:false) SnapCreateDur 32.786µs SnapOpenDur 144.407806ms 2015-08-03T20:51:21.925Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.254µs CommitTime 751.459709ms TotalFlushTime 9m42.580767584s TotalCommitTime 2m38.58814389s 2015-08-03T20:51:21.925Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15209780, 8104890, 248 committed:true) SnapCreateDur 778.105503ms SnapOpenDur 30.11µs 2015-08-03T20:51:21.953Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8220000 QueuedCount 1536 2015-08-03T20:51:22.06Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8950000 2015-08-03T20:51:22.17Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8230000 QueuedCount 1100 2015-08-03T20:51:22.245Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15232120, 8116060, 0 committed:false) SnapCreateDur 21.221µs SnapOpenDur 48.408998ms 2015-08-03T20:51:23.097Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8960000 2015-08-03T20:51:23.535Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8240000 QueuedCount 2578 2015-08-03T20:51:24.138Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8970000 2015-08-03T20:51:24.786Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 2015-08-03T20:51:24.869Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15254746, 8127373, 0 committed:false) SnapCreateDur 34.226µs SnapOpenDur 11.928689ms 2015-08-03T20:51:25.18Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8980000 2015-08-03T20:51:26.234Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8990000 2015-08-03T20:51:26.519Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.633µs CommitTime 291.813279ms TotalFlushTime 9m45.460307655s TotalCommitTime 2m38.879957169s 2015-08-03T20:51:26.519Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15258640, 8129320, 249 committed:true) SnapCreateDur 1.555813623s SnapOpenDur 31.184µs 2015-08-03T20:51:26.73Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8250000 QueuedCount 6012 2015-08-03T20:51:27.007Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15281128, 8140564, 0 committed:false) SnapCreateDur 21.01µs SnapOpenDur 56.080073ms 2015-08-03T20:51:27.188Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8260000 QueuedCount 7548 2015-08-03T20:51:27.282Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9000000 2015-08-03T20:51:28.316Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9010000 2015-08-03T20:51:28.376Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15303844, 8151922, 0 committed:false) SnapCreateDur 18.149µs SnapOpenDur 85.788409ms 2015-08-03T20:51:29.344Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.196µs CommitTime 878.287047ms TotalFlushTime 9m47.137406323s TotalCommitTime 2m39.758244216s 2015-08-03T20:51:29.344Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15307628, 8153814, 250 committed:true) SnapCreateDur 904.489959ms SnapOpenDur 43.107µs 2015-08-03T20:51:29.358Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9020000 2015-08-03T20:51:29.377Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8270000 QueuedCount 1396 2015-08-03T20:51:29.599Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8280000 QueuedCount 1020 2015-08-03T20:51:29.709Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15330368, 8165184, 0 committed:false) SnapCreateDur 23.684µs SnapOpenDur 81.214183ms 2015-08-03T20:51:30.391Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9030000 2015-08-03T20:51:30.985Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8290000 QueuedCount 2607 2015-08-03T20:51:31.195Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15353066, 8176533, 0 committed:false) SnapCreateDur 26.267µs SnapOpenDur 152.579081ms 2015-08-03T20:51:31.435Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9040000 2015-08-03T20:51:31.507Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8300000 QueuedCount 4147 2015-08-03T20:51:32.475Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9050000 2015-08-03T20:51:32.483Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15375756, 8187878, 0 committed:false) SnapCreateDur 12.474µs SnapOpenDur 73.208209ms 2015-08-03T20:51:33.211Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.977µs CommitTime 607.489221ms TotalFlushTime 9m49.918519353s TotalCommitTime 2m40.365733437s 2015-08-03T20:51:33.211Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15379516, 8189758, 251 committed:true) SnapCreateDur 634.531843ms SnapOpenDur 44.15µs 2015-08-03T20:51:33.328Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8310000 QueuedCount 3716 2015-08-03T20:51:33.511Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9060000 2015-08-03T20:51:33.577Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15402066, 8201033, 0 committed:false) SnapCreateDur 35.97µs SnapOpenDur 79.868561ms 2015-08-03T20:51:33.665Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8320000 QueuedCount 3295 2015-08-03T20:51:34.552Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9070000 2015-08-03T20:51:34.665Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15424788, 8212394, 0 committed:false) SnapCreateDur 12.546µs SnapOpenDur 83.865285ms 2015-08-03T20:51:34.725Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8330000 QueuedCount 2808 2015-08-03T20:51:35.586Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9080000 2015-08-03T20:51:35.878Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8340000 QueuedCount 570 2015-08-03T20:51:36.015Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15447462, 8223731, 0 committed:false) SnapCreateDur 21.25µs SnapOpenDur 118.446031ms 2015-08-03T20:51:36.638Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9090000 2015-08-03T20:51:36.698Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 21.32µs CommitTime 550.954429ms TotalFlushTime 9m52.406247324s TotalCommitTime 2m40.916687866s 2015-08-03T20:51:36.698Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15451356, 8225678, 252 committed:true) SnapCreateDur 579.035726ms SnapOpenDur 51.424µs 2015-08-03T20:51:36.991Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8350000 QueuedCount 4045 2015-08-03T20:51:37.242Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15473968, 8236984, 0 committed:false) SnapCreateDur 23.86µs SnapOpenDur 100.73806ms 2015-08-03T20:51:37.69Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9100000 2015-08-03T20:51:38.464Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8360000 QueuedCount 5570 2015-08-03T20:51:38.733Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15496662, 8248331, 0 committed:false) SnapCreateDur 28.13µs SnapOpenDur 155.413032ms 2015-08-03T20:51:38.737Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9110000 2015-08-03T20:51:38.933Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8370000 QueuedCount 7116 2015-08-03T20:51:39.775Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9120000 2015-08-03T20:51:40.154Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15519384, 8259692, 0 committed:false) SnapCreateDur 24.386µs SnapOpenDur 78.3005ms 2015-08-03T20:51:40.815Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9130000 2015-08-03T20:51:40.878Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.923µs CommitTime 598.909082ms TotalFlushTime 9m55.477155643s TotalCommitTime 2m41.515596948s 2015-08-03T20:51:40.878Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15523184, 8261592, 253 committed:true) SnapCreateDur 629.33869ms SnapOpenDur 39.849µs 2015-08-03T20:51:40.935Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8380000 QueuedCount 2914 2015-08-03T20:51:41.301Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8390000 QueuedCount 549 2015-08-03T20:51:41.411Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15545832, 8272916, 0 committed:false) SnapCreateDur 29.387µs SnapOpenDur 87.977181ms 2015-08-03T20:51:41.855Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9140000 2015-08-03T20:51:42.733Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8400000 QueuedCount 2131 2015-08-03T20:51:42.891Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9150000 2015-08-03T20:51:42.929Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15568590, 8284295, 0 committed:false) SnapCreateDur 22.777µs SnapOpenDur 151.068176ms 2015-08-03T20:51:43.121Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8410000 QueuedCount 3657 2015-08-03T20:51:43.95Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9160000 2015-08-03T20:51:44.304Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15591254, 8295627, 0 committed:false) SnapCreateDur 18.978µs SnapOpenDur 121.996932ms 2015-08-03T20:51:44.973Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9170000 2015-08-03T20:51:45.059Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.39µs CommitTime 622.750681ms TotalFlushTime 9m58.507827201s TotalCommitTime 2m42.138347629s 2015-08-03T20:51:45.059Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15595062, 8297531, 254 committed:true) SnapCreateDur 649.796732ms SnapOpenDur 44.537µs 2015-08-03T20:51:45.246Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8420000 QueuedCount 7031 2015-08-03T20:51:45.595Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15617566, 8308783, 0 committed:false) SnapCreateDur 35.78µs SnapOpenDur 90.602391ms 2015-08-03T20:51:45.738Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8430000 QueuedCount 6875 2015-08-03T20:51:46.008Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9180000 2015-08-03T20:51:47.022Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15640178, 8320089, 0 committed:false) SnapCreateDur 31.507µs SnapOpenDur 150.832872ms 2015-08-03T20:51:47.065Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9190000 2015-08-03T20:51:47.107Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8440000 QueuedCount 4308 2015-08-03T20:51:48.112Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9200000 2015-08-03T20:51:48.601Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8450000 QueuedCount 72 2015-08-03T20:51:48.736Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15662950, 8331475, 0 committed:false) SnapCreateDur 31.044µs SnapOpenDur 128.866291ms 2015-08-03T20:51:49.152Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9210000 2015-08-03T20:51:49.194Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.067µs CommitTime 368.090327ms TotalFlushTime 10m1.740749565s TotalCommitTime 2m42.506437956s 2015-08-03T20:51:49.194Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15666728, 8333364, 255 committed:true) SnapCreateDur 384.056583ms SnapOpenDur 38.646µs 2015-08-03T20:51:49.506Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8460000 QueuedCount 3555 2015-08-03T20:51:49.722Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15689530, 8344765, 0 committed:false) SnapCreateDur 28.733µs SnapOpenDur 82.738382ms 2015-08-03T20:51:50.19Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9220000 2015-08-03T20:51:50.609Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8470000 QueuedCount 5049 2015-08-03T20:51:50.856Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15712098, 8356049, 0 committed:false) SnapCreateDur 21.93µs SnapOpenDur 151.968043ms 2015-08-03T20:51:51.081Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8480000 QueuedCount 6614 2015-08-03T20:51:51.233Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9230000 2015-08-03T20:51:52.262Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15734912, 8367456, 0 committed:false) SnapCreateDur 22.557µs SnapOpenDur 132.652964ms 2015-08-03T20:51:52.276Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9240000 2015-08-03T20:51:53.002Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.35µs CommitTime 609.085799ms TotalFlushTime 10m4.403660901s TotalCommitTime 2m43.115523755s 2015-08-03T20:51:53.002Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15738686, 8369343, 256 committed:true) SnapCreateDur 635.403802ms SnapOpenDur 44.51µs 2015-08-03T20:51:53.084Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8490000 QueuedCount 4259 2015-08-03T20:51:53.311Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9250000 2015-08-03T20:51:53.454Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8500000 QueuedCount 13 2015-08-03T20:51:53.551Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15761282, 8380641, 0 committed:false) SnapCreateDur 30.343µs SnapOpenDur 93.000018ms 2015-08-03T20:51:54.352Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9260000 2015-08-03T20:51:54.786Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:460146460, Disk:2096304128, ExtraSnapshotData:0, Fragmentation:78.04963250065212% 2015-08-03T20:51:54.786Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:51:54.808Z-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 15451356. Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T20:51:55.387Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9270000 2015-08-03T20:51:55.487Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8510000 QueuedCount 1499 2015-08-03T20:51:55.564Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15783894, 8391947, 0 committed:false) SnapCreateDur 28.504µs SnapOpenDur 16.397624ms 2015-08-03T20:51:56.266Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 16.944µs CommitTime 570.181269ms TotalFlushTime 10m6.857770761s TotalCommitTime 2m43.685705024s 2015-08-03T20:51:56.266Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15787650, 8393825, 257 committed:true) SnapCreateDur 596.963198ms SnapOpenDur 40.589µs 2015-08-03T20:51:56.438Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8520000 QueuedCount 4770 2015-08-03T20:51:56.441Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9280000 2015-08-03T20:51:56.588Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15810026, 8405013, 0 committed:false) SnapCreateDur 18.985µs SnapOpenDur 47.421621ms 2015-08-03T20:51:57.419Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8530000 QueuedCount 6341 2015-08-03T20:51:57.486Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9290000 2015-08-03T20:51:57.674Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15832788, 8416394, 0 committed:false) SnapCreateDur 27.233µs SnapOpenDur 135.050479ms 2015-08-03T20:51:57.835Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8540000 QueuedCount 7901 2015-08-03T20:51:58.534Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9300000 2015-08-03T20:51:58.968Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15855572, 8427786, 0 committed:false) SnapCreateDur 28.19µs SnapOpenDur 112.090501ms 2015-08-03T20:51:59.586Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9310000 2015-08-03T20:51:59.648Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.76µs CommitTime 551.188015ms TotalFlushTime 10m9.233750725s TotalCommitTime 2m44.236893039s 2015-08-03T20:51:59.649Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15859372, 8429686, 258 committed:true) SnapCreateDur 576.784201ms SnapOpenDur 86.343µs 2015-08-03T20:51:59.675Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8550000 QueuedCount 1756 2015-08-03T20:52:00.005Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8560000 QueuedCount 1366 2015-08-03T20:52:00.137Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15882034, 8441017, 0 committed:false) SnapCreateDur 24.856µs SnapOpenDur 75.50772ms 2015-08-03T20:52:00.619Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9320000 2015-08-03T20:52:01.273Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8570000 QueuedCount 2922 2015-08-03T20:52:01.459Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15904766, 8452383, 0 committed:false) SnapCreateDur 22.767µs SnapOpenDur 133.760435ms 2015-08-03T20:52:01.662Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9330000 2015-08-03T20:52:01.735Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8580000 QueuedCount 4481 2015-08-03T20:52:02.709Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9340000 2015-08-03T20:52:02.79Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15927502, 8463751, 0 committed:false) SnapCreateDur 11.493µs SnapOpenDur 69.989281ms 2015-08-03T20:52:03.505Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.967µs CommitTime 594.934103ms TotalFlushTime 10m12.057400772s TotalCommitTime 2m44.831827142s 2015-08-03T20:52:03.505Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15931298, 8465649, 259 committed:true) SnapCreateDur 621.336389ms SnapOpenDur 47.587µs 2015-08-03T20:52:03.662Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8590000 QueuedCount 7969 2015-08-03T20:52:03.736Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9350000 2015-08-03T20:52:04.001Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15954046, 8477023, 0 committed:false) SnapCreateDur 22.013µs SnapOpenDur 71.712715ms 2015-08-03T20:52:04.094Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8600000 QueuedCount 5276 2015-08-03T20:52:04.81Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9360000 2015-08-03T20:52:05.218Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":20032780984,"bucket-1:by_email:avg_scan_wait_latency":20031845041,"bucket-1:by_email:avg_ts_interval":718371620,"bucket-1:by_email:avg_ts_items_count":8989,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":474851456,"bucket-1:by_email:delete_bytes":321079925,"bucket-1:by_email:disk_size":2355519488,"bucket-1:by_email:flush_queue_size":1690,"bucket-1:by_email:frag_percent":79,"bucket-1:by_email:get_bytes":327684467,"bucket-1:by_email:insert_bytes":889032270,"bucket-1:by_email:items_count":1000001,"bucket-1:by_email:num_commits":259,"bucket-1:by_email:num_compactions":13,"bucket-1:by_email:num_completed_requests":16131,"bucket-1:by_email:num_docs_indexed":8595126,"bucket-1:by_email:num_docs_pending":315,"bucket-1:by_email:num_docs_queued":756097,"bucket-1:by_email:num_flush_queued":8609497,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":16515,"bucket-1:by_email:num_rows_returned":273,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":1027,"bucket-1:by_email:scan_bytes_read":10920,"bucket-1:by_email:scan_wait_duration":330825920852854,"bucket-1:by_email:since_last_snapshot":495571268,"bucket-1:by_email:timings/storage_clone_handle":"16131 921662173 2747839815592365","bucket-1:by_email:timings/storage_commit":"259 164831827142 -6100023615990853756","bucket-1:by_email:timings/storage_del":"7486724 128317646030 4474383365268886208","bucket-1:by_email:timings/storage_get":"8607809 123650126332 4681293693673612506","bucket-1:by_email:timings/storage_handle_open":"259 15416497796 3478409569838213780","bucket-1:by_email:timings/storage_iterator_next":"273 5686658 165729263178","bucket-1:by_email:timings/storage_new_iterator":"16131 1333526407 2861622321295965","bucket-1:by_email:timings/storage_set":"16973446 350552166667 9127847787327809177","bucket-1:by_email:timings/storage_snapshot_close":"767 12506322479 869044868281978351","bucket-1:by_email:timings/storage_snapshot_create":"768 73127823476 -7588915373526272198","bucket-1:by_email:total_scan_duration":330841377966644,"bucket-1:mutation_queue_size":754448,"bucket-1:num_mutations_queued":9363945,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":393,"memory_quota":104857600000,"memory_used":865087488,"needs_restart":false,"num_connections":384,"timings/stats_response":"1501 525743193 4784619887603275"} 2015-08-03T20:52:05.331Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15976774, 8488387, 0 committed:false) SnapCreateDur 14.237µs SnapOpenDur 78.961542ms 2015-08-03T20:52:05.372Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8610000 QueuedCount 1408 2015-08-03T20:52:05.937Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9370000 2015-08-03T20:52:06.636Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8620000 QueuedCount 937 2015-08-03T20:52:06.781Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 15999302, 8499651, 0 committed:false) SnapCreateDur 28.947µs SnapOpenDur 117.973596ms 2015-08-03T20:52:06.99Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9380000 2015-08-03T20:52:07.455Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.757µs CommitTime 582.527506ms TotalFlushTime 10m14.990051149s TotalCommitTime 2m45.414354648s 2015-08-03T20:52:07.455Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16003126, 8501563, 260 committed:true) SnapCreateDur 610.577285ms SnapOpenDur 47.533µs 2015-08-03T20:52:07.719Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8630000 QueuedCount 4325 2015-08-03T20:52:07.953Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16025698, 8512849, 0 committed:false) SnapCreateDur 23.433µs SnapOpenDur 75.114698ms 2015-08-03T20:52:08.032Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9390000 2015-08-03T20:52:09.065Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9400000 2015-08-03T20:52:09.13Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8640000 QueuedCount 5840 2015-08-03T20:52:09.393Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16048338, 8524169, 0 committed:false) SnapCreateDur 20.89µs SnapOpenDur 137.756641ms 2015-08-03T20:52:09.504Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8650000 QueuedCount 3562 2015-08-03T20:52:10.107Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9410000 2015-08-03T20:52:10.445Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16071092, 8535546, 0 committed:false) SnapCreateDur 20.82µs SnapOpenDur 120.171957ms 2015-08-03T20:52:11.147Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9420000 2015-08-03T20:52:11.54Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.807µs CommitTime 595.51112ms TotalFlushTime 10m17.976315456s TotalCommitTime 2m46.009865768s 2015-08-03T20:52:11.54Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16074866, 8537433, 261 committed:true) SnapCreateDur 621.275099ms SnapOpenDur 39.624µs 2015-08-03T20:52:11.584Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8660000 QueuedCount 1241 2015-08-03T20:52:11.925Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8670000 QueuedCount 866 2015-08-03T20:52:12.042Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16097590, 8548795, 0 committed:false) SnapCreateDur 24.987µs SnapOpenDur 79.252752ms 2015-08-03T20:52:12.187Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9430000 2015-08-03T20:52:13.229Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9440000 2015-08-03T20:52:13.256Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8680000 QueuedCount 2382 2015-08-03T20:52:13.391Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16120238, 8560119, 0 committed:false) SnapCreateDur 30.864µs SnapOpenDur 82.934981ms 2015-08-03T20:52:13.554Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8690000 QueuedCount 49 2015-08-03T20:52:14.264Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9450000 2015-08-03T20:52:14.296Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.343µs CommitTime 729.127665ms TotalFlushTime 10m19.725939055s TotalCommitTime 2m46.738993433s 2015-08-03T20:52:14.296Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16135316, 8567658, 262 committed:true) SnapCreateDur 740.397323ms SnapOpenDur 28.647µs 2015-08-03T20:52:14.486Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8700000 QueuedCount 1601 2015-08-03T20:52:14.59Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16158108, 8579054, 0 committed:false) SnapCreateDur 21.73µs SnapOpenDur 72.767437ms 2015-08-03T20:52:15.311Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9460000 2015-08-03T20:52:15.34Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8710000 QueuedCount 2972 2015-08-03T20:52:15.529Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16180504, 8590252, 0 committed:false) SnapCreateDur 31.06µs SnapOpenDur 138.24766ms 2015-08-03T20:52:15.824Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8720000 QueuedCount 4541 2015-08-03T20:52:16.341Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9470000 2015-08-03T20:52:16.903Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16203276, 8601638, 0 committed:false) SnapCreateDur 26.454µs SnapOpenDur 112.633244ms 2015-08-03T20:52:17.271Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 1.786µs CommitTime 290.892212ms TotalFlushTime 10m21.944569702s TotalCommitTime 2m47.029885645s 2015-08-03T20:52:17.271Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16206918, 8603459, 263 committed:true) SnapCreateDur 303.69507ms SnapOpenDur 42.655µs 2015-08-03T20:52:17.395Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9480000 2015-08-03T20:52:17.417Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8730000 QueuedCount 7967 2015-08-03T20:52:17.734Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16229678, 8614839, 0 committed:false) SnapCreateDur 19.903µs SnapOpenDur 50.695842ms 2015-08-03T20:52:17.831Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8740000 QueuedCount 5650 2015-08-03T20:52:18.441Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9490000 2015-08-03T20:52:19.484Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9500000 2015-08-03T20:52:20.522Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9510000 2015-08-03T20:52:21.152Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16252352, 8626176, 0 committed:false) SnapCreateDur 28.434µs SnapOpenDur 764.680015ms 2015-08-03T20:52:21.195Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8750000 QueuedCount 1389 2015-08-03T20:52:21.563Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9520000 2015-08-03T20:52:22.012Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.41µs CommitTime 739.452958ms TotalFlushTime 10m25.005930342s TotalCommitTime 2m47.769338603s 2015-08-03T20:52:22.012Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16256060, 8628030, 264 committed:true) SnapCreateDur 765.393749ms SnapOpenDur 33.822µs 2015-08-03T20:52:22.327Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8760000 QueuedCount 2987 2015-08-03T20:52:22.492Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16278888, 8639444, 0 committed:false) SnapCreateDur 23.564µs SnapOpenDur 56.285053ms 2015-08-03T20:52:22.603Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9530000 2015-08-03T20:52:23.652Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9540000 2015-08-03T20:52:23.653Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8770000 QueuedCount 4489 2015-08-03T20:52:23.83Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16301494, 8650747, 0 committed:false) SnapCreateDur 13.845µs SnapOpenDur 85.575153ms 2015-08-03T20:52:24.052Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8780000 QueuedCount 5655 2015-08-03T20:52:24.687Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9550000 2015-08-03T20:52:25.28Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16323486, 8661743, 0 committed:false) SnapCreateDur 18.752µs SnapOpenDur 78.616301ms 2015-08-03T20:52:25.725Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9560000 2015-08-03T20:52:25.973Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.55µs CommitTime 602.878144ms TotalFlushTime 10m27.982347772s TotalCommitTime 2m48.372216747s 2015-08-03T20:52:25.973Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16327244, 8663622, 265 committed:true) SnapCreateDur 628.91497ms SnapOpenDur 48.59µs 2015-08-03T20:52:26.075Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8790000 QueuedCount 4203 2015-08-03T20:52:26.41Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16347774, 8673887, 0 committed:false) SnapCreateDur 21.397µs SnapOpenDur 65.927584ms 2015-08-03T20:52:26.505Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8800000 QueuedCount 3714 2015-08-03T20:52:26.789Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9570000 2015-08-03T20:52:27.815Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9580000 2015-08-03T20:52:27.871Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16370278, 8685139, 0 committed:false) SnapCreateDur 21.25µs SnapOpenDur 132.20433ms 2015-08-03T20:52:27.904Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8810000 QueuedCount 1264 2015-08-03T20:52:28.781Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8820000 QueuedCount 841 2015-08-03T20:52:28.857Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9590000 2015-08-03T20:52:28.923Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16392718, 8696359, 0 committed:false) SnapCreateDur 22.267µs SnapOpenDur 121.058665ms 2015-08-03T20:52:29.215Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 3.955µs CommitTime 226.848282ms TotalFlushTime 10m30.546329671s TotalCommitTime 2m48.599065029s 2015-08-03T20:52:29.896Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9600000 2015-08-03T20:52:30.23Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16396510, 8698255, 266 committed:true) SnapCreateDur 237.484277ms SnapOpenDur 1.014980148s 2015-08-03T20:52:30.515Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8830000 QueuedCount 4302 2015-08-03T20:52:30.739Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16419218, 8709609, 0 committed:false) SnapCreateDur 21.917µs SnapOpenDur 76.303727ms 2015-08-03T20:52:30.938Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9610000 2015-08-03T20:52:31.943Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8840000 QueuedCount 5823 2015-08-03T20:52:31.987Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9620000 2015-08-03T20:52:32.208Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16441866, 8720933, 0 committed:false) SnapCreateDur 33.766µs SnapOpenDur 146.3953ms 2015-08-03T20:52:33.014Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9630000 2015-08-03T20:52:33.259Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 13.043µs CommitTime 931.216275ms TotalFlushTime 10m32.298420355s TotalCommitTime 2m49.530281304s 2015-08-03T20:52:33.259Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16445676, 8722838, 267 committed:true) SnapCreateDur 956.863109ms SnapOpenDur 34.712µs 2015-08-03T20:52:33.357Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8850000 QueuedCount 5436 2015-08-03T20:52:33.749Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16468288, 8734144, 0 committed:false) SnapCreateDur 22.103µs SnapOpenDur 77.215248ms 2015-08-03T20:52:33.798Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8860000 QueuedCount 3147 2015-08-03T20:52:34.057Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9640000 2015-08-03T20:52:35.09Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9650000 2015-08-03T20:52:36.139Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9660000 2015-08-03T20:52:37.191Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9670000 2015-08-03T20:52:37.626Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 2015-08-03T20:52:37.718Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8870000 QueuedCount 801 2015-08-03T20:52:37.778Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16491026, 8745513, 0 committed:false) SnapCreateDur 25.907µs SnapOpenDur 24.740885ms 2015-08-03T20:52:38.241Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9680000 2015-08-03T20:52:39.281Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9690000 2015-08-03T20:52:39.553Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.114µs CommitTime 368.753303ms TotalFlushTime 10m36.69732623s TotalCommitTime 2m49.899034607s 2015-08-03T20:52:39.553Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16494702, 8747351, 268 committed:true) SnapCreateDur 1.680784216s SnapOpenDur 37.602µs 2015-08-03T20:52:39.718Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8880000 QueuedCount 4257 2015-08-03T20:52:39.853Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16517504, 8758752, 0 committed:false) SnapCreateDur 16.296µs SnapOpenDur 48.066717ms 2015-08-03T20:52:40.316Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9700000 2015-08-03T20:52:40.746Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8890000 QueuedCount 5734 2015-08-03T20:52:41.021Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16540080, 8770040, 0 committed:false) SnapCreateDur 28.07µs SnapOpenDur 154.832056ms 2015-08-03T20:52:41.362Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9710000 2015-08-03T20:52:41.958Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.574µs CommitTime 806.287372ms TotalFlushTime 10m37.969524907s TotalCommitTime 2m50.705321979s 2015-08-03T20:52:41.958Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16543872, 8771936, 269 committed:true) SnapCreateDur 832.705759ms SnapOpenDur 40.961µs 2015-08-03T20:52:42.032Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8900000 QueuedCount 3283 2015-08-03T20:52:42.319Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16566330, 8783165, 0 committed:false) SnapCreateDur 33.186µs SnapOpenDur 78.457692ms 2015-08-03T20:52:42.378Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8910000 QueuedCount 2935 2015-08-03T20:52:42.389Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9720000 2015-08-03T20:52:43.436Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9730000 2015-08-03T20:52:43.626Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8920000 QueuedCount 625 2015-08-03T20:52:43.733Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16589072, 8794536, 0 committed:false) SnapCreateDur 19.898µs SnapOpenDur 90.282184ms 2015-08-03T20:52:44.111Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8930000 QueuedCount 2205 2015-08-03T20:52:44.466Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9740000 2015-08-03T20:52:45.204Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16611868, 8805934, 0 committed:false) SnapCreateDur 35.52µs SnapOpenDur 130.154872ms 2015-08-03T20:52:45.511Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9750000 2015-08-03T20:52:45.665Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.178µs CommitTime 369.799514ms TotalFlushTime 10m40.849174623s TotalCommitTime 2m51.075121493s 2015-08-03T20:52:45.665Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16615580, 8807790, 270 committed:true) SnapCreateDur 385.84926ms SnapOpenDur 40.402µs 2015-08-03T20:52:45.901Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8940000 QueuedCount 5595 2015-08-03T20:52:46.194Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16638266, 8819133, 0 committed:false) SnapCreateDur 25.31µs SnapOpenDur 85.818933ms 2015-08-03T20:52:46.398Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8950000 QueuedCount 7183 2015-08-03T20:52:46.556Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9760000 2015-08-03T20:52:47.59Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16661120, 8830560, 0 committed:false) SnapCreateDur 22.014µs SnapOpenDur 150.77059ms 2015-08-03T20:52:47.61Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9770000 2015-08-03T20:52:47.675Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8960000 QueuedCount 2919 2015-08-03T20:52:48.638Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9780000 2015-08-03T20:52:48.65Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16683754, 8841877, 0 committed:false) SnapCreateDur 12.16µs SnapOpenDur 76.302835ms 2015-08-03T20:52:48.722Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8970000 QueuedCount 615 2015-08-03T20:52:49.406Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 14.25µs CommitTime 634.309594ms TotalFlushTime 10m43.480268827s TotalCommitTime 2m51.709431087s 2015-08-03T20:52:49.407Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16687588, 8843794, 271 committed:true) SnapCreateDur 661.602962ms SnapOpenDur 59.533µs 2015-08-03T20:52:49.685Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9790000 2015-08-03T20:52:49.77Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8980000 QueuedCount 2145 2015-08-03T20:52:49.937Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16710290, 8855145, 0 committed:false) SnapCreateDur 25.317µs SnapOpenDur 85.354958ms 2015-08-03T20:52:50.723Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9800000 2015-08-03T20:52:51.211Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 8990000 QueuedCount 3545 2015-08-03T20:52:51.368Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16732670, 8866335, 0 committed:false) SnapCreateDur 23.86µs SnapOpenDur 85.252999ms 2015-08-03T20:52:51.641Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9000000 QueuedCount 5005 2015-08-03T20:52:51.765Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9810000 2015-08-03T20:52:52.807Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9820000 2015-08-03T20:52:52.834Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16755248, 8877624, 0 committed:false) SnapCreateDur 22.577µs SnapOpenDur 114.140379ms 2015-08-03T20:52:53.55Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 14.284µs CommitTime 584.402135ms TotalFlushTime 10m46.58108007s TotalCommitTime 2m52.293833222s 2015-08-03T20:52:53.55Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16759144, 8879572, 272 committed:true) SnapCreateDur 610.884736ms SnapOpenDur 51.813µs 2015-08-03T20:52:53.684Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9010000 QueuedCount 6553 2015-08-03T20:52:53.841Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9830000 2015-08-03T20:52:54.089Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16781790, 8890895, 0 committed:false) SnapCreateDur 29.92µs SnapOpenDur 92.126831ms 2015-08-03T20:52:54.178Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9020000 QueuedCount 4266 2015-08-03T20:52:54.882Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9840000 2015-08-03T20:52:55.363Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9030000 QueuedCount 68 2015-08-03T20:52:55.528Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16804622, 8902311, 0 committed:false) SnapCreateDur 30.576µs SnapOpenDur 154.586898ms 2015-08-03T20:52:55.924Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9850000 2015-08-03T20:52:56.791Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9040000 QueuedCount 1470 2015-08-03T20:52:56.958Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16827036, 8913518, 0 committed:false) SnapCreateDur 19.22µs SnapOpenDur 136.084934ms 2015-08-03T20:52:56.965Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9860000 2015-08-03T20:52:57.405Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.96µs CommitTime 357.982044ms TotalFlushTime 10m49.533177927s TotalCommitTime 2m52.651815266s 2015-08-03T20:52:57.405Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16831112, 8915556, 273 committed:true) SnapCreateDur 371.46269ms SnapOpenDur 53.805µs 2015-08-03T20:52:57.57Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9050000 QueuedCount 5085 2015-08-03T20:52:57.738Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16853848, 8926924, 0 committed:false) SnapCreateDur 23.258µs SnapOpenDur 47.599808ms 2015-08-03T20:52:58.011Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9870000 2015-08-03T20:52:58.517Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9060000 QueuedCount 6645 2015-08-03T20:52:58.793Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16876540, 8938270, 0 committed:false) SnapCreateDur 24.786µs SnapOpenDur 151.462401ms 2015-08-03T20:52:58.886Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9070000 QueuedCount 2820 2015-08-03T20:52:59.057Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9880000 2015-08-03T20:52:59.655Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16899282, 8949641, 0 committed:false) SnapCreateDur 28.671µs SnapOpenDur 68.844883ms 2015-08-03T20:52:59.745Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9080000 QueuedCount 3990 2015-08-03T20:53:00.101Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9890000 2015-08-03T20:53:00.284Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16921888, 8960944, 0 committed:false) SnapCreateDur 17.55µs SnapOpenDur 196.749334ms 2015-08-03T20:53:00.318Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9090000 QueuedCount 1582 2015-08-03T20:53:01.136Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9900000 2015-08-03T20:53:01.789Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.207µs CommitTime 319.85261ms TotalFlushTime 10m52.927170829s TotalCommitTime 2m52.971667876s 2015-08-03T20:53:01.789Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16925608, 8962804, 274 committed:true) SnapCreateDur 350.553164ms SnapOpenDur 68.267µs 2015-08-03T20:53:02.123Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9100000 QueuedCount 2988 2015-08-03T20:53:02.174Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9910000 2015-08-03T20:53:02.321Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16948084, 8974042, 0 committed:false) SnapCreateDur 40.89µs SnapOpenDur 85.305142ms 2015-08-03T20:53:03.217Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9920000 2015-08-03T20:53:03.643Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9110000 QueuedCount 4513 2015-08-03T20:53:03.882Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16970762, 8985381, 0 committed:false) SnapCreateDur 28.996µs SnapOpenDur 136.543551ms 2015-08-03T20:53:04.259Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9930000 2015-08-03T20:53:04.828Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.167µs CommitTime 815.494843ms TotalFlushTime 10m54.793836418s TotalCommitTime 2m53.787162719s 2015-08-03T20:53:04.828Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16974564, 8987282, 275 committed:true) SnapCreateDur 841.73333ms SnapOpenDur 34.169µs 2015-08-03T20:53:05.055Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9120000 QueuedCount 7961 2015-08-03T20:53:05.219Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":20032780984,"bucket-1:by_email:avg_scan_wait_latency":20031845041,"bucket-1:by_email:avg_ts_interval":1069652926,"bucket-1:by_email:avg_ts_items_count":6087,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":446374416,"bucket-1:by_email:delete_bytes":343453126,"bucket-1:by_email:disk_size":1882640384,"bucket-1:by_email:flush_queue_size":3552,"bucket-1:by_email:frag_percent":76,"bucket-1:by_email:get_bytes":350417535,"bucket-1:by_email:insert_bytes":943664610,"bucket-1:by_email:items_count":1000001,"bucket-1:by_email:num_commits":275,"bucket-1:by_email:num_compactions":13,"bucket-1:by_email:num_completed_requests":16131,"bucket-1:by_email:num_docs_indexed":9117504,"bucket-1:by_email:num_docs_pending":373,"bucket-1:by_email:num_docs_queued":814334,"bucket-1:by_email:num_flush_queued":9127961,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":16515,"bucket-1:by_email:num_rows_returned":273,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":1085,"bucket-1:by_email:scan_bytes_read":10920,"bucket-1:by_email:scan_wait_duration":330825920852854,"bucket-1:by_email:since_last_snapshot":945899237,"bucket-1:by_email:timings/storage_clone_handle":"16131 921662173 2747839815592365","bucket-1:by_email:timings/storage_commit":"275 173787162719 -398736571733412367","bucket-1:by_email:timings/storage_del":"7995137 137374294433 -8087732263879692445","bucket-1:by_email:timings/storage_get":"9124410 131860865594 6569071372375407602","bucket-1:by_email:timings/storage_handle_open":"275 16432076467 4508585577600801853","bucket-1:by_email:timings/storage_iterator_next":"273 5686658 165729263178","bucket-1:by_email:timings/storage_new_iterator":"16131 1333526407 2861622321295965","bucket-1:by_email:timings/storage_set":"17990274 374171731624 8200204945823021426","bucket-1:by_email:timings/storage_snapshot_close":"810 13264905291 885314753213302653","bucket-1:by_email:timings/storage_snapshot_create":"810 77991840533 -6538709338581781905","bucket-1:by_email:total_scan_duration":330841377966644,"bucket-1:mutation_queue_size":811205,"bucket-1:num_mutations_queued":9939166,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":422,"memory_quota":104857600000,"memory_used":603045888,"needs_restart":false,"num_connections":384,"timings/stats_response":"1573 550820502 4793869674268058"} 2015-08-03T20:53:05.306Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9940000 2015-08-03T20:53:05.444Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 16997278, 8998639, 0 committed:false) SnapCreateDur 20.393µs SnapOpenDur 80.98814ms 2015-08-03T20:53:05.556Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9130000 QueuedCount 5656 2015-08-03T20:53:06.432Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9950000 2015-08-03T20:53:06.979Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17020026, 9010013, 0 committed:false) SnapCreateDur 33.257µs SnapOpenDur 157.040813ms 2015-08-03T20:53:07.02Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9140000 QueuedCount 1461 2015-08-03T20:53:07.514Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9960000 2015-08-03T20:53:07.626Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:452181689, Disk:1952182272, ExtraSnapshotData:0, Fragmentation:76.83711733860065% 2015-08-03T20:53:07.626Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:53:07.627Z-07:00 [Error] ForestDB Error (data.fdb.0/default) errcode: -15, msg: doc_body checksum mismatch error in a database file '/data/@2i/bucket-1_by_email_16558345882944865108_0.index/data.fdb.13' 2015-08-03T20:53:07.627Z-07:00 [Info] ForestDBSlice::Compact No Snapshot Found. Skipped Compaction.Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T20:53:07.627Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 2015-08-03T20:53:07.969Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.62µs CommitTime 858.207933ms TotalFlushTime 10m56.702909152s TotalCommitTime 2m54.645370652s 2015-08-03T20:53:07.969Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17023864, 9011932, 276 committed:true) SnapCreateDur 886.111305ms SnapOpenDur 34.637µs 2015-08-03T20:53:08.32Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9150000 QueuedCount 2964 2015-08-03T20:53:08.532Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17046496, 9023248, 0 committed:false) SnapCreateDur 33.51µs SnapOpenDur 88.544277ms 2015-08-03T20:53:08.556Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9970000 2015-08-03T20:53:09.592Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9980000 2015-08-03T20:53:09.75Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9160000 QueuedCount 4508 2015-08-03T20:53:10.002Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17069212, 9034606, 0 committed:false) SnapCreateDur 27.714µs SnapOpenDur 166.062869ms 2015-08-03T20:53:10.637Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9990000 2015-08-03T20:53:10.82Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.67µs CommitTime 732.091748ms TotalFlushTime 10m58.453357498s TotalCommitTime 2m55.3774624s 2015-08-03T20:53:10.82Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17073012, 9036506, 277 committed:true) SnapCreateDur 743.737562ms SnapOpenDur 34.179µs 2015-08-03T20:53:10.974Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9170000 QueuedCount 7975 2015-08-03T20:53:11.347Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17095758, 9047879, 0 committed:false) SnapCreateDur 24.18µs SnapOpenDur 82.639159ms 2015-08-03T20:53:11.45Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9180000 QueuedCount 5583 2015-08-03T20:53:11.665Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10000000 2015-08-03T20:53:12.716Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10010000 2015-08-03T20:53:12.762Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17118298, 9059149, 0 committed:false) SnapCreateDur 15.527µs SnapOpenDur 90.754362ms 2015-08-03T20:53:12.807Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9190000 QueuedCount 2116 2015-08-03T20:53:13.747Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10020000 2015-08-03T20:53:14.122Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9200000 QueuedCount 890 2015-08-03T20:53:14.282Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17140906, 9070453, 0 committed:false) SnapCreateDur 30.15µs SnapOpenDur 138.493818ms 2015-08-03T20:53:14.785Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10030000 2015-08-03T20:53:15.029Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 17.107µs CommitTime 615.459398ms TotalFlushTime 11m1.560489694s TotalCommitTime 2m55.992921798s 2015-08-03T20:53:15.029Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17144666, 9072333, 278 committed:true) SnapCreateDur 642.167951ms SnapOpenDur 58.2µs 2015-08-03T20:53:15.353Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9210000 QueuedCount 4320 2015-08-03T20:53:15.629Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17167318, 9083659, 0 committed:false) SnapCreateDur 24.767µs SnapOpenDur 84.080206ms 2015-08-03T20:53:15.825Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10040000 2015-08-03T20:53:16.778Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9220000 QueuedCount 5902 2015-08-03T20:53:16.874Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10050000 2015-08-03T20:53:17.055Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17190068, 9095034, 0 committed:false) SnapCreateDur 27.657µs SnapOpenDur 159.338847ms 2015-08-03T20:53:17.173Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9230000 QueuedCount 4251 2015-08-03T20:53:17.932Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10060000 2015-08-03T20:53:18.197Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17212692, 9106346, 0 committed:false) SnapCreateDur 22.74µs SnapOpenDur 127.671062ms 2015-08-03T20:53:18.952Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.46µs CommitTime 652.783919ms TotalFlushTime 11m4.282522502s TotalCommitTime 2m56.645705717s 2015-08-03T20:53:18.952Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17216564, 9108282, 279 committed:true) SnapCreateDur 679.447015ms SnapOpenDur 47.283µs 2015-08-03T20:53:18.97Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10070000 2015-08-03T20:53:18.991Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9240000 QueuedCount 1248 2015-08-03T20:53:19.282Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9250000 QueuedCount 909 2015-08-03T20:53:19.353Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17239240, 9119620, 0 committed:false) SnapCreateDur 9.921µs SnapOpenDur 47.217748ms 2015-08-03T20:53:20.011Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10080000 2015-08-03T20:53:20.337Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9260000 QueuedCount 2450 2015-08-03T20:53:20.473Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17261968, 9130984, 0 committed:false) SnapCreateDur 11.551µs SnapOpenDur 88.097945ms 2015-08-03T20:53:20.653Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9270000 QueuedCount 3957 2015-08-03T20:53:21.049Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10090000 2015-08-03T20:53:21.573Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17284638, 9142319, 0 committed:false) SnapCreateDur 12.707µs SnapOpenDur 75.953989ms 2015-08-03T20:53:22.089Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10100000 2015-08-03T20:53:22.284Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.747µs CommitTime 621.326137ms TotalFlushTime 11m6.63285803s TotalCommitTime 2m57.267031854s 2015-08-03T20:53:22.284Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17288462, 9144231, 280 committed:true) SnapCreateDur 647.402456ms SnapOpenDur 31.856µs 2015-08-03T20:53:22.454Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9280000 QueuedCount 7294 2015-08-03T20:53:22.798Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17310904, 9155452, 0 committed:false) SnapCreateDur 38.48µs SnapOpenDur 79.836439ms 2015-08-03T20:53:22.936Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9290000 QueuedCount 6852 2015-08-03T20:53:23.136Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10110000 2015-08-03T20:53:24.178Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10120000 2015-08-03T20:53:24.323Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17333482, 9166741, 0 committed:false) SnapCreateDur 13.449µs SnapOpenDur 90.722103ms 2015-08-03T20:53:24.373Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9300000 QueuedCount 2646 2015-08-03T20:53:25.119Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9310000 QueuedCount 330 2015-08-03T20:53:25.215Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10130000 2015-08-03T20:53:25.243Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17356222, 9178111, 0 committed:false) SnapCreateDur 26.08µs SnapOpenDur 117.9006ms 2015-08-03T20:53:25.924Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.974µs CommitTime 590.32969ms TotalFlushTime 11m9.236972391s TotalCommitTime 2m57.857361544s 2015-08-03T20:53:25.924Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17360014, 9180007, 281 committed:true) SnapCreateDur 616.337396ms SnapOpenDur 46.412µs 2015-08-03T20:53:26.225Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9320000 QueuedCount 3682 2015-08-03T20:53:26.261Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10140000 2015-08-03T20:53:26.442Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17382542, 9191271, 0 committed:false) SnapCreateDur 25.52µs SnapOpenDur 84.685057ms 2015-08-03T20:53:27.323Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10150000 2015-08-03T20:53:27.707Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9330000 QueuedCount 5200 2015-08-03T20:53:27.973Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17405242, 9202621, 0 committed:false) SnapCreateDur 34.746µs SnapOpenDur 159.232ms 2015-08-03T20:53:28.19Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9340000 QueuedCount 6665 2015-08-03T20:53:28.362Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10160000 2015-08-03T20:53:29.406Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10170000 2015-08-03T20:53:29.539Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17427844, 9213922, 0 committed:false) SnapCreateDur 14.525µs SnapOpenDur 82.417212ms 2015-08-03T20:53:30.295Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.473µs CommitTime 624.058606ms TotalFlushTime 11m12.485885252s TotalCommitTime 2m58.48142015s 2015-08-03T20:53:30.295Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17431832, 9215916, 282 committed:true) SnapCreateDur 651.451958ms SnapOpenDur 28.34µs 2015-08-03T20:53:30.363Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9350000 QueuedCount 2507 2015-08-03T20:53:30.44Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10180000 2015-08-03T20:53:30.819Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17453952, 9226976, 0 committed:false) SnapCreateDur 26.167µs SnapOpenDur 80.581515ms 2015-08-03T20:53:30.843Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9360000 QueuedCount 1891 2015-08-03T20:53:31.475Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10190000 2015-08-03T20:53:32.225Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9370000 QueuedCount 628 2015-08-03T20:53:32.328Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17475038, 9237519, 0 committed:false) SnapCreateDur 19.781µs SnapOpenDur 86.115268ms 2015-08-03T20:53:32.524Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10200000 2015-08-03T20:53:32.716Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9380000 QueuedCount 2029 2015-08-03T20:53:33.556Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10210000 2015-08-03T20:53:33.88Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17497474, 9248737, 0 committed:false) SnapCreateDur 22.403µs SnapOpenDur 129.513091ms 2015-08-03T20:53:34.535Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.35µs CommitTime 563.577728ms TotalFlushTime 11m15.695371248s TotalCommitTime 2m59.044997878s 2015-08-03T20:53:34.536Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17501258, 9250629, 283 committed:true) SnapCreateDur 589.902714ms SnapOpenDur 45.536µs 2015-08-03T20:53:34.589Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10220000 2015-08-03T20:53:34.799Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9390000 QueuedCount 5482 2015-08-03T20:53:35.106Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17524012, 9262006, 0 committed:false) SnapCreateDur 29.403µs SnapOpenDur 85.98772ms 2015-08-03T20:53:35.637Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10230000 2015-08-03T20:53:36.307Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9400000 QueuedCount 7088 2015-08-03T20:53:36.634Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17546912, 9273456, 0 committed:false) SnapCreateDur 36.084µs SnapOpenDur 184.282878ms 2015-08-03T20:53:36.68Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10240000 2015-08-03T20:53:36.732Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9410000 QueuedCount 4739 2015-08-03T20:53:37.627Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:505689061, Disk:2557722624, ExtraSnapshotData:0, Fragmentation:80.22893271322918% 2015-08-03T20:53:37.627Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:53:37.714Z-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 17216564. Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T20:53:37.717Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10250000 2015-08-03T20:53:38.158Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17569674, 9284837, 0 committed:false) SnapCreateDur 42.047µs SnapOpenDur 9.843464ms 2015-08-03T20:53:38.212Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9420000 QueuedCount 561 2015-08-03T20:53:38.751Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.19µs CommitTime 491.046427ms TotalFlushTime 11m18.969585928s TotalCommitTime 2m59.536044305s 2015-08-03T20:53:38.751Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17573400, 9286700, 284 committed:true) SnapCreateDur 517.866ms SnapOpenDur 44.52µs 2015-08-03T20:53:38.761Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10260000 2015-08-03T20:53:39.114Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9430000 QueuedCount 2073 2015-08-03T20:53:39.268Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17596052, 9298026, 0 committed:false) SnapCreateDur 19.377µs SnapOpenDur 75.031035ms 2015-08-03T20:53:39.807Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10270000 2015-08-03T20:53:40.535Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9440000 QueuedCount 3605 2015-08-03T20:53:40.699Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17618700, 9309350, 0 committed:false) SnapCreateDur 10.679µs SnapOpenDur 88.335314ms 2015-08-03T20:53:40.846Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10280000 2015-08-03T20:53:40.967Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9450000 QueuedCount 5164 2015-08-03T20:53:41.88Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10290000 2015-08-03T20:53:42.248Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17641404, 9320702, 0 committed:false) SnapCreateDur 27.954µs SnapOpenDur 127.018273ms 2015-08-03T20:53:42.925Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10300000 2015-08-03T20:53:43.004Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.977µs CommitTime 634.022317ms TotalFlushTime 11m22.130022986s TotalCommitTime 3m0.170066622s 2015-08-03T20:53:43.005Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17645204, 9322602, 285 committed:true) SnapCreateDur 662.873344ms SnapOpenDur 49.571µs 2015-08-03T20:53:43.137Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9460000 QueuedCount 6712 2015-08-03T20:53:43.517Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17667934, 9333967, 0 committed:false) SnapCreateDur 18.173µs SnapOpenDur 78.016167ms 2015-08-03T20:53:43.666Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9470000 QueuedCount 4347 2015-08-03T20:53:43.961Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10310000 2015-08-03T20:53:44.821Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9480000 QueuedCount 39 2015-08-03T20:53:44.973Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17690404, 9345202, 0 committed:false) SnapCreateDur 26.04µs SnapOpenDur 146.571526ms 2015-08-03T20:53:45.01Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10320000 2015-08-03T20:53:46.039Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10330000 2015-08-03T20:53:46.449Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9490000 QueuedCount 1539 2015-08-03T20:53:46.612Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17713052, 9356526, 0 committed:false) SnapCreateDur 31.927µs SnapOpenDur 127.699514ms 2015-08-03T20:53:47.087Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10340000 2015-08-03T20:53:47.408Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.253µs CommitTime 670.493429ms TotalFlushTime 11m25.341817502s TotalCommitTime 3m0.840560051s 2015-08-03T20:53:47.408Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17716822, 9358411, 286 committed:true) SnapCreateDur 701.751557ms SnapOpenDur 49.884µs 2015-08-03T20:53:47.661Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9500000 QueuedCount 4993 2015-08-03T20:53:47.932Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17739546, 9369773, 0 committed:false) SnapCreateDur 21.266µs SnapOpenDur 81.919354ms 2015-08-03T20:53:48.136Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10350000 2015-08-03T20:53:49.176Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10360000 2015-08-03T20:53:49.401Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9510000 QueuedCount 6474 2015-08-03T20:53:49.783Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17762146, 9381073, 0 committed:false) SnapCreateDur 32.946µs SnapOpenDur 148.276327ms 2015-08-03T20:53:50.214Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10370000 2015-08-03T20:53:50.956Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.157µs CommitTime 1.049387381s TotalFlushTime 11m27.472368238s TotalCommitTime 3m1.889947432s 2015-08-03T20:53:50.956Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17766030, 9383015, 287 committed:true) SnapCreateDur 1.078573978s SnapOpenDur 60.17µs 2015-08-03T20:53:51.034Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9520000 QueuedCount 4224 2015-08-03T20:53:51.256Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10380000 2015-08-03T20:53:51.389Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9530000 QueuedCount 27 2015-08-03T20:53:51.471Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17788832, 9394416, 0 committed:false) SnapCreateDur 24.37µs SnapOpenDur 80.321463ms 2015-08-03T20:53:52.3Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10390000 2015-08-03T20:53:52.758Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9540000 QueuedCount 1568 2015-08-03T20:53:52.876Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17811564, 9405782, 0 committed:false) SnapCreateDur 17.224µs SnapOpenDur 81.045346ms 2015-08-03T20:53:53.334Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10400000 2015-08-03T20:53:53.925Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.31µs CommitTime 905.29485ms TotalFlushTime 11m29.239963301s TotalCommitTime 3m2.795242282s 2015-08-03T20:53:54.307Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17815304, 9407652, 288 committed:true) SnapCreateDur 935.591763ms SnapOpenDur 381.923224ms 2015-08-03T20:53:54.366Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10410000 2015-08-03T20:53:54.559Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9550000 QueuedCount 4969 2015-08-03T20:53:54.823Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17838010, 9419005, 0 committed:false) SnapCreateDur 21.327µs SnapOpenDur 83.702874ms 2015-08-03T20:53:55.412Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10420000 2015-08-03T20:53:56.458Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10430000 2015-08-03T20:53:56.951Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9560000 QueuedCount 6460 2015-08-03T20:53:57.397Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17860680, 9430340, 0 committed:false) SnapCreateDur 22.771µs SnapOpenDur 151.376231ms 2015-08-03T20:53:57.506Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10440000 2015-08-03T20:53:58.419Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.303µs CommitTime 891.362388ms TotalFlushTime 11m32.094834984s TotalCommitTime 3m3.68660467s 2015-08-03T20:53:58.554Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10450000 2015-08-03T20:53:59.321Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17864366, 9432183, 289 committed:true) SnapCreateDur 918.228615ms SnapOpenDur 901.269768ms 2015-08-03T20:53:59.406Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9570000 QueuedCount 4072 2015-08-03T20:53:59.588Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10460000 2015-08-03T20:53:59.859Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17886998, 9443499, 0 committed:false) SnapCreateDur 18.806µs SnapOpenDur 84.335612ms 2015-08-03T20:53:59.888Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9580000 QueuedCount 1760 2015-08-03T20:54:00.633Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10470000 2015-08-03T20:54:01.675Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10480000 2015-08-03T20:54:01.686Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9590000 QueuedCount 1362 2015-08-03T20:54:01.88Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17909706, 9454853, 0 committed:false) SnapCreateDur 23.374µs SnapOpenDur 157.881734ms 2015-08-03T20:54:02.715Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10490000 2015-08-03T20:54:02.897Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.99µs CommitTime 886.097974ms TotalFlushTime 11m34.411557311s TotalCommitTime 3m4.572702644s 2015-08-03T20:54:02.897Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17913524, 9456762, 290 committed:true) SnapCreateDur 912.386285ms SnapOpenDur 29.098µs 2015-08-03T20:54:03.094Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9600000 QueuedCount 4837 2015-08-03T20:54:03.351Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17936160, 9468080, 0 committed:false) SnapCreateDur 23.398µs SnapOpenDur 81.659643ms 2015-08-03T20:54:03.761Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10500000 2015-08-03T20:54:04.585Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9610000 QueuedCount 6450 2015-08-03T20:54:04.798Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10510000 2015-08-03T20:54:04.811Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17959008, 9479504, 0 committed:false) SnapCreateDur 9.28µs SnapOpenDur 80.634342ms 2015-08-03T20:54:04.966Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9620000 QueuedCount 3973 2015-08-03T20:54:05.219Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":21539013205,"bucket-1:by_email:avg_scan_wait_latency":21538081184,"bucket-1:by_email:avg_ts_interval":1171123097,"bucket-1:by_email:avg_ts_items_count":10200,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":540600736,"bucket-1:by_email:delete_bytes":364544196,"bucket-1:by_email:disk_size":3135860736,"bucket-1:by_email:flush_queue_size":0,"bucket-1:by_email:frag_percent":82,"bucket-1:by_email:get_bytes":371875997,"bucket-1:by_email:insert_bytes":995166060,"bucket-1:by_email:items_count":1000001,"bucket-1:by_email:num_commits":290,"bucket-1:by_email:num_compactions":15,"bucket-1:by_email:num_completed_requests":16515,"bucket-1:by_email:num_docs_indexed":9617453,"bucket-1:by_email:num_docs_pending":187,"bucket-1:by_email:num_docs_queued":894623,"bucket-1:by_email:num_flush_queued":9623973,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":16899,"bucket-1:by_email:num_rows_returned":279,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":1141,"bucket-1:by_email:scan_bytes_read":11160,"bucket-1:by_email:scan_wait_duration":363972033944937,"bucket-1:by_email:since_last_snapshot":1460521331,"bucket-1:by_email:timings/storage_clone_handle":"16515 942167695 2749257031590261","bucket-1:by_email:timings/storage_commit":"290 184572702644 7713405535509302420","bucket-1:by_email:timings/storage_del":"8486917 145285219835 -3939252997051254567","bucket-1:by_email:timings/storage_get":"9623973 138796958281 6569190804966124583","bucket-1:by_email:timings/storage_handle_open":"290 17715765886 5466725328053113036","bucket-1:by_email:timings/storage_iterator_next":"279 5778900 167251171030","bucket-1:by_email:timings/storage_new_iterator":"16515 1365631589 2864937961101695","bucket-1:by_email:timings/storage_set":"18973834 399863916271 -8325302836294440971","bucket-1:by_email:timings/storage_snapshot_close":"850 13943918564 898982777454826046","bucket-1:by_email:timings/storage_snapshot_create":"851 82243432835 -6043109274318378263","bucket-1:by_email:total_scan_duration":363987784152274,"bucket-1:mutation_queue_size":889738,"bucket-1:num_mutations_queued":10513711,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":463,"memory_quota":104857600000,"memory_used":1329082368,"needs_restart":false,"num_connections":384,"timings/stats_response":"1645 575568625 4802735406654991"} 2015-08-03T20:54:05.888Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10520000 2015-08-03T20:54:06.402Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 1.956µs CommitTime 1.26451575s TotalFlushTime 11m36.367472776s TotalCommitTime 3m5.837218394s 2015-08-03T20:54:06.402Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17973834, 9486917, 291 committed:true) SnapCreateDur 1.277457787s SnapOpenDur 37.839µs 2015-08-03T20:54:06.64Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9630000 QueuedCount 5603 2015-08-03T20:54:06.933Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 17996724, 9498362, 0 committed:false) SnapCreateDur 21.866µs SnapOpenDur 80.491438ms 2015-08-03T20:54:06.984Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10530000 2015-08-03T20:54:08.081Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10540000 2015-08-03T20:54:08.097Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9640000 QueuedCount 7198 2015-08-03T20:54:08.345Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18019582, 9509791, 0 committed:false) SnapCreateDur 14.171µs SnapOpenDur 90.328018ms 2015-08-03T20:54:09.131Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10550000 2015-08-03T20:54:09.359Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.017µs CommitTime 894.244401ms TotalFlushTime 11m38.130331179s TotalCommitTime 3m6.731462795s 2015-08-03T20:54:09.359Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18023294, 9511647, 292 committed:true) SnapCreateDur 920.203979ms SnapOpenDur 32.371µs 2015-08-03T20:54:09.412Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9650000 QueuedCount 3019 2015-08-03T20:54:09.777Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9660000 QueuedCount 590 2015-08-03T20:54:09.885Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18045972, 9522986, 0 committed:false) SnapCreateDur 30.317µs SnapOpenDur 83.787592ms 2015-08-03T20:54:10.165Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10560000 2015-08-03T20:54:11.194Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9670000 QueuedCount 1984 2015-08-03T20:54:11.212Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10570000 2015-08-03T20:54:11.327Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18068390, 9534195, 0 committed:false) SnapCreateDur 21.602µs SnapOpenDur 90.763935ms 2015-08-03T20:54:12.26Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10580000 2015-08-03T20:54:12.464Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.127µs CommitTime 1.003557483s TotalFlushTime 11m39.933392865s TotalCommitTime 3m7.735020278s 2015-08-03T20:54:13.291Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10590000 2015-08-03T20:54:14.195Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18075972, 9537986, 293 committed:true) SnapCreateDur 1.029725686s SnapOpenDur 1.730295953s 2015-08-03T20:54:14.335Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10600000 2015-08-03T20:54:14.37Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9680000 QueuedCount 7388 2015-08-03T20:54:14.712Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18098704, 9549352, 0 committed:false) SnapCreateDur 22.463µs SnapOpenDur 83.725554ms 2015-08-03T20:54:15.375Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10610000 2015-08-03T20:54:16.131Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.617µs CommitTime 1.296728879s TotalFlushTime 11m40.400805605s TotalCommitTime 3m9.031749157s 2015-08-03T20:54:16.132Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18102446, 9551223, 294 committed:true) SnapCreateDur 1.323738186s SnapOpenDur 54.117µs 2015-08-03T20:54:16.177Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9690000 QueuedCount 2219 2015-08-03T20:54:16.414Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10620000 2015-08-03T20:54:16.532Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9700000 QueuedCount 844 2015-08-03T20:54:16.657Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18125204, 9562602, 0 committed:false) SnapCreateDur 30.893µs SnapOpenDur 83.448745ms 2015-08-03T20:54:17.462Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10630000 2015-08-03T20:54:17.853Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9710000 QueuedCount 2378 2015-08-03T20:54:17.99Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18147922, 9573961, 0 committed:false) SnapCreateDur 19.197µs SnapOpenDur 88.249825ms 2015-08-03T20:54:18.509Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10640000 2015-08-03T20:54:18.951Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.56µs CommitTime 870.873729ms TotalFlushTime 11m42.052051331s TotalCommitTime 3m9.902622886s 2015-08-03T20:54:18.951Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18151708, 9575854, 295 committed:true) SnapCreateDur 897.034862ms SnapOpenDur 36.204µs 2015-08-03T20:54:19.108Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9720000 QueuedCount 5856 2015-08-03T20:54:19.551Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10650000 2015-08-03T20:54:20.321Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18174448, 9587224, 0 committed:false) SnapCreateDur 27.367µs SnapOpenDur 86.946484ms 2015-08-03T20:54:20.502Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9730000 QueuedCount 7415 2015-08-03T20:54:20.585Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10660000 2015-08-03T20:54:21.627Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10670000 2015-08-03T20:54:22.664Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10680000 2015-08-03T20:54:22.773Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18197176, 9598588, 0 committed:false) SnapCreateDur 34.553µs SnapOpenDur 160.064986ms 2015-08-03T20:54:23.709Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10690000 2015-08-03T20:54:23.866Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.3µs CommitTime 972.560973ms TotalFlushTime 11m45.619207349s TotalCommitTime 3m10.875183859s 2015-08-03T20:54:23.866Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18200962, 9600481, 296 committed:true) SnapCreateDur 999.039664ms SnapOpenDur 36.269µs 2015-08-03T20:54:23.916Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9740000 QueuedCount 3195 2015-08-03T20:54:24.276Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9750000 QueuedCount 896 2015-08-03T20:54:24.386Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18223730, 9611865, 0 committed:false) SnapCreateDur 28.687µs SnapOpenDur 75.835107ms 2015-08-03T20:54:24.74Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10700000 2015-08-03T20:54:25.786Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10710000 2015-08-03T20:54:26.836Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10720000 2015-08-03T20:54:27.874Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10730000 2015-08-03T20:54:28.612Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 2015-08-03T20:54:28.773Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9760000 QueuedCount 2378 2015-08-03T20:54:28.916Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10740000 2015-08-03T20:54:28.918Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18246350, 9623175, 0 committed:false) SnapCreateDur 40.877µs SnapOpenDur 47.201677ms 2015-08-03T20:54:29.963Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10750000 2015-08-03T20:54:30.923Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 16.663µs CommitTime 512.243523ms TotalFlushTime 11m50.536365463s TotalCommitTime 3m11.387427382s 2015-08-03T20:54:30.923Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18250134, 9625067, 297 committed:true) SnapCreateDur 1.900431129s SnapOpenDur 27.749µs 2015-08-03T20:54:31.002Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10760000 2015-08-03T20:54:31.049Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9770000 QueuedCount 3855 2015-08-03T20:54:31.232Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18272788, 9636394, 0 committed:false) SnapCreateDur 21.834µs SnapOpenDur 77.633397ms 2015-08-03T20:54:31.414Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9780000 QueuedCount 7325 2015-08-03T20:54:32.036Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10770000 2015-08-03T20:54:32.765Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18295438, 9647719, 0 committed:false) SnapCreateDur 30.931µs SnapOpenDur 160.798126ms 2015-08-03T20:54:33.082Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10780000 2015-08-03T20:54:33.809Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 14.7µs CommitTime 914.160167ms TotalFlushTime 11m52.140311868s TotalCommitTime 3m12.301587549s 2015-08-03T20:54:33.809Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18299252, 9649626, 298 committed:true) SnapCreateDur 940.226004ms SnapOpenDur 29.005µs 2015-08-03T20:54:33.847Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9790000 QueuedCount 1143 2015-08-03T20:54:34.067Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9800000 QueuedCount 796 2015-08-03T20:54:34.121Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10790000 2015-08-03T20:54:34.173Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18321916, 9660958, 0 committed:false) SnapCreateDur 28.18µs SnapOpenDur 81.325758ms 2015-08-03T20:54:35.163Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10800000 2015-08-03T20:54:35.361Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9810000 QueuedCount 2354 2015-08-03T20:54:35.566Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18344704, 9672352, 0 committed:false) SnapCreateDur 32.35µs SnapOpenDur 155.74414ms 2015-08-03T20:54:35.923Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9820000 QueuedCount 3816 2015-08-03T20:54:36.211Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10810000 2015-08-03T20:54:36.974Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18367320, 9683660, 0 committed:false) SnapCreateDur 17.149µs SnapOpenDur 78.481341ms 2015-08-03T20:54:37.259Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10820000 2015-08-03T20:54:37.733Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.974µs CommitTime 658.846714ms TotalFlushTime 11m54.921398286s TotalCommitTime 3m12.960434263s 2015-08-03T20:54:37.734Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18371280, 9685640, 299 committed:true) SnapCreateDur 685.239181ms SnapOpenDur 42.175µs 2015-08-03T20:54:37.855Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9830000 QueuedCount 5378 2015-08-03T20:54:38.113Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18393960, 9696980, 0 committed:false) SnapCreateDur 24.027µs SnapOpenDur 85.924528ms 2015-08-03T20:54:38.208Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9840000 QueuedCount 4193 2015-08-03T20:54:38.302Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10830000 2015-08-03T20:54:39.336Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10840000 2015-08-03T20:54:39.395Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18416674, 9708337, 0 committed:false) SnapCreateDur 37.454µs SnapOpenDur 158.578242ms 2015-08-03T20:54:39.467Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9850000 QueuedCount 4567 2015-08-03T20:54:40.377Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10850000 2015-08-03T20:54:40.771Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9860000 QueuedCount 297 2015-08-03T20:54:40.857Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18439072, 9719536, 0 committed:false) SnapCreateDur 24.87µs SnapOpenDur 79.09007ms 2015-08-03T20:54:41.412Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10860000 2015-08-03T20:54:41.652Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.403µs CommitTime 673.366071ms TotalFlushTime 11m57.68116389s TotalCommitTime 3m13.633800334s 2015-08-03T20:54:41.652Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18442790, 9721395, 300 committed:true) SnapCreateDur 701.607179ms SnapOpenDur 64.42µs 2015-08-03T20:54:41.962Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9870000 QueuedCount 3728 2015-08-03T20:54:42.196Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18465520, 9732760, 0 committed:false) SnapCreateDur 37.15µs SnapOpenDur 87.828539ms 2015-08-03T20:54:42.458Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10870000 2015-08-03T20:54:43.308Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9880000 QueuedCount 5267 2015-08-03T20:54:43.491Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10880000 2015-08-03T20:54:43.573Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18488238, 9744119, 0 committed:false) SnapCreateDur 22.49µs SnapOpenDur 159.396779ms 2015-08-03T20:54:43.791Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9890000 QueuedCount 6801 2015-08-03T20:54:44.531Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10890000 2015-08-03T20:54:44.925Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18510944, 9755472, 0 committed:false) SnapCreateDur 14.672µs SnapOpenDur 78.397827ms 2015-08-03T20:54:45.479Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.304µs CommitTime 437.005891ms TotalFlushTime 12m0.573725282s TotalCommitTime 3m14.070806225s 2015-08-03T20:54:45.48Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18514752, 9757376, 301 committed:true) SnapCreateDur 450.3195ms SnapOpenDur 53.871µs 2015-08-03T20:54:45.547Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9900000 QueuedCount 2626 2015-08-03T20:54:45.565Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10900000 2015-08-03T20:54:45.91Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9910000 QueuedCount 287 2015-08-03T20:54:46.013Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18537520, 9768760, 0 committed:false) SnapCreateDur 30.15µs SnapOpenDur 90.35875ms 2015-08-03T20:54:46.615Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10910000 2015-08-03T20:54:47.288Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9920000 QueuedCount 1864 2015-08-03T20:54:47.49Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18560300, 9780150, 0 committed:false) SnapCreateDur 21.976µs SnapOpenDur 162.181207ms 2015-08-03T20:54:47.661Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10920000 2015-08-03T20:54:47.824Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9930000 QueuedCount 3348 2015-08-03T20:54:48.709Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10930000 2015-08-03T20:54:48.939Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18582930, 9791465, 0 committed:false) SnapCreateDur 22.98µs SnapOpenDur 135.136435ms 2015-08-03T20:54:49.396Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.392µs CommitTime 369.460535ms TotalFlushTime 12m3.576778389s TotalCommitTime 3m14.44026676s 2015-08-03T20:54:49.397Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18586730, 9793365, 302 committed:true) SnapCreateDur 382.385575ms SnapOpenDur 29.335µs 2015-08-03T20:54:49.519Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9940000 QueuedCount 4621 2015-08-03T20:54:49.736Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10940000 2015-08-03T20:54:49.747Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18608246, 9804123, 0 committed:false) SnapCreateDur 19.44µs SnapOpenDur 88.421156ms 2015-08-03T20:54:49.928Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9950000 QueuedCount 7002 2015-08-03T20:54:50.788Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10950000 2015-08-03T20:54:51.206Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18629500, 9814750, 0 committed:false) SnapCreateDur 14.327µs SnapOpenDur 86.287577ms 2015-08-03T20:54:51.348Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9960000 QueuedCount 8509 2015-08-03T20:54:51.824Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10960000 2015-08-03T20:54:52.636Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18652192, 9826096, 0 committed:false) SnapCreateDur 21.127µs SnapOpenDur 78.376362ms 2015-08-03T20:54:52.69Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9970000 QueuedCount 420 2015-08-03T20:54:52.864Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10970000 2015-08-03T20:54:53.028Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.177µs CommitTime 313.723372ms TotalFlushTime 12m6.490350351s TotalCommitTime 3m14.753990132s 2015-08-03T20:54:53.028Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18655964, 9827982, 303 committed:true) SnapCreateDur 327.434176ms SnapOpenDur 36.585µs 2015-08-03T20:54:53.231Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9980000 QueuedCount 1993 2015-08-03T20:54:53.355Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18678730, 9839365, 0 committed:false) SnapCreateDur 22.367µs SnapOpenDur 84.50542ms 2015-08-03T20:54:53.91Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10980000 2015-08-03T20:54:54.678Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 9990000 QueuedCount 3551 2015-08-03T20:54:54.925Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18701440, 9850720, 0 committed:false) SnapCreateDur 34.206µs SnapOpenDur 166.499885ms 2015-08-03T20:54:54.941Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10990000 2015-08-03T20:54:55.216Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10000000 QueuedCount 5170 2015-08-03T20:54:55.985Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11000000 2015-08-03T20:54:56.429Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18724382, 9862191, 0 committed:false) SnapCreateDur 23.07µs SnapOpenDur 139.862308ms 2015-08-03T20:54:57.037Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11010000 2015-08-03T20:54:57.276Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.097µs CommitTime 741.878937ms TotalFlushTime 12m9.44123857s TotalCommitTime 3m15.495869069s 2015-08-03T20:54:57.276Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18728072, 9864036, 304 committed:true) SnapCreateDur 771.771552ms SnapOpenDur 45.148µs 2015-08-03T20:54:57.366Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10010000 QueuedCount 2826 2015-08-03T20:54:57.652Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18750836, 9875418, 0 committed:false) SnapCreateDur 19.8µs SnapOpenDur 82.667513ms 2015-08-03T20:54:57.733Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10020000 QueuedCount 4386 2015-08-03T20:54:58.086Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11020000 2015-08-03T20:54:58.613Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:453081179, Disk:2071937024, ExtraSnapshotData:0, Fragmentation:78.1324830942352% 2015-08-03T20:54:58.613Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:54:58.634Z-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 18442790. Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T20:54:59.115Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10030000 QueuedCount 179 2015-08-03T20:54:59.128Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11030000 2015-08-03T20:54:59.175Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18773654, 9886827, 0 committed:false) SnapCreateDur 30.75µs SnapOpenDur 49.154655ms 2015-08-03T20:55:00.161Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11040000 2015-08-03T20:55:00.662Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10040000 QueuedCount 1752 2015-08-03T20:55:00.81Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18796422, 9898211, 0 committed:false) SnapCreateDur 30.78µs SnapOpenDur 78.897678ms 2015-08-03T20:55:01.206Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11050000 2015-08-03T20:55:01.574Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.773µs CommitTime 643.309705ms TotalFlushTime 12m12.724832653s TotalCommitTime 3m16.139178774s 2015-08-03T20:55:01.575Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18800182, 9900091, 305 committed:true) SnapCreateDur 670.86857ms SnapOpenDur 46.72µs 2015-08-03T20:55:01.819Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10050000 QueuedCount 4935 2015-08-03T20:55:02.082Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18822358, 9911179, 0 committed:false) SnapCreateDur 21.844µs SnapOpenDur 84.484243ms 2015-08-03T20:55:02.251Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11060000 2015-08-03T20:55:03.286Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11070000 2015-08-03T20:55:04.003Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10060000 QueuedCount 6500 2015-08-03T20:55:04.327Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11080000 2015-08-03T20:55:04.388Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18845120, 9922560, 0 committed:false) SnapCreateDur 21.39µs SnapOpenDur 142.946611ms 2015-08-03T20:55:05.22Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":21539013205,"bucket-1:by_email:avg_scan_wait_latency":21538081184,"bucket-1:by_email:avg_ts_interval":1547618888,"bucket-1:by_email:avg_ts_items_count":10205,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":455397523,"bucket-1:by_email:delete_bytes":382492912,"bucket-1:by_email:disk_size":2135552000,"bucket-1:by_email:flush_queue_size":0,"bucket-1:by_email:frag_percent":78,"bucket-1:by_email:get_bytes":390107567,"bucket-1:by_email:insert_bytes":1038994320,"bucket-1:by_email:items_count":1000001,"bucket-1:by_email:num_commits":305,"bucket-1:by_email:num_compactions":16,"bucket-1:by_email:num_completed_requests":16515,"bucket-1:by_email:num_docs_indexed":10042766,"bucket-1:by_email:num_docs_pending":311,"bucket-1:by_email:num_docs_queued":1000449,"bucket-1:by_email:num_flush_queued":10068391,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":16899,"bucket-1:by_email:num_rows_returned":279,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":1192,"bucket-1:by_email:scan_bytes_read":11160,"bucket-1:by_email:scan_wait_duration":363972033944937,"bucket-1:by_email:since_last_snapshot":2305723782,"bucket-1:by_email:timings/storage_clone_handle":"16515 942167695 2749257031590261","bucket-1:by_email:timings/storage_commit":"305 196139178774 -566023682650202056","bucket-1:by_email:timings/storage_del":"8924411 153807211079 1751759130418296143","bucket-1:by_email:timings/storage_get":"10068391 148185127597 -3578781245783753201","bucket-1:by_email:timings/storage_handle_open":"305 19446570862 8460631254806782616","bucket-1:by_email:timings/storage_iterator_next":"279 5778900 167251171030","bucket-1:by_email:timings/storage_new_iterator":"16515 1365631589 2864937961101695","bucket-1:by_email:timings/storage_set":"19848822 420323524025 7520553628981731647","bucket-1:by_email:timings/storage_snapshot_close":"886 14532167783 910286118130240853","bucket-1:by_email:timings/storage_snapshot_create":"887 85887032541 -5631765619936896713","bucket-1:by_email:total_scan_duration":363987784152274,"bucket-1:mutation_queue_size":1019781,"bucket-1:num_mutations_queued":11088172,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":531,"memory_quota":104857600000,"memory_used":677122048,"needs_restart":false,"num_connections":384,"timings/stats_response":"1716 600386372 4811798437114348"} 2015-08-03T20:55:05.367Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.46µs CommitTime 857.747451ms TotalFlushTime 12m15.301243757s TotalCommitTime 3m16.996926225s 2015-08-03T20:55:05.367Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18848822, 9924411, 306 committed:true) SnapCreateDur 884.015769ms SnapOpenDur 27.32µs 2015-08-03T20:55:05.41Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11090000 2015-08-03T20:55:05.443Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10070000 QueuedCount 4205 2015-08-03T20:55:05.87Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18871540, 9935770, 0 committed:false) SnapCreateDur 27.005µs SnapOpenDur 79.858728ms 2015-08-03T20:55:05.894Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10080000 QueuedCount 1856 2015-08-03T20:55:06.535Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11100000 2015-08-03T20:55:07.172Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10090000 QueuedCount 1454 2015-08-03T20:55:07.299Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18894236, 9947118, 0 committed:false) SnapCreateDur 14.621µs SnapOpenDur 86.833012ms 2015-08-03T20:55:07.614Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11110000 2015-08-03T20:55:08.611Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10100000 QueuedCount 1111 2015-08-03T20:55:08.661Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11120000 2015-08-03T20:55:08.86Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.247µs CommitTime 207.577386ms TotalFlushTime 12m18.28974272s TotalCommitTime 3m17.204503611s 2015-08-03T20:55:08.86Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18913284, 9956642, 307 committed:true) SnapCreateDur 223.933757ms SnapOpenDur 27.626µs 2015-08-03T20:55:09.076Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10110000 QueuedCount 2647 2015-08-03T20:55:09.198Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18936008, 9968004, 0 committed:false) SnapCreateDur 15.091µs SnapOpenDur 54.278457ms 2015-08-03T20:55:09.703Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11130000 2015-08-03T20:55:10.143Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10120000 QueuedCount 4054 2015-08-03T20:55:10.382Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18958472, 9979236, 0 committed:false) SnapCreateDur 23.717µs SnapOpenDur 149.856359ms 2015-08-03T20:55:10.739Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11140000 2015-08-03T20:55:11.554Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.557µs CommitTime 972.781631ms TotalFlushTime 12m19.66832912s TotalCommitTime 3m18.177285242s 2015-08-03T20:55:11.554Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18966284, 9983142, 308 committed:true) SnapCreateDur 1.003729906s SnapOpenDur 45.553µs 2015-08-03T20:55:11.644Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10130000 QueuedCount 3709 2015-08-03T20:55:11.782Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11150000 2015-08-03T20:55:12.035Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 18988838, 9994419, 0 committed:false) SnapCreateDur 15.49µs SnapOpenDur 58.414602ms 2015-08-03T20:55:12.07Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10140000 QueuedCount 1342 2015-08-03T20:55:12.816Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11160000 2015-08-03T20:55:13.862Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11170000 2015-08-03T20:55:14.896Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11180000 2015-08-03T20:55:15.514Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10150000 QueuedCount 750 2015-08-03T20:55:15.705Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19011066, 10005533, 0 committed:false) SnapCreateDur 30.567µs SnapOpenDur 146.164356ms 2015-08-03T20:55:15.939Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11190000 2015-08-03T20:55:16.736Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 10.637µs CommitTime 894.942357ms TotalFlushTime 12m23.607414237s TotalCommitTime 3m19.072227599s 2015-08-03T20:55:16.736Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19014820, 10007410, 309 committed:true) SnapCreateDur 921.658029ms SnapOpenDur 31.406µs 2015-08-03T20:55:16.991Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11200000 2015-08-03T20:55:17.018Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10160000 QueuedCount 4228 2015-08-03T20:55:17.263Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19037602, 10018801, 0 committed:false) SnapCreateDur 27.496µs SnapOpenDur 83.652575ms 2015-08-03T20:55:18.036Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11210000 2015-08-03T20:55:18.457Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10170000 QueuedCount 5803 2015-08-03T20:55:18.676Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19060428, 10030214, 0 committed:false) SnapCreateDur 13.735µs SnapOpenDur 91.0218ms 2015-08-03T20:55:18.855Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10180000 QueuedCount 1608 2015-08-03T20:55:19.083Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11220000 2015-08-03T20:55:19.639Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 20.73µs CommitTime 711.293783ms TotalFlushTime 12m25.499463335s TotalCommitTime 3m19.783521382s 2015-08-03T20:55:19.639Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19071836, 10035918, 310 committed:true) SnapCreateDur 722.443607ms SnapOpenDur 36.805µs 2015-08-03T20:55:19.954Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10190000 QueuedCount 3140 2015-08-03T20:55:20.111Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11230000 2015-08-03T20:55:20.152Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19094462, 10047231, 0 committed:false) SnapCreateDur 29.54µs SnapOpenDur 80.66729ms 2015-08-03T20:55:21.16Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11240000 2015-08-03T20:55:21.27Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10200000 QueuedCount 4685 2015-08-03T20:55:21.508Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19117194, 10058597, 0 committed:false) SnapCreateDur 28.383µs SnapOpenDur 143.461131ms 2015-08-03T20:55:21.732Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10210000 QueuedCount 6262 2015-08-03T20:55:22.203Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11250000 2015-08-03T20:55:23.235Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11260000 2015-08-03T20:55:24.017Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19139988, 10069994, 0 committed:false) SnapCreateDur 21.327µs SnapOpenDur 137.904444ms 2015-08-03T20:55:24.282Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11270000 2015-08-03T20:55:24.776Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.88µs CommitTime 636.239148ms TotalFlushTime 12m29.463430562s TotalCommitTime 3m20.41976053s 2015-08-03T20:55:24.777Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19143734, 10071867, 311 committed:true) SnapCreateDur 664.316088ms SnapOpenDur 34.478µs 2015-08-03T20:55:24.836Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10220000 QueuedCount 2021 2015-08-03T20:55:25.128Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19166478, 10083239, 0 committed:false) SnapCreateDur 27.317µs SnapOpenDur 79.710626ms 2015-08-03T20:55:25.159Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10230000 QueuedCount 1658 2015-08-03T20:55:25.312Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11280000 2015-08-03T20:55:26.326Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10240000 QueuedCount 1247 2015-08-03T20:55:26.36Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11290000 2015-08-03T20:55:26.488Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19189136, 10094568, 0 committed:false) SnapCreateDur 24.277µs SnapOpenDur 137.590402ms 2015-08-03T20:55:26.849Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10250000 QueuedCount 2713 2015-08-03T20:55:27.41Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11300000 2015-08-03T20:55:28.45Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11310000 2015-08-03T20:55:28.592Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19211742, 10105871, 0 committed:false) SnapCreateDur 31.74µs SnapOpenDur 122.69904ms 2015-08-03T20:55:29.281Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.77µs CommitTime 546.709072ms TotalFlushTime 12m32.914762988s TotalCommitTime 3m20.966469602s 2015-08-03T20:55:29.281Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19215576, 10107788, 312 committed:true) SnapCreateDur 574.424142ms SnapOpenDur 29.829µs 2015-08-03T20:55:29.413Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10260000 QueuedCount 4272 2015-08-03T20:55:29.49Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11320000 2015-08-03T20:55:29.629Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19238206, 10119103, 0 committed:false) SnapCreateDur 21.994µs SnapOpenDur 75.594463ms 2015-08-03T20:55:29.792Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10270000 QueuedCount 7739 2015-08-03T20:55:30.528Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11330000 2015-08-03T20:55:30.99Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19261014, 10130507, 0 committed:false) SnapCreateDur 23.284µs SnapOpenDur 148.053987ms 2015-08-03T20:55:31.102Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10280000 QueuedCount 5374 2015-08-03T20:55:31.573Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11340000 2015-08-03T20:55:32.415Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19283612, 10141806, 0 committed:false) SnapCreateDur 28.267µs SnapOpenDur 124.200975ms 2015-08-03T20:55:32.466Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10290000 QueuedCount 1200 2015-08-03T20:55:32.612Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11350000 2015-08-03T20:55:33.195Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 13.477µs CommitTime 657.666802ms TotalFlushTime 12m35.66267434s TotalCommitTime 3m21.624136404s 2015-08-03T20:55:33.195Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19287454, 10143727, 313 committed:true) SnapCreateDur 684.939686ms SnapOpenDur 36.704µs 2015-08-03T20:55:33.562Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10300000 QueuedCount 2671 2015-08-03T20:55:33.657Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11360000 2015-08-03T20:55:33.734Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19310036, 10155018, 0 committed:false) SnapCreateDur 20.43µs SnapOpenDur 55.112486ms 2015-08-03T20:55:34.691Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11370000 2015-08-03T20:55:34.963Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10310000 QueuedCount 4246 2015-08-03T20:55:35.134Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19332796, 10166398, 0 committed:false) SnapCreateDur 18.564µs SnapOpenDur 86.115034ms 2015-08-03T20:55:35.392Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10320000 QueuedCount 5778 2015-08-03T20:55:35.734Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11380000 2015-08-03T20:55:36.499Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19355482, 10177741, 0 committed:false) SnapCreateDur 32.177µs SnapOpenDur 132.82663ms 2015-08-03T20:55:36.787Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11390000 2015-08-03T20:55:37.242Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.413µs CommitTime 640.197129ms TotalFlushTime 12m38.614165962s TotalCommitTime 3m22.264333533s 2015-08-03T20:55:37.242Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19359244, 10179622, 314 committed:true) SnapCreateDur 668.423891ms SnapOpenDur 34.97µs 2015-08-03T20:55:37.353Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10330000 QueuedCount 5328 2015-08-03T20:55:37.787Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19381654, 10190827, 0 committed:false) SnapCreateDur 27.404µs SnapOpenDur 92.474672ms 2015-08-03T20:55:37.834Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11400000 2015-08-03T20:55:37.848Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10340000 QueuedCount 3001 2015-08-03T20:55:38.869Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11410000 2015-08-03T20:55:39.149Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10350000 QueuedCount 664 2015-08-03T20:55:39.324Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19404446, 10202223, 0 committed:false) SnapCreateDur 22.587µs SnapOpenDur 152.944648ms 2015-08-03T20:55:39.911Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11420000 2015-08-03T20:55:40.224Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10360000 QueuedCount 2230 2015-08-03T20:55:40.399Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19427218, 10213609, 0 committed:false) SnapCreateDur 22.764µs SnapOpenDur 129.628511ms 2015-08-03T20:55:40.952Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11430000 2015-08-03T20:55:41.216Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.57µs CommitTime 692.496808ms TotalFlushTime 12m41.346066282s TotalCommitTime 3m22.956830341s 2015-08-03T20:55:41.988Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11440000 2015-08-03T20:55:42.657Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19430892, 10215446, 315 committed:true) SnapCreateDur 723.551546ms SnapOpenDur 1.441022095s 2015-08-03T20:55:42.657Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 2015-08-03T20:55:42.892Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10370000 QueuedCount 5645 2015-08-03T20:55:43.033Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11450000 2015-08-03T20:55:43.197Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19453614, 10226807, 0 committed:false) SnapCreateDur 31.72µs SnapOpenDur 94.879073ms 2015-08-03T20:55:44.076Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11460000 2015-08-03T20:55:44.483Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10380000 QueuedCount 7188 2015-08-03T20:55:44.799Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19476292, 10238146, 0 committed:false) SnapCreateDur 34.574µs SnapOpenDur 155.516497ms 2015-08-03T20:55:45.113Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11470000 2015-08-03T20:55:46.159Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11480000 2015-08-03T20:55:46.521Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.974µs CommitTime 529.032881ms TotalFlushTime 12m43.233403736s TotalCommitTime 3m23.485863222s 2015-08-03T20:55:46.521Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19479982, 10239991, 316 committed:true) SnapCreateDur 1.617286329s SnapOpenDur 31.89µs 2015-08-03T20:55:46.575Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10390000 QueuedCount 2958 2015-08-03T20:55:46.945Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10400000 QueuedCount 714 2015-08-03T20:55:47.062Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19502946, 10251473, 0 committed:false) SnapCreateDur 34.81µs SnapOpenDur 88.116967ms 2015-08-03T20:55:47.212Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11490000 2015-08-03T20:55:48.246Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11500000 2015-08-03T20:55:48.314Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10410000 QueuedCount 2209 2015-08-03T20:55:48.52Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19525598, 10262799, 0 committed:false) SnapCreateDur 25.19µs SnapOpenDur 163.529442ms 2015-08-03T20:55:49.287Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11510000 2015-08-03T20:55:49.465Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.517µs CommitTime 843.626482ms TotalFlushTime 12m44.956832412s TotalCommitTime 3m24.329489704s 2015-08-03T20:55:49.466Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19529306, 10264653, 317 committed:true) SnapCreateDur 871.025005ms SnapOpenDur 37.008µs 2015-08-03T20:55:49.706Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10420000 QueuedCount 5716 2015-08-03T20:55:50.005Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19552146, 10276073, 0 committed:false) SnapCreateDur 25.434µs SnapOpenDur 85.090434ms 2015-08-03T20:55:50.327Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11520000 2015-08-03T20:55:51.17Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10430000 QueuedCount 7056 2015-08-03T20:55:51.375Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11530000 2015-08-03T20:55:51.468Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19574448, 10287224, 0 committed:false) SnapCreateDur 22.836µs SnapOpenDur 156.178488ms 2015-08-03T20:55:51.564Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10440000 QueuedCount 3607 2015-08-03T20:55:52.41Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11540000 2015-08-03T20:55:52.597Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.344µs CommitTime 917.414523ms TotalFlushTime 12m46.812576956s TotalCommitTime 3m25.246904227s 2015-08-03T20:55:52.597Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19589534, 10294767, 318 committed:true) SnapCreateDur 930.786001ms SnapOpenDur 32.61µs 2015-08-03T20:55:52.727Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10450000 QueuedCount 4356 2015-08-03T20:55:52.893Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19612258, 10306129, 0 committed:false) SnapCreateDur 23.863µs SnapOpenDur 44.543179ms 2015-08-03T20:55:52.994Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10460000 QueuedCount 3945 2015-08-03T20:55:53.437Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11550000 2015-08-03T20:55:53.774Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19634956, 10317478, 0 committed:false) SnapCreateDur 10.647µs SnapOpenDur 77.665179ms 2015-08-03T20:55:53.835Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10470000 QueuedCount 1649 2015-08-03T20:55:54.483Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11560000 2015-08-03T20:55:54.658Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19657594, 10328797, 0 committed:false) SnapCreateDur 33.863µs SnapOpenDur 111.865212ms 2015-08-03T20:55:54.692Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10480000 QueuedCount 1298 2015-08-03T20:55:55.016Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.364µs CommitTime 282.846965ms TotalFlushTime 12m48.579065071s TotalCommitTime 3m25.529751192s 2015-08-03T20:55:55.016Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19661472, 10330736, 319 committed:true) SnapCreateDur 293.777892ms SnapOpenDur 31.439µs 2015-08-03T20:55:55.333Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10490000 QueuedCount 2772 2015-08-03T20:55:55.507Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19684042, 10342021, 0 committed:false) SnapCreateDur 19.65µs SnapOpenDur 78.14801ms 2015-08-03T20:55:55.513Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11570000 2015-08-03T20:55:56.564Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11580000 2015-08-03T20:55:56.92Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10500000 QueuedCount 4260 2015-08-03T20:55:57.186Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19706664, 10353332, 0 committed:false) SnapCreateDur 26.417µs SnapOpenDur 176.33045ms 2015-08-03T20:55:57.427Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10510000 QueuedCount 5426 2015-08-03T20:55:57.615Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11590000 2015-08-03T20:55:58.567Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19728632, 10364316, 0 committed:false) SnapCreateDur 34.524µs SnapOpenDur 126.990876ms 2015-08-03T20:55:58.66Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11600000 2015-08-03T20:55:59.284Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 17.616µs CommitTime 584.141335ms TotalFlushTime 12m51.711282273s TotalCommitTime 3m26.113892527s 2015-08-03T20:55:59.284Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19732402, 10366201, 320 committed:true) SnapCreateDur 611.581895ms SnapOpenDur 42.537µs 2015-08-03T20:55:59.406Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10520000 QueuedCount 6447 2015-08-03T20:55:59.701Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11610000 2015-08-03T20:55:59.812Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19753992, 10376996, 0 committed:false) SnapCreateDur 25.857µs SnapOpenDur 85.173631ms 2015-08-03T20:55:59.904Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10530000 QueuedCount 3502 2015-08-03T20:56:00.741Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11620000 2015-08-03T20:56:01.377Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19775478, 10387739, 0 committed:false) SnapCreateDur 28.515µs SnapOpenDur 150.807842ms 2015-08-03T20:56:01.419Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10540000 QueuedCount 1121 2015-08-03T20:56:01.785Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11630000 2015-08-03T20:56:02.384Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10550000 QueuedCount 699 2015-08-03T20:56:02.482Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19798172, 10399086, 0 committed:false) SnapCreateDur 22.487µs SnapOpenDur 80.169328ms 2015-08-03T20:56:02.825Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11640000 2015-08-03T20:56:03.196Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.747µs CommitTime 618.205125ms TotalFlushTime 12m54.5235324s TotalCommitTime 3m26.732097652s 2015-08-03T20:56:03.196Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19802044, 10401022, 321 committed:true) SnapCreateDur 650.01387ms SnapOpenDur 43.087µs 2015-08-03T20:56:03.439Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10560000 QueuedCount 4132 2015-08-03T20:56:03.648Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19824600, 10412300, 0 committed:false) SnapCreateDur 29.48µs SnapOpenDur 85.038485ms 2015-08-03T20:56:03.862Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11650000 2015-08-03T20:56:04.782Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10570000 QueuedCount 5672 2015-08-03T20:56:04.902Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11660000 2015-08-03T20:56:05.051Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19847296, 10423648, 0 committed:false) SnapCreateDur 21.993µs SnapOpenDur 157.800282ms 2015-08-03T20:56:05.221Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":23481064892,"bucket-1:by_email:avg_scan_wait_latency":23480138115,"bucket-1:by_email:avg_ts_interval":1045656277,"bucket-1:by_email:avg_ts_items_count":10211,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":431110918,"bucket-1:by_email:delete_bytes":403406263,"bucket-1:by_email:disk_size":1702744064,"bucket-1:by_email:flush_queue_size":8088,"bucket-1:by_email:frag_percent":74,"bucket-1:by_email:get_bytes":411393384,"bucket-1:by_email:insert_bytes":1090061805,"bucket-1:by_email:items_count":1000001,"bucket-1:by_email:num_commits":321,"bucket-1:by_email:num_compactions":16,"bucket-1:by_email:num_completed_requests":16899,"bucket-1:by_email:num_docs_indexed":10540416,"bucket-1:by_email:num_docs_pending":191,"bucket-1:by_email:num_docs_queued":1083293,"bucket-1:by_email:num_flush_queued":10587196,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":17283,"bucket-1:by_email:num_rows_returned":286,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":1248,"bucket-1:by_email:scan_bytes_read":11440,"bucket-1:by_email:scan_wait_duration":405807227055501,"bucket-1:by_email:since_last_snapshot":1403883864,"bucket-1:by_email:timings/storage_clone_handle":"16899 960778101 2750320027977669","bucket-1:by_email:timings/storage_commit":"321 206732097652 7128212945065113926","bucket-1:by_email:timings/storage_del":"9427031 164639557973 9052586354445652733","bucket-1:by_email:timings/storage_get":"10579110 155505099591 -3141048533242573237","bucket-1:by_email:timings/storage_handle_open":"321 20888061214 -7909585599070494178","bucket-1:by_email:timings/storage_iterator_next":"286 5931229 171965816823","bucket-1:by_email:timings/storage_new_iterator":"16899 1391753967 2867322690242565","bucket-1:by_email:timings/storage_set":"20854060 442438577068 5921624803277053258","bucket-1:by_email:timings/storage_snapshot_close":"926 15213237718 923494223499527908","bucket-1:by_email:timings/storage_snapshot_create":"927 90253710315 -5104349263812335065","bucket-1:by_email:total_scan_duration":405823244545207,"bucket-1:mutation_queue_size":1075864,"bucket-1:num_mutations_queued":11663060,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":560,"memory_quota":104857600000,"memory_used":565080064,"needs_restart":false,"num_connections":384,"timings/stats_response":"1788 624869435 4820510631597353"} 2015-08-03T20:56:05.255Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10580000 QueuedCount 7196 2015-08-03T20:56:05.991Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11670000 2015-08-03T20:56:06.49Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19869982, 10434991, 0 committed:false) SnapCreateDur 34.747µs SnapOpenDur 135.281458ms 2015-08-03T20:56:07.091Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11680000 2015-08-03T20:56:07.279Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.133µs CommitTime 656.250205ms TotalFlushTime 12m57.403301254s TotalCommitTime 3m27.388347857s 2015-08-03T20:56:07.279Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19873758, 10436879, 322 committed:true) SnapCreateDur 684.418454ms SnapOpenDur 37.931µs 2015-08-03T20:56:07.329Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10590000 QueuedCount 2835 2015-08-03T20:56:07.705Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10600000 QueuedCount 651 2015-08-03T20:56:07.824Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19896458, 10448229, 0 committed:false) SnapCreateDur 40.02µs SnapOpenDur 91.160755ms 2015-08-03T20:56:08.14Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11690000 2015-08-03T20:56:09.168Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10610000 QueuedCount 2169 2015-08-03T20:56:09.215Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11700000 2015-08-03T20:56:09.377Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19919094, 10459547, 0 committed:false) SnapCreateDur 29.136µs SnapOpenDur 159.528188ms 2015-08-03T20:56:09.555Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10620000 QueuedCount 3653 2015-08-03T20:56:10.261Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11710000 2015-08-03T20:56:10.54Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.047µs CommitTime 276.255954ms TotalFlushTime 13m0.015835276s TotalCommitTime 3m27.664603811s 2015-08-03T20:56:10.54Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19941710, 10470855, 323 committed:true) SnapCreateDur 289.60586ms SnapOpenDur 32.649µs 2015-08-03T20:56:10.682Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10630000 QueuedCount 5145 2015-08-03T20:56:10.863Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19964326, 10482163, 0 committed:false) SnapCreateDur 29.673µs SnapOpenDur 80.783535ms 2015-08-03T20:56:11.291Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11720000 2015-08-03T20:56:12.118Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10640000 QueuedCount 6722 2015-08-03T20:56:12.336Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11730000 2015-08-03T20:56:12.425Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 19987130, 10493565, 0 committed:false) SnapCreateDur 25.437µs SnapOpenDur 160.180336ms 2015-08-03T20:56:12.529Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10650000 QueuedCount 4406 2015-08-03T20:56:12.658Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:452227374, Disk:1925648384, ExtraSnapshotData:0, Fragmentation:76.51557897290557% 2015-08-03T20:56:12.658Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:56:12.681Z-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 19732402. Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T20:56:13.381Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11740000 2015-08-03T20:56:13.504Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.534µs CommitTime 869.762512ms TotalFlushTime 13m1.754033408s TotalCommitTime 3m28.534366323s 2015-08-03T20:56:13.51Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20002242, 10501121, 324 committed:true) SnapCreateDur 883.05136ms SnapOpenDur 5.928019ms 2015-08-03T20:56:13.64Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10660000 QueuedCount 4000 2015-08-03T20:56:13.947Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20024964, 10512482, 0 committed:false) SnapCreateDur 33.314µs SnapOpenDur 83.880138ms 2015-08-03T20:56:14.408Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11750000 2015-08-03T20:56:15.123Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10670000 QueuedCount 7466 2015-08-03T20:56:15.44Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20047684, 10523842, 0 committed:false) SnapCreateDur 26.46µs SnapOpenDur 151.557296ms 2015-08-03T20:56:15.46Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11760000 2015-08-03T20:56:15.519Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10680000 QueuedCount 3287 2015-08-03T20:56:16.507Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11770000 2015-08-03T20:56:16.73Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.853µs CommitTime 1.068132716s TotalFlushTime 13m3.541758618s TotalCommitTime 3m29.602499039s 2015-08-03T20:56:16.731Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20062852, 10531426, 325 committed:true) SnapCreateDur 1.096088597s SnapOpenDur 29.968µs 2015-08-03T20:56:16.839Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10690000 QueuedCount 2918 2015-08-03T20:56:17.005Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20085322, 10542661, 0 committed:false) SnapCreateDur 12.242µs SnapOpenDur 41.43174ms 2015-08-03T20:56:17.153Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10700000 QueuedCount 8258 2015-08-03T20:56:17.54Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11780000 2015-08-03T20:56:18.361Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20108270, 10554135, 0 committed:false) SnapCreateDur 22.342µs SnapOpenDur 84.929098ms 2015-08-03T20:56:18.451Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10710000 QueuedCount 4019 2015-08-03T20:56:18.59Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11790000 2015-08-03T20:56:19.634Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11800000 2015-08-03T20:56:19.803Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20130806, 10565403, 0 committed:false) SnapCreateDur 25.603µs SnapOpenDur 120.403103ms 2015-08-03T20:56:19.832Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10720000 QueuedCount 1621 2015-08-03T20:56:20.292Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 3.415µs CommitTime 400.069658ms TotalFlushTime 13m6.315172589s TotalCommitTime 3m30.002568697s 2015-08-03T20:56:20.292Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20134574, 10567287, 326 committed:true) SnapCreateDur 415.137313ms SnapOpenDur 28.734µs 2015-08-03T20:56:20.487Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10730000 QueuedCount 3143 2015-08-03T20:56:20.641Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20157234, 10578617, 0 committed:false) SnapCreateDur 22.864µs SnapOpenDur 85.306608ms 2015-08-03T20:56:20.665Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11810000 2015-08-03T20:56:21.709Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11820000 2015-08-03T20:56:21.787Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10740000 QueuedCount 4646 2015-08-03T20:56:21.967Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20179876, 10589938, 0 committed:false) SnapCreateDur 13.245µs SnapOpenDur 85.677785ms 2015-08-03T20:56:22.091Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10750000 QueuedCount 4271 2015-08-03T20:56:22.761Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11830000 2015-08-03T20:56:22.862Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20202538, 10601269, 0 committed:false) SnapCreateDur 11.436µs SnapOpenDur 72.912213ms 2015-08-03T20:56:23.306Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.169µs CommitTime 338.692858ms TotalFlushTime 13m8.608875698s TotalCommitTime 3m30.341261555s 2015-08-03T20:56:23.307Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20206368, 10603184, 327 committed:true) SnapCreateDur 350.991828ms SnapOpenDur 29.299µs 2015-08-03T20:56:23.393Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10760000 QueuedCount 3851 2015-08-03T20:56:23.787Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11840000 2015-08-03T20:56:23.82Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20229020, 10614510, 0 committed:false) SnapCreateDur 24.167µs SnapOpenDur 82.208562ms 2015-08-03T20:56:23.856Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10770000 QueuedCount 1547 2015-08-03T20:56:24.835Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11850000 2015-08-03T20:56:25.084Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10780000 QueuedCount 1152 2015-08-03T20:56:25.262Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20251734, 10625867, 0 committed:false) SnapCreateDur 33.993µs SnapOpenDur 149.493143ms 2015-08-03T20:56:25.873Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11860000 2015-08-03T20:56:26.555Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10790000 QueuedCount 2679 2015-08-03T20:56:26.739Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20274416, 10637208, 0 committed:false) SnapCreateDur 22.526µs SnapOpenDur 126.266387ms 2015-08-03T20:56:26.928Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11870000 2015-08-03T20:56:27.529Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.326µs CommitTime 670.893169ms TotalFlushTime 13m11.641241759s TotalCommitTime 3m31.012154724s 2015-08-03T20:56:27.53Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20278300, 10639150, 328 committed:true) SnapCreateDur 686.314362ms SnapOpenDur 38.891µs 2015-08-03T20:56:27.74Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10800000 QueuedCount 6141 2015-08-03T20:56:27.965Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11880000 2015-08-03T20:56:28.046Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20300840, 10650420, 0 committed:false) SnapCreateDur 25.464µs SnapOpenDur 83.069373ms 2015-08-03T20:56:28.214Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10810000 QueuedCount 7576 2015-08-03T20:56:29.016Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11890000 2015-08-03T20:56:29.395Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20323352, 10661676, 0 committed:false) SnapCreateDur 14.736µs SnapOpenDur 85.752214ms 2015-08-03T20:56:29.467Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10820000 QueuedCount 3312 2015-08-03T20:56:30.051Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11900000 2015-08-03T20:56:30.357Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20345970, 10672985, 0 committed:false) SnapCreateDur 12.164µs SnapOpenDur 71.044854ms 2015-08-03T20:56:30.396Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10830000 QueuedCount 890 2015-08-03T20:56:30.731Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 3.495µs CommitTime 306.761068ms TotalFlushTime 13m14.148754634s TotalCommitTime 3m31.318915792s 2015-08-03T20:56:30.731Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20349526, 10674763, 329 committed:true) SnapCreateDur 318.949571ms SnapOpenDur 26.39µs 2015-08-03T20:56:31.072Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10840000 QueuedCount 2550 2015-08-03T20:56:31.086Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11910000 2015-08-03T20:56:31.239Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20372456, 10686228, 0 committed:false) SnapCreateDur 20.63µs SnapOpenDur 75.66743ms 2015-08-03T20:56:32.121Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11920000 2015-08-03T20:56:32.476Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10850000 QueuedCount 4108 2015-08-03T20:56:32.709Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20395190, 10697595, 0 committed:false) SnapCreateDur 33.35µs SnapOpenDur 146.757038ms 2015-08-03T20:56:32.963Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10860000 QueuedCount 5653 2015-08-03T20:56:33.16Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11930000 2015-08-03T20:56:34.209Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11940000 2015-08-03T20:56:34.649Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20417896, 10708948, 0 committed:false) SnapCreateDur 32.113µs SnapOpenDur 136.548932ms 2015-08-03T20:56:35.239Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11950000 2015-08-03T20:56:35.464Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 15.193µs CommitTime 681.251906ms TotalFlushTime 13m17.658154425s TotalCommitTime 3m32.000167698s 2015-08-03T20:56:35.464Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20421678, 10710839, 330 committed:true) SnapCreateDur 709.75287ms SnapOpenDur 36.963µs 2015-08-03T20:56:35.556Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10870000 QueuedCount 3406 2015-08-03T20:56:35.842Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20444478, 10722239, 0 committed:false) SnapCreateDur 11.544µs SnapOpenDur 44.107933ms 2015-08-03T20:56:35.896Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10880000 QueuedCount 2984 2015-08-03T20:56:36.285Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11960000 2015-08-03T20:56:37.222Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10890000 QueuedCount 554 2015-08-03T20:56:37.319Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20466942, 10733471, 0 committed:false) SnapCreateDur 10.09µs SnapOpenDur 84.174257ms 2015-08-03T20:56:37.338Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11970000 2015-08-03T20:56:38.36Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11980000 2015-08-03T20:56:38.632Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10900000 QueuedCount 2268 2015-08-03T20:56:38.763Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20489996, 10744998, 0 committed:false) SnapCreateDur 12.077µs SnapOpenDur 81.772333ms 2015-08-03T20:56:39.404Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11990000 2015-08-03T20:56:39.573Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.563µs CommitTime 715.4247ms TotalFlushTime 13m20.679139979s TotalCommitTime 3m32.715592398s 2015-08-03T20:56:39.573Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20493712, 10746856, 331 committed:true) SnapCreateDur 745.778641ms SnapOpenDur 46.507µs 2015-08-03T20:56:39.887Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10910000 QueuedCount 5631 2015-08-03T20:56:40.169Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20516254, 10758127, 0 committed:false) SnapCreateDur 25.73µs SnapOpenDur 80.548196ms 2015-08-03T20:56:40.441Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12000000 2015-08-03T20:56:41.264Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10920000 QueuedCount 7087 2015-08-03T20:56:41.48Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12010000 2015-08-03T20:56:41.537Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20538812, 10769406, 0 committed:false) SnapCreateDur 24.517µs SnapOpenDur 143.962813ms 2015-08-03T20:56:41.671Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10930000 QueuedCount 6700 2015-08-03T20:56:42.516Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12020000 2015-08-03T20:56:43.56Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12030000 2015-08-03T20:56:44.6Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12040000 2015-08-03T20:56:44.661Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20561566, 10780783, 0 committed:false) SnapCreateDur 28.994µs SnapOpenDur 141.954978ms 2015-08-03T20:56:44.736Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10940000 QueuedCount 572 2015-08-03T20:56:45.453Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.027µs CommitTime 658.766395ms TotalFlushTime 13m25.359781224s TotalCommitTime 3m33.374358793s 2015-08-03T20:56:45.453Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20565384, 10782692, 332 committed:true) SnapCreateDur 687.721523ms SnapOpenDur 32.954µs 2015-08-03T20:56:45.634Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12050000 2015-08-03T20:56:45.8Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10950000 QueuedCount 2101 2015-08-03T20:56:45.952Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20588074, 10794037, 0 committed:false) SnapCreateDur 22.454µs SnapOpenDur 77.970484ms 2015-08-03T20:56:46.683Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12060000 2015-08-03T20:56:47.733Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12070000 2015-08-03T20:56:48.774Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12080000 2015-08-03T20:56:48.993Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10960000 QueuedCount 3643 2015-08-03T20:56:49.295Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20610794, 10805397, 0 committed:false) SnapCreateDur 21.716µs SnapOpenDur 149.702246ms 2015-08-03T20:56:49.811Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12090000 2015-08-03T20:56:50.345Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.87µs CommitTime 926.958303ms TotalFlushTime 13m28.96579788s TotalCommitTime 3m34.301317096s 2015-08-03T20:56:50.345Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20614560, 10807280, 333 committed:true) SnapCreateDur 955.936829ms SnapOpenDur 28.295µs 2015-08-03T20:56:50.527Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10970000 QueuedCount 7018 2015-08-03T20:56:50.852Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20637098, 10818549, 0 committed:false) SnapCreateDur 21.043µs SnapOpenDur 74.850741ms 2015-08-03T20:56:50.862Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12100000 2015-08-03T20:56:50.993Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10980000 QueuedCount 6966 2015-08-03T20:56:51.89Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12110000 2015-08-03T20:56:52.353Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20660012, 10830006, 0 committed:false) SnapCreateDur 14.005µs SnapOpenDur 87.424284ms 2015-08-03T20:56:52.403Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10990000 QueuedCount 576 2015-08-03T20:56:52.933Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12120000 2015-08-03T20:56:53.367Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.963µs CommitTime 918.379756ms TotalFlushTime 13m30.777681624s TotalCommitTime 3m35.219696852s 2015-08-03T20:56:53.368Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20663826, 10831913, 334 committed:true) SnapCreateDur 950.207523ms SnapOpenDur 33.906µs 2015-08-03T20:56:53.736Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11000000 QueuedCount 2096 2015-08-03T20:56:53.886Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20686506, 10843253, 0 committed:false) SnapCreateDur 23.62µs SnapOpenDur 75.030715ms 2015-08-03T20:56:53.969Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12130000 2015-08-03T20:56:55.012Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12140000 2015-08-03T20:56:55.148Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11010000 QueuedCount 3443 2015-08-03T20:56:55.383Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20708806, 10854403, 0 committed:false) SnapCreateDur 24.57µs SnapOpenDur 144.086995ms 2015-08-03T20:56:56.052Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12150000 2015-08-03T20:56:56.394Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 13.474µs CommitTime 878.940316ms TotalFlushTime 13m32.569259165s TotalCommitTime 3m36.098637168s 2015-08-03T20:56:56.394Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20712564, 10856282, 335 committed:true) SnapCreateDur 906.111602ms SnapOpenDur 34.008µs 2015-08-03T20:56:56.581Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11020000 QueuedCount 6905 2015-08-03T20:56:56.906Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20735330, 10867665, 0 committed:false) SnapCreateDur 22.58µs SnapOpenDur 77.661389ms 2015-08-03T20:56:57.102Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12160000 2015-08-03T20:56:57.819Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11030000 QueuedCount 8453 2015-08-03T20:56:58.136Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12170000 2015-08-03T20:56:59.041Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20758036, 10879018, 0 committed:false) SnapCreateDur 25.826µs SnapOpenDur 164.055669ms 2015-08-03T20:56:59.12Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11040000 QueuedCount 332 2015-08-03T20:56:59.183Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12180000 2015-08-03T20:56:59.997Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.68µs CommitTime 834.815826ms TotalFlushTime 13m34.975623439s TotalCommitTime 3m36.933452994s 2015-08-03T20:56:59.998Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20761728, 10880864, 336 committed:true) SnapCreateDur 862.551184ms SnapOpenDur 43.498µs 2015-08-03T20:57:00.216Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12190000 2015-08-03T20:57:00.354Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11050000 QueuedCount 1914 2015-08-03T20:57:00.478Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20784538, 10892269, 0 committed:false) SnapCreateDur 17.783µs SnapOpenDur 58.08661ms 2015-08-03T20:57:01.261Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12200000 2015-08-03T20:57:02.299Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12210000 2015-08-03T20:57:02.846Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 2015-08-03T20:57:02.991Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11060000 QueuedCount 3474 2015-08-03T20:57:03.176Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20807260, 10903630, 0 committed:false) SnapCreateDur 25.723µs SnapOpenDur 55.683671ms 2015-08-03T20:57:03.338Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12220000 2015-08-03T20:57:04.379Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12230000 2015-08-03T20:57:05.111Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 9.783µs CommitTime 567.665907ms TotalFlushTime 13m38.038156413s TotalCommitTime 3m37.501118901s 2015-08-03T20:57:05.111Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20810964, 10905482, 337 committed:true) SnapCreateDur 1.840009372s SnapOpenDur 26.682µs 2015-08-03T20:57:05.221Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":23481064892,"bucket-1:by_email:avg_scan_wait_latency":23480138115,"bucket-1:by_email:avg_ts_interval":1853488919,"bucket-1:by_email:avg_ts_items_count":6027,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":530807066,"bucket-1:by_email:delete_bytes":425534622,"bucket-1:by_email:disk_size":2919714816,"bucket-1:by_email:flush_queue_size":5248,"bucket-1:by_email:frag_percent":81,"bucket-1:by_email:get_bytes":433880406,"bucket-1:by_email:insert_bytes":1144096170,"bucket-1:by_email:items_count":1000001,"bucket-1:by_email:num_commits":337,"bucket-1:by_email:num_compactions":17,"bucket-1:by_email:num_completed_requests":16899,"bucket-1:by_email:num_docs_indexed":11064674,"bucket-1:by_email:num_docs_pending":260,"bucket-1:by_email:num_docs_queued":1139139,"bucket-1:by_email:num_flush_queued":11073051,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":17283,"bucket-1:by_email:num_rows_returned":286,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":1302,"bucket-1:by_email:scan_bytes_read":11440,"bucket-1:by_email:scan_wait_duration":405807227055501,"bucket-1:by_email:since_last_snapshot":1934497260,"bucket-1:by_email:timings/storage_clone_handle":"16899 960778101 2750320027977669","bucket-1:by_email:timings/storage_commit":"337 217501118901 -3190905518215897449","bucket-1:by_email:timings/storage_del":"9907904 177426274125 2081428077278663973","bucket-1:by_email:timings/storage_get":"11067805 163789906007 -970198456043681113","bucket-1:by_email:timings/storage_handle_open":"337 20894439803 -7909550485277930289","bucket-1:by_email:timings/storage_iterator_next":"286 5931229 171965816823","bucket-1:by_email:timings/storage_new_iterator":"16899 1391753967 2867322690242565","bucket-1:by_email:timings/storage_set":"21815807 462678187136 2421182997701530324","bucket-1:by_email:timings/storage_snapshot_close":"965 15838205479 935240310638453989","bucket-1:by_email:timings/storage_snapshot_create":"965 94104421386 -4665893291789258288","bucket-1:by_email:total_scan_duration":405823244545207,"bucket-1:mutation_queue_size":1165063,"bucket-1:num_mutations_queued":12238114,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":606,"memory_quota":104857600000,"memory_used":2284433408,"needs_restart":false,"num_connections":384,"timings/stats_response":"1860 648768416 4828843873695474"} 2015-08-03T20:57:05.299Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11070000 QueuedCount 6890 2015-08-03T20:57:05.428Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12240000 2015-08-03T20:57:05.611Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20833668, 10916834, 0 committed:false) SnapCreateDur 23.523µs SnapOpenDur 77.300343ms 2015-08-03T20:57:05.756Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11080000 QueuedCount 7982 2015-08-03T20:57:06.511Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12250000 2015-08-03T20:57:06.961Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20855484, 10927742, 0 committed:false) SnapCreateDur 15.753µs SnapOpenDur 87.452403ms 2015-08-03T20:57:07.591Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12260000 2015-08-03T20:57:07.929Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 16.126µs CommitTime 877.128879ms TotalFlushTime 13m39.690758789s TotalCommitTime 3m38.37824778s 2015-08-03T20:57:07.929Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20859404, 10929702, 338 committed:true) SnapCreateDur 904.489044ms SnapOpenDur 39.904µs 2015-08-03T20:57:07.953Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11090000 QueuedCount 1471 2015-08-03T20:57:08.318Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11100000 QueuedCount 703 2015-08-03T20:57:08.434Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20880534, 10940267, 0 committed:false) SnapCreateDur 27.035µs SnapOpenDur 82.482945ms 2015-08-03T20:57:08.637Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12270000 2015-08-03T20:57:09.702Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12280000 2015-08-03T20:57:09.751Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11110000 QueuedCount 1804 2015-08-03T20:57:09.872Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20902436, 10951218, 0 committed:false) SnapCreateDur 13.342µs SnapOpenDur 81.725026ms 2015-08-03T20:57:10.054Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11120000 QueuedCount 1374 2015-08-03T20:57:10.74Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12290000 2015-08-03T20:57:10.9Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.416µs CommitTime 251.598782ms TotalFlushTime 13m42.107871573s TotalCommitTime 3m38.629846562s 2015-08-03T20:57:10.9Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20921250, 10960625, 339 committed:true) SnapCreateDur 278.722192ms SnapOpenDur 57.814µs 2015-08-03T20:57:11.233Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11130000 QueuedCount 2967 2015-08-03T20:57:11.431Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20944078, 10972039, 0 committed:false) SnapCreateDur 31.83µs SnapOpenDur 88.41312ms 2015-08-03T20:57:11.777Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12300000 2015-08-03T20:57:12.7Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11140000 QueuedCount 4394 2015-08-03T20:57:12.816Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12310000 2015-08-03T20:57:12.885Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20966594, 10983297, 0 committed:false) SnapCreateDur 31.663µs SnapOpenDur 89.17853ms 2015-08-03T20:57:13.115Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11150000 QueuedCount 4075 2015-08-03T20:57:13.859Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12320000 2015-08-03T20:57:14.58Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.147µs CommitTime 367.935291ms TotalFlushTime 13m45.102629346s TotalCommitTime 3m38.997781853s 2015-08-03T20:57:14.58Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 20985594, 10992797, 340 committed:true) SnapCreateDur 395.962934ms SnapOpenDur 49.34µs 2015-08-03T20:57:14.823Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11160000 QueuedCount 5589 2015-08-03T20:57:14.908Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12330000 2015-08-03T20:57:15.111Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21008252, 11004126, 0 committed:false) SnapCreateDur 26.32µs SnapOpenDur 86.158681ms 2015-08-03T20:57:15.936Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12340000 2015-08-03T20:57:16.253Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11170000 QueuedCount 7169 2015-08-03T20:57:16.545Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21031058, 11015529, 0 committed:false) SnapCreateDur 25.417µs SnapOpenDur 159.097138ms 2015-08-03T20:57:16.985Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12350000 2015-08-03T20:57:17.525Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 16.38µs CommitTime 877.250743ms TotalFlushTime 13m46.798387939s TotalCommitTime 3m39.875032596s 2015-08-03T20:57:17.525Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21034858, 11017429, 341 committed:true) SnapCreateDur 905.284935ms SnapOpenDur 42.684µs 2015-08-03T20:57:17.568Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11180000 QueuedCount 1000 2015-08-03T20:57:17.808Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11190000 QueuedCount 571 2015-08-03T20:57:17.917Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21057376, 11028688, 0 committed:false) SnapCreateDur 30.297µs SnapOpenDur 89.795967ms 2015-08-03T20:57:18.036Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12360000 2015-08-03T20:57:19.08Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12370000 2015-08-03T20:57:19.246Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11200000 QueuedCount 1995 2015-08-03T20:57:19.447Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21079824, 11039912, 0 committed:false) SnapCreateDur 22.273µs SnapOpenDur 158.762535ms 2015-08-03T20:57:19.792Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11210000 QueuedCount 3502 2015-08-03T20:57:20.126Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12380000 2015-08-03T20:57:21.161Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12390000 2015-08-03T20:57:21.361Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.51µs CommitTime 541.903965ms TotalFlushTime 13m49.697220165s TotalCommitTime 3m40.416936561s 2015-08-03T20:57:21.362Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21102492, 11051246, 342 committed:true) SnapCreateDur 571.01942ms SnapOpenDur 45.086µs 2015-08-03T20:57:21.635Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11220000 QueuedCount 5039 2015-08-03T20:57:21.915Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21125198, 11062599, 0 committed:false) SnapCreateDur 37.39µs SnapOpenDur 88.226583ms 2015-08-03T20:57:22.21Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12400000 2015-08-03T20:57:23.145Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11230000 QueuedCount 6626 2015-08-03T20:57:23.24Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12410000 2015-08-03T20:57:23.453Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21147994, 11073997, 0 committed:false) SnapCreateDur 27.117µs SnapOpenDur 163.344064ms 2015-08-03T20:57:24.286Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12420000 2015-08-03T20:57:24.484Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.284µs CommitTime 898.036427ms TotalFlushTime 13m51.529147569s TotalCommitTime 3m41.314972988s 2015-08-03T20:57:24.485Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21151720, 11075860, 343 committed:true) SnapCreateDur 926.809809ms SnapOpenDur 41.765µs 2015-08-03T20:57:24.539Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11240000 QueuedCount 2366 2015-08-03T20:57:24.759Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11250000 QueuedCount 49 2015-08-03T20:57:24.855Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21174380, 11087190, 0 committed:false) SnapCreateDur 22.31µs SnapOpenDur 86.100263ms 2015-08-03T20:57:25.36Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12430000 2015-08-03T20:57:26.231Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11260000 QueuedCount 1573 2015-08-03T20:57:26.365Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12440000 2015-08-03T20:57:26.426Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21197076, 11098538, 0 committed:false) SnapCreateDur 31.097µs SnapOpenDur 156.259084ms 2015-08-03T20:57:27.414Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12450000 2015-08-03T20:57:27.745Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.61µs CommitTime 1.018248342s TotalFlushTime 13m53.390180372s TotalCommitTime 3m42.33322133s 2015-08-03T20:57:27.746Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21208462, 11104231, 344 committed:true) SnapCreateDur 1.047228264s SnapOpenDur 27.488µs 2015-08-03T20:57:27.82Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11270000 QueuedCount 3116 2015-08-03T20:57:28.056Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21231274, 11115637, 0 committed:false) SnapCreateDur 33.154µs SnapOpenDur 77.095966ms 2015-08-03T20:57:28.096Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11280000 QueuedCount 822 2015-08-03T20:57:28.455Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12460000 2015-08-03T20:57:29.035Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11290000 QueuedCount 418 2015-08-03T20:57:29.187Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21253910, 11126955, 0 committed:false) SnapCreateDur 22.047µs SnapOpenDur 138.275645ms 2015-08-03T20:57:29.494Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12470000 2015-08-03T20:57:30.201Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11300000 QueuedCount 1883 2015-08-03T20:57:30.41Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21276492, 11138246, 0 committed:false) SnapCreateDur 24.17µs SnapOpenDur 111.345845ms 2015-08-03T20:57:30.536Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12480000 2015-08-03T20:57:31.107Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 2.698µs CommitTime 571.955141ms TotalFlushTime 13m55.696320845s TotalCommitTime 3m42.905176471s 2015-08-03T20:57:31.107Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21280208, 11140104, 345 committed:true) SnapCreateDur 590.943187ms SnapOpenDur 60.125µs 2015-08-03T20:57:31.351Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11310000 QueuedCount 5307 2015-08-03T20:57:31.582Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12490000 2015-08-03T20:57:31.656Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21302908, 11151454, 0 committed:false) SnapCreateDur 32.484µs SnapOpenDur 103.452101ms 2015-08-03T20:57:32.624Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12500000 2015-08-03T20:57:32.846Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:456048122, Disk:2060361728, ExtraSnapshotData:0, Fragmentation:77.86562836018666% 2015-08-03T20:57:32.846Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:57:32.869Z-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 21034858. Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T20:57:33.206Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11320000 QueuedCount 6843 2015-08-03T20:57:33.537Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21325636, 11162818, 0 committed:false) SnapCreateDur 31.547µs SnapOpenDur 63.754047ms 2015-08-03T20:57:33.657Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12510000 2015-08-03T20:57:34.69Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12520000 2015-08-03T20:57:34.72Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 25.866µs CommitTime 1.048496245s TotalFlushTime 13m57.94976069s TotalCommitTime 3m43.953672716s 2015-08-03T20:57:34.72Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21329294, 11164647, 346 committed:true) SnapCreateDur 1.077908443s SnapOpenDur 44.356µs 2015-08-03T20:57:34.787Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11330000 QueuedCount 2600 2015-08-03T20:57:35.152Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11340000 QueuedCount 313 2015-08-03T20:57:35.248Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21352144, 11176072, 0 committed:false) SnapCreateDur 18.975µs SnapOpenDur 83.553422ms 2015-08-03T20:57:35.733Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12530000 2015-08-03T20:57:36.724Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11350000 QueuedCount 1822 2015-08-03T20:57:36.783Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12540000 2015-08-03T20:57:36.865Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21374764, 11187382, 0 committed:false) SnapCreateDur 19.533µs SnapOpenDur 93.478328ms 2015-08-03T20:57:37.833Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12550000 2015-08-03T20:57:37.874Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.235µs CommitTime 878.100622ms TotalFlushTime 13m59.913666514s TotalCommitTime 3m44.831773338s 2015-08-03T20:57:37.874Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21378556, 11189278, 347 committed:true) SnapCreateDur 905.270081ms SnapOpenDur 32.962µs 2015-08-03T20:57:38.031Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11360000 QueuedCount 5338 2015-08-03T20:57:38.23Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21401352, 11200676, 0 committed:false) SnapCreateDur 22.11µs SnapOpenDur 84.216164ms 2015-08-03T20:57:38.874Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12560000 2015-08-03T20:57:39.049Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11370000 QueuedCount 6807 2015-08-03T20:57:39.326Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21423906, 11211953, 0 committed:false) SnapCreateDur 29.364µs SnapOpenDur 143.157617ms 2015-08-03T20:57:39.48Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11380000 QueuedCount 8261 2015-08-03T20:57:39.905Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12570000 2015-08-03T20:57:40.845Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21446404, 11223202, 0 committed:false) SnapCreateDur 12.27µs SnapOpenDur 128.457044ms 2015-08-03T20:57:40.933Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11390000 QueuedCount 209 2015-08-03T20:57:40.958Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12580000 2015-08-03T20:57:41.678Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 7.934µs CommitTime 701.330569ms TotalFlushTime 14m2.495174289s TotalCommitTime 3m45.533103907s 2015-08-03T20:57:41.989Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12590000 2015-08-03T20:57:43.032Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12600000 2015-08-03T20:57:43.081Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21450258, 11225129, 348 committed:true) SnapCreateDur 728.581637ms SnapOpenDur 1.402515129s 2015-08-03T20:57:43.451Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11400000 QueuedCount 1658 2015-08-03T20:57:43.598Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21472846, 11236423, 0 committed:false) SnapCreateDur 22.8µs SnapOpenDur 85.3752ms 2015-08-03T20:57:44.065Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12610000 2015-08-03T20:57:44.796Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11410000 QueuedCount 3108 2015-08-03T20:57:45.011Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21495344, 11247672, 0 committed:false) SnapCreateDur 31.32µs SnapOpenDur 150.443508ms 2015-08-03T20:57:45.105Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12620000 2015-08-03T20:57:45.966Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 5.426µs CommitTime 852.848597ms TotalFlushTime 14m4.163992875s TotalCommitTime 3m46.385952504s 2015-08-03T20:57:45.966Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21499298, 11249649, 349 committed:true) SnapCreateDur 880.626965ms SnapOpenDur 36.366µs 2015-08-03T20:57:46.146Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12630000 2015-08-03T20:57:46.159Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11420000 QueuedCount 6644 2015-08-03T20:57:46.478Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21522062, 11261031, 0 committed:false) SnapCreateDur 24.583µs SnapOpenDur 80.170907ms 2015-08-03T20:57:46.603Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11430000 QueuedCount 6271 2015-08-03T20:57:47.198Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12640000 2015-08-03T20:57:47.858Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21544760, 11272380, 0 committed:false) SnapCreateDur 21.58µs SnapOpenDur 88.367761ms 2015-08-03T20:57:47.917Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11440000 QueuedCount 2032 2015-08-03T20:57:48.24Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12650000 2015-08-03T20:57:48.777Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21567450, 11283725, 0 committed:false) SnapCreateDur 10.938µs SnapOpenDur 75.580852ms 2015-08-03T20:57:48.81Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11450000 QueuedCount 1672 2015-08-03T20:57:49.286Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12660000 2015-08-03T20:57:49.596Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.743µs CommitTime 688.182238ms TotalFlushTime 14m6.694168404s TotalCommitTime 3m47.074134742s 2015-08-03T20:57:49.596Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21571302, 11285651, 350 committed:true) SnapCreateDur 715.229053ms SnapOpenDur 55.09µs 2015-08-03T20:57:49.912Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11460000 QueuedCount 3185 2015-08-03T20:57:50.112Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21593990, 11296995, 0 committed:false) SnapCreateDur 20.69µs SnapOpenDur 81.772755ms 2015-08-03T20:57:50.326Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12670000 2015-08-03T20:57:51.272Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11470000 QueuedCount 4747 2015-08-03T20:57:51.363Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12680000 2015-08-03T20:57:51.914Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21616744, 11308372, 0 committed:false) SnapCreateDur 23.617µs SnapOpenDur 148.491681ms 2015-08-03T20:57:52.402Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12690000 2015-08-03T20:57:52.921Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.99µs CommitTime 875.769389ms TotalFlushTime 14m8.780577315s TotalCommitTime 3m47.949904131s 2015-08-03T20:57:52.921Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21620612, 11310306, 351 committed:true) SnapCreateDur 903.511102ms SnapOpenDur 28.578µs 2015-08-03T20:57:53.061Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11480000 QueuedCount 7450 2015-08-03T20:57:53.426Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21642894, 11321447, 0 committed:false) SnapCreateDur 17.964µs SnapOpenDur 84.43809ms 2015-08-03T20:57:53.435Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12700000 2015-08-03T20:57:53.528Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11490000 QueuedCount 5681 2015-08-03T20:57:54.48Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12710000 2015-08-03T20:57:55.514Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12720000 2015-08-03T20:57:56.44Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21665564, 11332782, 0 committed:false) SnapCreateDur 28.797µs SnapOpenDur 151.023256ms 2015-08-03T20:57:56.479Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11500000 QueuedCount 1461 2015-08-03T20:57:56.565Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12730000 2015-08-03T20:57:57.423Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 11.897µs CommitTime 859.995282ms TotalFlushTime 14m12.058013305s TotalCommitTime 3m48.809899413s 2015-08-03T20:57:57.423Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21669336, 11334668, 352 committed:true) SnapCreateDur 888.101403ms SnapOpenDur 36.64µs 2015-08-03T20:57:57.609Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12740000 2015-08-03T20:57:57.748Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11510000 QueuedCount 3046 2015-08-03T20:57:57.945Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21692160, 11346080, 0 committed:false) SnapCreateDur 20.389µs SnapOpenDur 79.096286ms 2015-08-03T20:57:58.643Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12750000 2015-08-03T20:57:59.176Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11520000 QueuedCount 4567 2015-08-03T20:57:59.43Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21714860, 11357430, 0 committed:false) SnapCreateDur 35.073µs SnapOpenDur 151.458446ms 2015-08-03T20:57:59.688Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12760000 2015-08-03T20:58:00.628Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 13.99µs CommitTime 993.327829ms TotalFlushTime 14m13.896913796s TotalCommitTime 3m49.803227242s 2015-08-03T20:58:00.628Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21722472, 11361236, 353 committed:true) SnapCreateDur 1.021703092s SnapOpenDur 39.483µs 2015-08-03T20:58:00.695Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11530000 QueuedCount 2305 2015-08-03T20:58:00.955Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21745194, 11372597, 0 committed:false) SnapCreateDur 18.554µs SnapOpenDur 43.11531ms 2015-08-03T20:58:00.977Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11540000 QueuedCount 636 2015-08-03T20:58:02.252Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11550000 QueuedCount 1555 2015-08-03T20:58:02.373Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21767998, 11383999, 0 committed:false) SnapCreateDur 14.101µs SnapOpenDur 84.252039ms 2015-08-03T20:58:03.094Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11560000 QueuedCount 1164 2015-08-03T20:58:03.301Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 3.899µs CommitTime 170.280674ms TotalFlushTime 14m16.157640363s TotalCommitTime 3m49.973507916s 2015-08-03T20:58:03.302Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21786906, 11393453, 354 committed:true) SnapCreateDur 180.602119ms SnapOpenDur 24.521µs 2015-08-03T20:58:03.642Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11570000 QueuedCount 2769 2015-08-03T20:58:03.819Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21809768, 11404884, 0 committed:false) SnapCreateDur 22.977µs SnapOpenDur 75.94515ms 2015-08-03T20:58:05.024Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11580000 QueuedCount 4130 2015-08-03T20:58:05.198Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21832154, 11416077, 0 committed:false) SnapCreateDur 24.58µs SnapOpenDur 87.636743ms 2015-08-03T20:58:05.222Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":25627285630,"bucket-1:by_email:avg_scan_wait_latency":25626360730,"bucket-1:by_email:avg_ts_interval":1071984641,"bucket-1:by_email:avg_ts_items_count":11086,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":497985967,"bucket-1:by_email:delete_bytes":446193542,"bucket-1:by_email:disk_size":2695065600,"bucket-1:by_email:flush_queue_size":1579,"bucket-1:by_email:frag_percent":81,"bucket-1:by_email:get_bytes":454905084,"bucket-1:by_email:insert_bytes":1194542327,"bucket-1:by_email:items_count":1000001,"bucket-1:by_email:num_commits":354,"bucket-1:by_email:num_compactions":18,"bucket-1:by_email:num_completed_requests":17283,"bucket-1:by_email:num_docs_indexed":11552775,"bucket-1:by_email:num_docs_pending":0,"bucket-1:by_email:num_docs_queued":1218225,"bucket-1:by_email:num_flush_queued":11585750,"bucket-1:by_email:num_last_snapshot_reply":0,"bucket-1:by_email:num_requests":17667,"bucket-1:by_email:num_rows_returned":294,"bucket-1:by_email:num_snapshot_waiters":384,"bucket-1:by_email:num_snapshots":1358,"bucket-1:by_email:scan_bytes_read":11760,"bucket-1:by_email:scan_wait_duration":452740915029484,"bucket-1:by_email:since_last_snapshot":1379321838,"bucket-1:by_email:timings/storage_clone_handle":"17283 980623472 2751433338218378","bucket-1:by_email:timings/storage_commit":"354 229973507916 7096263203353563930","bucket-1:by_email:timings/storage_del":"10416113 187239595443 8469023437077803271","bucket-1:by_email:timings/storage_get":"11584167 171546488638 -216306729293891424","bucket-1:by_email:timings/storage_handle_open":"354 22297548368 -5942521109238901726","bucket-1:by_email:timings/storage_iterator_next":"294 6041810 173622442512","bucket-1:by_email:timings/storage_new_iterator":"17283 1422456896 2870504463394916","bucket-1:by_email:timings/storage_set":"22832226 484155930448 -495200455440674568","bucket-1:by_email:timings/storage_snapshot_close":"1003 16462208597 947799827428984511","bucket-1:by_email:timings/storage_snapshot_create":"1004 98092447847 -4218582997073918265","bucket-1:by_email:total_scan_duration":452757255236391,"bucket-1:mutation_queue_size":1182024,"bucket-1:num_mutations_queued":12767808,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":617,"memory_quota":104857600000,"memory_used":1292132352,"needs_restart":false,"num_connections":384,"timings/stats_response":"1932 672490540 4836856654498616"} 2015-08-03T20:58:05.439Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11590000 QueuedCount 5616 2015-08-03T20:58:06.451Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21854742, 11427371, 0 committed:false) SnapCreateDur 13.464µs SnapOpenDur 71.684324ms 2015-08-03T20:58:07.231Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 12.773µs CommitTime 658.684537ms TotalFlushTime 14m19.019264345s TotalCommitTime 3m50.632192453s 2015-08-03T20:58:07.231Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21858472, 11429236, 355 committed:true) SnapCreateDur 685.962811ms SnapOpenDur 27.87µs 2015-08-03T20:58:07.313Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11600000 QueuedCount 3294 2015-08-03T20:58:08.432Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21881218, 11440609, 0 committed:false) SnapCreateDur 26.086µs SnapOpenDur 80.68686ms 2015-08-03T20:58:08.488Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11610000 QueuedCount 2920 2015-08-03T20:58:09.726Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11620000 QueuedCount 540 2015-08-03T20:58:09.835Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21903746, 11451873, 0 committed:false) SnapCreateDur 23.582µs SnapOpenDur 90.667714ms 2015-08-03T20:58:10.859Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 18.18µs CommitTime 922.622175ms TotalFlushTime 14m21.42328s TotalCommitTime 3m51.554814628s 2015-08-03T20:58:10.859Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21907622, 11453811, 356 committed:true) SnapCreateDur 949.833391ms SnapOpenDur 32.163µs 2015-08-03T20:58:11.049Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11630000 QueuedCount 4048 2015-08-03T20:58:11.176Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21930344, 11465172, 0 committed:false) SnapCreateDur 10.078µs SnapOpenDur 45.269083ms 2015-08-03T20:58:12.279Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11640000 QueuedCount 5583 2015-08-03T20:58:12.464Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21953028, 11476514, 0 committed:false) SnapCreateDur 11.224µs SnapOpenDur 84.920574ms 2015-08-03T20:58:12.575Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11650000 QueuedCount 3258 2015-08-03T20:58:13.475Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21975722, 11487861, 0 committed:false) SnapCreateDur 38.143µs SnapOpenDur 129.841519ms 2015-08-03T20:58:14.142Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 3.477µs CommitTime 552.632528ms TotalFlushTime 14m23.75800538s TotalCommitTime 3m52.107447156s 2015-08-03T20:58:14.142Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 21979490, 11489745, 357 committed:true) SnapCreateDur 572.314145ms SnapOpenDur 47.343µs 2015-08-03T20:58:14.197Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11660000 QueuedCount 2891 2015-08-03T20:58:14.558Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11670000 QueuedCount 131 2015-08-03T20:58:14.637Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22001284, 11500642, 0 committed:false) SnapCreateDur 20.083µs SnapOpenDur 72.802047ms 2015-08-03T20:58:17.111Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11680000 QueuedCount 1015 2015-08-03T20:58:17.254Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22022712, 11511356, 0 committed:false) SnapCreateDur 21.88µs SnapOpenDur 105.821018ms 2015-08-03T20:58:17.856Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 6.088µs CommitTime 513.82516ms TotalFlushTime 14m26.670800818s TotalCommitTime 3m52.621272316s 2015-08-03T20:58:17.856Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22026446, 11513223, 358 committed:true) SnapCreateDur 527.387426ms SnapOpenDur 31.127µs 2015-08-03T20:58:18.017Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11690000 QueuedCount 4145 2015-08-03T20:58:18.14Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22048564, 11524282, 0 committed:false) SnapCreateDur 15.623µs SnapOpenDur 41.987148ms 2015-08-03T20:58:19.282Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11700000 QueuedCount 5641 2015-08-03T20:58:19.49Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22071174, 11535587, 0 committed:false) SnapCreateDur 16.316µs SnapOpenDur 87.221516ms 2015-08-03T20:58:19.679Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11710000 QueuedCount 7175 2015-08-03T20:58:22.496Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 2015-08-03T20:58:22.622Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22093882, 11546941, 0 committed:false) SnapCreateDur 39.924µs SnapOpenDur 18.653384ms 2015-08-03T20:58:24.173Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.43µs CommitTime 321.686766ms TotalFlushTime 14m31.265004851s TotalCommitTime 3m52.942959082s 2015-08-03T20:58:24.174Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22097642, 11548821, 359 committed:true) SnapCreateDur 1.446876772s SnapOpenDur 41.599µs 2015-08-03T20:58:24.226Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11720000 QueuedCount 2935 2015-08-03T20:58:24.581Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11730000 QueuedCount 635 2015-08-03T20:58:24.667Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22120420, 11560210, 0 committed:false) SnapCreateDur 28.307µs SnapOpenDur 59.999357ms 2015-08-03T20:58:26.338Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11740000 QueuedCount 2175 2015-08-03T20:58:26.487Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22143152, 11571576, 0 committed:false) SnapCreateDur 15.475µs SnapOpenDur 100.386843ms 2015-08-03T20:58:27.127Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 32.79µs CommitTime 551.641086ms TotalFlushTime 14m33.392283216s TotalCommitTime 3m53.494600168s 2015-08-03T20:58:27.128Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22146988, 11573494, 360 committed:true) SnapCreateDur 566.10421ms SnapOpenDur 25.791µs 2015-08-03T20:58:27.263Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11750000 QueuedCount 3804 2015-08-03T20:58:27.405Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22169806, 11584903, 0 committed:false) SnapCreateDur 11.767µs SnapOpenDur 36.411748ms 2015-08-03T20:58:28.587Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11760000 QueuedCount 7224 2015-08-03T20:58:28.834Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22192488, 11596244, 0 committed:false) SnapCreateDur 14.255µs SnapOpenDur 88.938954ms 2015-08-03T20:58:28.915Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11770000 QueuedCount 2921 2015-08-03T20:58:29.791Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22215002, 11607501, 0 committed:false) SnapCreateDur 15.678µs SnapOpenDur 75.769548ms 2015-08-03T20:58:29.836Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11780000 QueuedCount 574 2015-08-03T20:58:30.148Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 1.68µs CommitTime 292.242038ms TotalFlushTime 14m35.787061426s TotalCommitTime 3m53.786842206s 2015-08-03T20:58:30.148Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22218770, 11609385, 361 committed:true) SnapCreateDur 302.490407ms SnapOpenDur 34.742µs 2015-08-03T20:58:30.487Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11790000 QueuedCount 2057 2015-08-03T20:58:30.661Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22241342, 11620671, 0 committed:false) SnapCreateDur 27.26µs SnapOpenDur 92.796525ms 2015-08-03T20:58:31.823Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11800000 QueuedCount 3608 2015-08-03T20:58:31.984Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22264054, 11632027, 0 committed:false) SnapCreateDur 16.845µs SnapOpenDur 88.584607ms 2015-08-03T20:58:32.142Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11810000 QueuedCount 5058 2015-08-03T20:58:32.824Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22286570, 11643285, 0 committed:false) SnapCreateDur 23.15µs SnapOpenDur 67.508579ms 2015-08-03T20:58:33.18Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 1.617µs CommitTime 291.306537ms TotalFlushTime 14m38.136694837s TotalCommitTime 3m54.078148743s 2015-08-03T20:58:33.18Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22290498, 11645249, 362 committed:true) SnapCreateDur 301.569513ms SnapOpenDur 23.515µs 2015-08-03T20:58:33.246Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11820000 QueuedCount 879 2015-08-03T20:58:33.414Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22313146, 11656573, 0 committed:false) SnapCreateDur 13.678µs SnapOpenDur 33.27737ms 2015-08-03T20:58:33.486Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11830000 QueuedCount 4346 2015-08-03T20:58:34.763Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11840000 QueuedCount 82 2015-08-03T20:58:34.855Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22335850, 11667925, 0 committed:false) SnapCreateDur 15.595µs SnapOpenDur 89.699842ms 2015-08-03T20:58:36.195Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11850000 QueuedCount 1609 2015-08-03T20:58:36.32Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22358558, 11679279, 0 committed:false) SnapCreateDur 25.75µs SnapOpenDur 83.155602ms 2015-08-03T20:58:37.278Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 17.676µs CommitTime 836.587746ms TotalFlushTime 14m41.040499217s TotalCommitTime 3m54.914736489s 2015-08-03T20:58:37.279Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22362342, 11681171, 363 committed:true) SnapCreateDur 864.549368ms SnapOpenDur 44.914µs 2015-08-03T20:58:37.539Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11860000 QueuedCount 5053 2015-08-03T20:58:37.826Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22385006, 11692503, 0 committed:false) SnapCreateDur 28.227µs SnapOpenDur 92.583205ms 2015-08-03T20:58:39.086Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11870000 QueuedCount 6504 2015-08-03T20:58:39.321Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22407520, 11703760, 0 committed:false) SnapCreateDur 14.096µs SnapOpenDur 91.349855ms 2015-08-03T20:58:40.357Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 16.974µs CommitTime 944.416225ms TotalFlushTime 14m42.858512632s TotalCommitTime 3m55.859152714s 2015-08-03T20:58:40.357Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22411318, 11705659, 364 committed:true) SnapCreateDur 971.845446ms SnapOpenDur 45.844µs 2015-08-03T20:58:40.435Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11880000 QueuedCount 4205 2015-08-03T20:58:40.873Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22433728, 11716864, 0 committed:false) SnapCreateDur 27.617µs SnapOpenDur 82.686227ms 2015-08-03T20:58:40.903Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11890000 QueuedCount 1884 2015-08-03T20:58:42.226Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11900000 QueuedCount 1511 2015-08-03T20:58:42.428Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22456708, 11728354, 0 committed:false) SnapCreateDur 33.973µs SnapOpenDur 161.132118ms 2015-08-03T20:58:43.441Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 14.12µs CommitTime 880.73634ms TotalFlushTime 14m44.683168391s TotalCommitTime 3m56.739889054s 2015-08-03T20:58:43.441Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22460484, 11730242, 365 committed:true) SnapCreateDur 908.602953ms SnapOpenDur 38.053µs 2015-08-03T20:58:43.634Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11910000 QueuedCount 4972 2015-08-03T20:58:43.909Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22483204, 11741602, 0 committed:false) SnapCreateDur 26.03µs SnapOpenDur 85.181478ms 2015-08-03T20:58:45.137Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11920000 QueuedCount 6539 2015-08-03T20:58:45.363Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22505942, 11752971, 0 committed:false) SnapCreateDur 14.36µs SnapOpenDur 91.136842ms 2015-08-03T20:58:45.521Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11930000 QueuedCount 4195 2015-08-03T20:58:46.808Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 5.782µs CommitTime 143.968265ms TotalFlushTime 14m47.60999242s TotalCommitTime 3m56.883857319s 2015-08-03T20:58:46.808Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22521018, 11760509, 366 committed:true) SnapCreateDur 159.956499ms SnapOpenDur 29.008µs 2015-08-03T20:58:46.941Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11940000 QueuedCount 3860 2015-08-03T20:58:47.074Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22543830, 11771915, 0 committed:false) SnapCreateDur 13.186µs SnapOpenDur 34.086815ms 2015-08-03T20:58:48.2Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11950000 QueuedCount 7366 2015-08-03T20:58:48.433Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22566664, 11783332, 0 committed:false) SnapCreateDur 16.174µs SnapOpenDur 89.113962ms 2015-08-03T20:58:48.511Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11960000 QueuedCount 1197 2015-08-03T20:58:49.159Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.658µs CommitTime 605.31249ms TotalFlushTime 14m49.115544008s TotalCommitTime 3m57.489169809s 2015-08-03T20:58:49.159Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22574198, 11787099, 367 committed:true) SnapCreateDur 618.602034ms SnapOpenDur 27.717µs 2015-08-03T20:58:49.323Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11970000 QueuedCount 2665 2015-08-03T20:58:49.404Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22596802, 11798401, 0 committed:false) SnapCreateDur 14.044µs SnapOpenDur 33.654374ms 2015-08-03T20:58:50.646Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11980000 QueuedCount 4203 2015-08-03T20:58:50.833Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22619482, 11809741, 0 committed:false) SnapCreateDur 13.865µs SnapOpenDur 89.135288ms 2015-08-03T20:58:51.074Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 11990000 QueuedCount 5791 2015-08-03T20:58:52.298Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22642306, 11821153, 0 committed:false) SnapCreateDur 34.483µs SnapOpenDur 84.399859ms 2015-08-03T20:58:52.496Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:459579012, Disk:2134937600, ExtraSnapshotData:0, Fragmentation:78.47342179930692% 2015-08-03T20:58:52.496Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:58:52.496Z-07:00 [Error] ForestDB Error (data.fdb.0/default) errcode: -15, msg: doc_length checksum mismatch error in a database file '/data/@2i/bucket-1_by_email_16558345882944865108_0.index/data.fdb.17' 2015-08-03T20:58:52.496Z-07:00 [Error] ForestDB Error (data.fdb.0/default) errcode: -15, msg: doc_length checksum mismatch error in a database file '/data/@2i/bucket-1_by_email_16558345882944865108_0.index/data.fdb.17' Assertion in (nil) != 0x7f4e01acf9a0 in /home/couchbase/jenkins/workspace/sherlock-unix/forestdb/src/hbtrie.cc:84 2015-08-03T20:58:52.497Z-07:00 [Error] ForestDB Error (data.fdb.0/default) errcode: -15, msg: doc_body checksum mismatch error in a database file '/data/@2i/bucket-1_by_email_16558345882944865108_0.index/data.fdb.17' 2015-08-03T20:58:52.677Z-07:00 [Info] ForestDBSlice::Compact No Snapshot Found. Skipped Compaction.Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T20:58:52.678Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 [New LWP 59925] [New LWP 56583] [New LWP 56582] [New LWP 56579] [New LWP 56578] [New LWP 56577] [New LWP 56576] [New LWP 56575] [New LWP 56574] [New LWP 56573] [New LWP 56572] [New LWP 56571] [New LWP 56570] [New LWP 56522] [New LWP 56521] [New LWP 56520] [New LWP 56515] [New LWP 56513] [New LWP 56512] [New LWP 56511] [New LWP 56510] [New LWP 56509] [New LWP 56508] [New LWP 56507] [New LWP 56506] [New LWP 56505] [New LWP 56504] [New LWP 56503] [New LWP 56502] [New LWP 56501] [New LWP 56500] [New LWP 56499] [New LWP 56498] [New LWP 56497] [New LWP 56496] [New LWP 56491] [New LWP 56482] [New LWP 56337] [New LWP 56336] [New LWP 56335] [New LWP 56334] [New LWP 56227] [New LWP 56226] [New LWP 56205] [New LWP 56204] [New LWP 56203] [New LWP 56202] [New LWP 56201] [New LWP 56200] [New LWP 56199] [Thread debugging using libthread_db enabled] 0x00000000005a1dd4 in ?? () Thread 51 (Thread 0x7f5d6b3ff700 (LWP 56199)): #0 0x00000000004431fd in ?? () #1 0x0000000000000000 in ?? () Thread 50 (Thread 0x7f5d6a9fe700 (LWP 56200)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x0000000001608bc0 in ?? () #3 0x0000000000000000 in ?? () Thread 49 (Thread 0x7f5d69ffd700 (LWP 56201)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c208020b58 in ?? () #3 0x0000000000000000 in ?? () Thread 48 (Thread 0x7f5d695fc700 (LWP 56202)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c2080215d8 in ?? () #3 0x0000000000000000 in ?? () Thread 47 (Thread 0x7f5d68bfb700 (LWP 56203)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c208021958 in ?? () #3 0x0000000000000000 in ?? () Thread 46 (Thread 0x7f5d5bfff700 (LWP 56204)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c208021cd8 in ?? () #3 0x0000000000000000 in ?? () Thread 45 (Thread 0x7f5d5b5fe700 (LWP 56205)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c209fa00d8 in ?? () #3 0x0000000000000000 in ?? () Thread 44 (Thread 0x7f5d5abfd700 (LWP 56226)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c209fa0458 in ?? () #3 0x0000000000000000 in ?? () Thread 43 (Thread 0x7f5d5a1fc700 (LWP 56227)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c2080540d8 in ?? () #3 0x0000000000000000 in ?? () Thread 42 (Thread 0x7f4e017ff700 (LWP 56334)): #0 0x00000033d720b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5d6bee66be in compactor_thread(void*) () from /opt/couchbase/bin/../lib/libforestdb.so #2 0x00000033d72079d1 in start_thread () from /lib64/libpthread.so.0 #3 0x00000033d6ee8b6d in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7f4e00dfe700 (LWP 56335)): #0 0x00000033d720b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5d6bee66be in compactor_thread(void*) () from /opt/couchbase/bin/../lib/libforestdb.so #2 0x00000033d72079d1 in start_thread () from /lib64/libpthread.so.0 #3 0x00000033d6ee8b6d in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7f4e003fd700 (LWP 56336)): #0 0x00000033d720b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5d6bee66be in compactor_thread(void*) () from /opt/couchbase/bin/../lib/libforestdb.so #2 0x00000033d72079d1 in start_thread () from /lib64/libpthread.so.0 #3 0x00000033d6ee8b6d in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7f4dff9fc700 (LWP 56337)): #0 0x00000033d720b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f5d6bee66be in compactor_thread(void*) () from /opt/couchbase/bin/../lib/libforestdb.so #2 0x00000033d72079d1 in start_thread () from /lib64/libpthread.so.0 #3 0x00000033d6ee8b6d in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7f4dfcfff700 (LWP 56482)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20b8cc0d8 in ?? () #3 0x0000000000000000 in ?? () Thread 37 (Thread 0x7f4df7bff700 (LWP 56491)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20b56e0d8 in ?? () #3 0x0000000000000000 in ?? () Thread 36 (Thread 0x7f4df6bff700 (LWP 56496)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c208055258 in ?? () #3 0x0000000000000000 in ?? () Thread 35 (Thread 0x7f4df5bff700 (LWP 56497)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20be1f258 in ?? () #3 0x0000000000000000 in ?? () Thread 34 (Thread 0x7f4df51fe700 (LWP 56498)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20ca0e0d8 in ?? () #3 0x0000000000000000 in ?? () Thread 33 (Thread 0x7f4de7fff700 (LWP 56499)): #0 0x00000033d720f2ad in waitpid () from /lib64/libpthread.so.0 #1 0x00007f5d6bf0bc6f in dump_backtraces() () from /opt/couchbase/bin/../lib/libforestdb.so #2 0x00007f5d6bf0bd44 in _dbg_assert(int, char const*, unsigned long, unsigned long) () from /opt/couchbase/bin/../lib/libforestdb.so #3 0x00007f5d6befc00e in _hbtrie_reform_key () from /opt/couchbase/bin/../lib/libforestdb.so #4 0x00007f5d6beff608 in _hbtrie_insert(hbtrie*, void*, int, void*, void*, unsigned char) () from /opt/couchbase/bin/../lib/libforestdb.so #5 0x00007f5d6befb71a in _fdb_wal_flush_func(void*, wal_item*) () from /opt/couchbase/bin/../lib/libforestdb.so #6 0x00007f5d6bf057d7 in wal_flush () from /opt/couchbase/bin/../lib/libforestdb.so #7 0x00007f5d6bef61c2 in _fdb_commit () from /opt/couchbase/bin/../lib/libforestdb.so Thread 32 (Thread 0x7f4de75fe700 (LWP 56500)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20ca840d8 in ?? () #3 0x0000000000000000 in ?? () Thread 31 (Thread 0x7f4de6bfd700 (LWP 56501)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20cabc0d8 in ?? () #3 0x0000000000000000 in ?? () Thread 30 (Thread 0x7f4de61fc700 (LWP 56502)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20cb080d8 in ?? () #3 0x0000000000000000 in ?? () Thread 29 (Thread 0x7f4de57fb700 (LWP 56503)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20cb4a0d8 in ?? () #3 0x0000000000000000 in ?? () Thread 28 (Thread 0x7f4de4dfa700 (LWP 56504)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20cb960d8 in ?? () #3 0x0000000000000000 in ?? () Thread 27 (Thread 0x7f4dcbfff700 (LWP 56505)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20ca84458 in ?? () #3 0x0000000000000000 in ?? () Thread 26 (Thread 0x7f4dcb5fe700 (LWP 56506)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20ca44458 in ?? () #3 0x0000000000000000 in ?? () Thread 25 (Thread 0x7f4dcabfd700 (LWP 56507)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20cb08458 in ?? () #3 0x0000000000000000 in ?? () Thread 24 (Thread 0x7f4dc9bff700 (LWP 56508)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20d1b20d8 in ?? () #3 0x0000000000000000 in ?? () Thread 23 (Thread 0x7f4dc91fe700 (LWP 56509)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20be1f5d8 in ?? () #3 0x0000000000000000 in ?? () Thread 22 (Thread 0x7f4dc3fff700 (LWP 56510)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20d2420d8 in ?? () #3 0x0000000000000000 in ?? () Thread 21 (Thread 0x7f4dc35fe700 (LWP 56511)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20d28e0d8 in ?? () #3 0x0000000000000000 in ?? () Thread 20 (Thread 0x7f4dc2bfd700 (LWP 56512)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20cabc458 in ?? () #3 0x0000000000000000 in ?? () Thread 19 (Thread 0x7f4dc21fc700 (LWP 56513)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20d1b2458 in ?? () #3 0x0000000000000000 in ?? () Thread 18 (Thread 0x7f4dc17fb700 (LWP 56515)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20c1d6ed8 in ?? () #3 0x0000000000000000 in ?? () Thread 17 (Thread 0x7f4d77fff700 (LWP 56520)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20c0bc0d8 in ?? () #3 0x0000000000000000 in ?? () Thread 16 (Thread 0x7f4d7ffff700 (LWP 56521)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20c1d60d8 in ?? () #3 0x0000000000000000 in ?? () Thread 15 (Thread 0x7f4d7f5fe700 (LWP 56522)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20b56e458 in ?? () #3 0x0000000000000000 in ?? () Thread 14 (Thread 0x7f4d76fff700 (LWP 56570)): #0 0x0000000000443639 in ?? () #1 0x0000000000417123 in crosscall_amd64 () #2 0x00007f4d00000005 in ?? () #3 0x00007f4d76ffe740 in ?? () #4 0xffffffff00000080 in ?? () #5 0x00007f4d00000000 in ?? () #6 0xffffffff00000080 in ?? () #7 0x0000000000000000 in ?? () Thread 13 (Thread 0x7f4d765fe700 (LWP 56571)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20b9580d8 in ?? () #3 0x0000000000000000 in ?? () Thread 12 (Thread 0x7f4d75bfd700 (LWP 56572)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20c1d6458 in ?? () #3 0x0000000000000000 in ?? () Thread 11 (Thread 0x7f4d751fc700 (LWP 56573)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20b24a0d8 in ?? () #3 0x0000000000000000 in ?? () Thread 10 (Thread 0x7f4d67fff700 (LWP 56574)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20b3c40d8 in ?? () #3 0x0000000000000000 in ?? () Thread 9 (Thread 0x7f4d675fe700 (LWP 56575)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c208055958 in ?? () #3 0x0000000000000000 in ?? () Thread 8 (Thread 0x7f4d66bfd700 (LWP 56576)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20b8cc7d8 in ?? () #3 0x0000000000000000 in ?? () Thread 7 (Thread 0x7f4d661fc700 (LWP 56577)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20d28e458 in ?? () #3 0x0000000000000000 in ?? () Thread 6 (Thread 0x7f4d657fb700 (LWP 56578)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20b56eb58 in ?? () #3 0x0000000000000000 in ?? () Thread 5 (Thread 0x7f4d64dfa700 (LWP 56579)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20d28e7d8 in ?? () #3 0x0000000000000000 in ?? () Thread 4 (Thread 0x7f4d37fff700 (LWP 56582)): #0 0x000000000040fde0 in crosscall_amd64 () #1 0x000000000058d7be in ?? () #2 0x0000000000ae21a0 in ?? () #3 0x0000000000ae21a0 in ?? () #4 0x000000c22a0ea360 in ?? () #5 0x000000c208001c20 in ?? () #6 0x000000c200000000 in ?? () #7 0x000000c200000001 in ?? () Thread 3 (Thread 0x7f4d375fe700 (LWP 56583)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20be1e0d8 in ?? () #3 0x0000000000000000 in ?? () Thread 2 (Thread 0x7f4d24bff700 (LWP 59925)): #0 0x00000000004434c3 in ?? () #1 0x0000000000436f97 in ?? () #2 0x000000c20d242b58 in ?? () #3 0x0000000000000000 in ?? () Thread 1 (Thread 0x7f5d6ba70760 (LWP 56194)): #0 0x00000000005a1dd4 in ?? () #1 0x000000000059f59e in ?? () #2 0x0000000000000000 in ?? () gdb dumping successful indexer: /home/couchbase/jenkins/workspace/sherlock-unix/forestdb/src/hbtrie.cc:84: int _hbtrie_reform_key(hbtrie*, void*, int, void*): Assertion `rsize && rsize <= trie->chunksize' failed. Dumping backtrace for /opt/couchbase/bin/indexer pid=--pid=56194.. SIGABRT: abort PC=0x33d6e32925 signal arrived during cgo execution goroutine 9859892 [syscall, locked to thread]: runtime.cgocall_errno(0x402e05, 0xc249a2b590, 0x0) /usr/local/go/src/runtime/cgocall.go:130 +0xf5 fp=0xc249a2b570 sp=0xc249a2b548 github.com/couchbase/indexing/secondary/fdb._Cfunc_fdb_commit(0x7f5d59023090, 0xef5d01, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/fdb/:52 +0x43 fp=0xc249a2b590 sp=0xc249a2b570 github.com/couchbase/indexing/secondary/fdb.(*File).Commit(0xc20b9cbec0, 0xc216aafc01, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/fdb/file.go:67 +0x1f5 fp=0xc249a2b610 sp=0xc249a2b590 github.com/couchbase/indexing/secondary/indexer.(*fdbSlice).NewSnapshot(0xc20b42f080, 0xc21de34c40, 0x15fce01, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:793 +0x617 fp=0xc249a2b858 sp=0xc249a2b610 github.com/couchbase/indexing/secondary/indexer.func·074(0xe5cb0fd9c4fb0354, 0xc20b9cbd40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:313 +0x1673 fp=0xc249a2bfd0 sp=0xc249a2b858 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc249a2bfd8 sp=0xc249a2bfd0 created by github.com/couchbase/indexing/secondary/indexer.(*storageMgr).createSnapshotWorker /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:383 +0x41d goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0xc20800d680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:396 +0x28c github.com/couchbase/indexing/secondary/indexer.NewIndexer(0xc208058630, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:282 +0x26d9 main.main() /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:80 +0xa65 goroutine 5 [syscall, 27 minutes]: os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:21 +0x1f created by os/signal.init·1 /usr/local/go/src/os/signal/signal_unix.go:27 +0x35 goroutine 9 [IO wait, 27 minutes]: net.(*pollDesc).Wait(0xc208010ed0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc208010ed0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc208010e70, 0xc2080e7000, 0x1000, 0x1000, 0x0, 0x7f5d6ba40e70, 0xc20808c2f0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20802e118, 0xc2080e7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc bufio.(*Reader).fill(0xc20805cba0) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(*Reader).Read(0xc20805cba0, 0xc20803e001, 0x5ff, 0x5ff, 0xc20802d440, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:174 +0x26c github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0xc20801f180, 0xc20803e001, 0x5ff, 0x5ff, 0x2, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:100 +0x5c encoding/json.(*Decoder).readValue(0xc2080ec000, 0x417af0, 0x0, 0x0) /usr/local/go/src/encoding/json/stream.go:124 +0x5e1 encoding/json.(*Decoder).Decode(0xc2080ec000, 0xc74bc0, 0xc20805cc20, 0x0, 0x0) /usr/local/go/src/encoding/json/stream.go:44 +0x7b net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0xc20805cc00, 0xc20801f480, 0x0, 0x0) /usr/local/go/src/net/rpc/jsonrpc/server.go:66 +0x9c net/rpc.(*Server).readRequestHeader(0xc20802c200, 0x7f5d6ba4cb40, 0xc20805cc00, 0x0, 0x0, 0xc20801f480, 0x410c00, 0x0, 0x0) /usr/local/go/src/net/rpc/server.go:575 +0x97 net/rpc.(*Server).readRequest(0xc20802c200, 0x7f5d6ba4cb40, 0xc20805cc00, 0xc2080a4000, 0xc20801ea20, 0xabb2c0, 0x0, 0x0, 0x0, 0x0, ...) /usr/local/go/src/net/rpc/server.go:542 +0xa2 net/rpc.(*Server).ServeCodec(0xc20802c200, 0x7f5d6ba4cb40, 0xc20805cc00) /usr/local/go/src/net/rpc/server.go:461 +0x6d github.com/couchbase/cbauth/revrpc.(*Service).Run(0xc208058420, 0xc2080b7f90, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:146 +0x6c8 github.com/couchbase/cbauth/revrpc.BabysitService(0xc2080b7f90, 0xc208058420, 0x7f5d6ba41498, 0xc20801f0e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:242 +0x80 github.com/couchbase/cbauth.runRPCForSvc(0xc208058420, 0xc208058450, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/godeps/src/github.com/couchbase/cbauth/default.go:56 +0x193 github.com/couchbase/cbauth.func·004() /home/couchbase/jenkins/workspace/sherlock-unix/godeps/src/github.com/couchbase/cbauth/default.go:63 +0x36 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/sherlock-unix/godeps/src/github.com/couchbase/cbauth/default.go:64 +0x1c9 goroutine 10 [sleep]: github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:281 +0x41 created by github.com/couchbase/indexing/secondary/common.init·1 /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:23 +0x99 goroutine 11 [chan receive, 27 minutes]: github.com/couchbase/indexing/secondary/platform.DumpOnSignal() /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/platform/platform.go:27 +0x173 created by main.main /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:60 +0x4d2 goroutine 12 [syscall, 27 minutes]: syscall.Syscall(0x0, 0x0, 0xc208019f64, 0x4, 0x0, 0x0, 0x0) /usr/local/go/src/syscall/asm_linux_amd64.s:21 +0x5 syscall.read(0x0, 0xc208019f64, 0x4, 0x4, 0x0, 0x0, 0x0) /usr/local/go/src/syscall/zsyscall_linux_amd64.go:867 +0x6e syscall.Read(0x0, 0xc208019f64, 0x4, 0x4, 0x0, 0x0, 0x0) /usr/local/go/src/syscall/syscall_unix.go:136 +0x58 os.(*File).read(0xc20802e000, 0xc208019f64, 0x4, 0x4, 0x0, 0x0, 0x0) /usr/local/go/src/os/file_unix.go:191 +0x5e os.(*File).Read(0xc20802e000, 0xc208019f64, 0x4, 0x4, 0x0, 0x0, 0x0) /usr/local/go/src/os/file.go:95 +0x91 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:316 +0x99 created by main.main /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:61 +0x4e3 goroutine 17 [select, 25 minutes]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0xc20802af80, 0xe080f0, 0x1, 0xc20b193fb0, 0xc20805d3e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x8bb github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(0xc20802af80, 0xe080f0, 0x1, 0xc20b193fb0, 0xc20805d3e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:244 +0x80 github.com/couchbase/cbauth/metakv.RunObserveChildren(0xe080f0, 0x1, 0xc20b193fb0, 0xc20805d3e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:367 +0x64 github.com/couchbase/indexing/secondary/indexer.func·069() /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:66 +0x94 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:73 +0x4de goroutine 15 [select, 25 minutes]: net/http.(*persistConn).readLoop(0xc208066420) /usr/local/go/src/net/http/transport.go:928 +0x9ce created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:660 +0xc9f goroutine 16 [select, 25 minutes]: net/http.(*persistConn).writeLoop(0xc208066420) /usr/local/go/src/net/http/transport.go:945 +0x41d created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:661 +0xcbc goroutine 33 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0xc208066160) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:113 +0xb9 created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:93 +0x46b goroutine 34 [chan receive, 22 minutes]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0xc209f7a000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:83 +0x61 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:68 +0x45e goroutine 35 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0xc20805cd20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:109 +0x61 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:94 +0x26f goroutine 49 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0xc208066160) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:137 +0x152 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:106 +0x55 goroutine 50 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0xc208066160) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:160 +0x468 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:107 +0x6f goroutine 65 [IO wait, 21 minutes]: net.(*pollDesc).Wait(0xc20af1a0d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af1a0d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).accept(0xc20af1a070, 0x0, 0x7f5d6ba40e70, 0xc218455f70) /usr/local/go/src/net/fd_unix.go:419 +0x40b net.(*TCPListener).AcceptTCP(0xc20af1c000, 0xc20af27f30, 0x0, 0x0) /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e net.(*TCPListener).Accept(0xc20af1c000, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/tcpsock_posix.go:244 +0x4c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0xc20af1a000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:107 +0x84 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:63 +0x6e5 goroutine 66 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0xc2080a4100) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:276 +0x61 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:217 +0x4d2 goroutine 67 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0xc2080a4100) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:225 +0x6a created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:218 +0x4ec goroutine 20 [IO wait, 25 minutes]: net.(*pollDesc).Wait(0xc2080106f0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2080106f0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc208010690, 0xc2080e6000, 0x1000, 0x1000, 0x0, 0x7f5d6ba40e70, 0xc20af340b8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20802e120, 0xc2080e6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc net/http.noteEOFReader.Read(0x7f5d6ba4c740, 0xc20802e120, 0xc208066478, 0xc2080e6000, 0x1000, 0x1000, 0x64fe6b, 0x0, 0x0) /usr/local/go/src/net/http/transport.go:1270 +0x6e net/http.(*noteEOFReader).Read(0xc20801f760, 0xc2080e6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) :125 +0xd4 bufio.(*Reader).fill(0xc20805d740) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(*Reader).ReadSlice(0xc20805d740, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:295 +0x257 net/http/internal.readLine(0xc20805d740, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/internal/chunked.go:110 +0x5a net/http/internal.(*chunkedReader).beginChunk(0xc2080591a0) /usr/local/go/src/net/http/internal/chunked.go:47 +0x46 net/http/internal.(*chunkedReader).Read(0xc2080591a0, 0xc20af33000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/internal/chunked.go:77 +0xbb net/http.(*body).readLocked(0xc20802ddc0, 0xc20af33000, 0xe00, 0xe00, 0xffffffff, 0x0, 0x0) /usr/local/go/src/net/http/transfer.go:584 +0x7a net/http.(*body).Read(0xc20802ddc0, 0xc20af33000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/transfer.go:579 +0x115 net/http.(*bodyEOFSignal).Read(0xc20802de00, 0xc20af33000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/transport.go:1193 +0x285 encoding/json.(*Decoder).readValue(0xc2080ec180, 0xc20802dac0, 0x0, 0x0) /usr/local/go/src/encoding/json/stream.go:124 +0x5e1 encoding/json.(*Decoder).Decode(0xc2080ec180, 0xabb500, 0xc20802df00, 0x0, 0x0) /usr/local/go/src/encoding/json/stream.go:44 +0x7b github.com/couchbase/cbauth/metakv.func·003() /home/couchbase/jenkins/workspace/sherlock-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:267 +0x134 created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/sherlock-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:275 +0x46a goroutine 9131071 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b77058, 0xc214556c00, 0xc2150a7260, 0xc2150a72c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130539 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21332d190, 0xc213292060, 0xc262295c80, 0xc262295ce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 81 [syscall, 25 minutes, locked to thread]: runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 goroutine 69 [IO wait, 22 minutes]: net.(*pollDesc).Wait(0xc20af1b100, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af1b100, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).accept(0xc20af1b0a0, 0x0, 0x7f5d6ba40e70, 0xc20bcf0bb0) /usr/local/go/src/net/fd_unix.go:419 +0x40b net.(*TCPListener).AcceptTCP(0xc20af1c0d8, 0x82626e, 0x0, 0x0) /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e net.(*TCPListener).Accept(0xc20af1c0d8, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/tcpsock_posix.go:244 +0x4c github.com/couchbase/gometa/common.(*PeerListener).listen(0xc20af14c40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:135 +0x90 created by github.com/couchbase/gometa/common.StartPeerListener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:63 +0x1ad goroutine 70 [select, 21 minutes]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0xc20b02e140, 0xc20af1b030, 0xc20af34120, 0x7f5d6ba57420, 0xc20af30c00, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:384 +0x8fe github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0xc20affacd4, 0x5, 0xc20af14c40, 0x7f5d6ba57608, 0xc20afe28a0, 0x7f5d6ba57668, 0xc20af1afc0, 0x7f5d6ba57498, 0x1607428, 0x7f5d6ba57420, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:130 +0x6dd github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0xc20af86460) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:452 +0x1f5 github.com/couchbase/gometa/server.(*EmbeddedServer).run(0xc20af86460) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:354 +0x28 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:82 +0x270 goroutine 71 [select, 21 minutes]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0xc20af30c00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:160 +0x30c created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:71 +0x51 goroutine 72 [chan receive, 21 minutes]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0xc209fae330) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:120 +0xab created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:98 +0xc41 goroutine 73 [chan receive, 21 minutes]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0xc20af2a690) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:449 +0x62 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:337 +0x330 goroutine 74 [sleep]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0xc20af2a690) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:489 +0x1d6 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:338 +0x34a goroutine 21 [chan receive, 21 minutes]: github.com/couchbase/gometa/protocol.(*Leader).listen(0xc20af48090) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:472 +0xee created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:160 +0x3c5 goroutine 22 [select, 22 minutes]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0xc20b02e140, 0xc20af34120) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:186 +0x691 created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:125 +0x69c goroutine 57 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0xc20b178360) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:98 +0x954 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:43 +0x123 goroutine 52 [IO wait, 22 minutes]: net.(*pollDesc).Wait(0xc20af44d10, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af44d10, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af44cb0, 0xc20b0e2000, 0x1000, 0x1000, 0x0, 0x7f5d6ba40e70, 0xc20dd090e8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af4e0a8, 0xc20b0e2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc net/http.noteEOFReader.Read(0x7f5d6ba4c740, 0xc20af4e0a8, 0xc20af74268, 0xc20b0e2000, 0x1000, 0x1000, 0xc13380, 0x0, 0x0) /usr/local/go/src/net/http/transport.go:1270 +0x6e net/http.(*noteEOFReader).Read(0xc20b0e00c0, 0xc20b0e2000, 0x1000, 0x1000, 0xc209f84000, 0x0, 0x0) :125 +0xd4 bufio.(*Reader).fill(0xc20af5aae0) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(*Reader).Peek(0xc20af5aae0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:132 +0xf0 net/http.(*persistConn).readLoop(0xc20af74210) /usr/local/go/src/net/http/transport.go:842 +0xa4 created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:660 +0xc9f goroutine 53 [select, 22 minutes]: net/http.(*persistConn).writeLoop(0xc20af74210) /usr/local/go/src/net/http/transport.go:945 +0x41d created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:661 +0xcbc goroutine 23 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0xc209f78900) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:144 +0x61 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:129 +0x57e goroutine 76 [IO wait, 22 minutes]: net.(*pollDesc).Wait(0xc20af1b870, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af1b870, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af1b810, 0xc20b0e8000, 0x1000, 0x1000, 0x0, 0x7f5d6ba40e70, 0xc20ede2120) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af1c140, 0xc20b0e8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc net/http.noteEOFReader.Read(0x7f5d6ba4c740, 0xc20af1c140, 0xc20af963c8, 0xc20b0e8000, 0x1000, 0x1000, 0xc13380, 0x0, 0x0) /usr/local/go/src/net/http/transport.go:1270 +0x6e net/http.(*noteEOFReader).Read(0xc20b0706a0, 0xc20b0e8000, 0x1000, 0x1000, 0xc209f7c000, 0x0, 0x0) :125 +0xd4 bufio.(*Reader).fill(0xc20af312c0) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(*Reader).Peek(0xc20af312c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:132 +0xf0 net/http.(*persistConn).readLoop(0xc20af96370) /usr/local/go/src/net/http/transport.go:842 +0xa4 created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:660 +0xc9f goroutine 77 [select, 22 minutes]: net/http.(*persistConn).writeLoop(0xc20af96370) /usr/local/go/src/net/http/transport.go:945 +0x41d created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:661 +0xcbc goroutine 55 [IO wait, 22 minutes]: net.(*pollDesc).Wait(0xc20af45100, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af45100, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af450a0, 0xc20b0ec000, 0x1000, 0x1000, 0x0, 0x7f5d6ba40e70, 0xc20eaeea78) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af4e0e8, 0xc20b0ec000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc net/http.noteEOFReader.Read(0x7f5d6ba4c740, 0xc20af4e0e8, 0xc20af74318, 0xc20b0ec000, 0x1000, 0x1000, 0xc13380, 0x0, 0x0) /usr/local/go/src/net/http/transport.go:1270 +0x6e net/http.(*noteEOFReader).Read(0xc20b0e03a0, 0xc20b0ec000, 0x1000, 0x1000, 0xc209f7c000, 0x0, 0x0) :125 +0xd4 bufio.(*Reader).fill(0xc20af5af60) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(*Reader).Peek(0xc20af5af60, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:132 +0xf0 net/http.(*persistConn).readLoop(0xc20af742c0) /usr/local/go/src/net/http/transport.go:842 +0xa4 created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:660 +0xc9f goroutine 56 [select, 22 minutes]: net/http.(*persistConn).writeLoop(0xc20af742c0) /usr/local/go/src/net/http/transport.go:945 +0x41d created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:661 +0xcbc goroutine 79 [IO wait, 22 minutes]: net.(*pollDesc).Wait(0xc20af1bbf0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af1bbf0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af1bb90, 0xc20b0f8000, 0x1000, 0x1000, 0x0, 0x7f5d6ba40e70, 0xc20b172750) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af1c180, 0xc20b0f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc net/http.noteEOFReader.Read(0x7f5d6ba4c740, 0xc20af1c180, 0xc20af96478, 0xc20b0f8000, 0x1000, 0x1000, 0xc13380, 0x0, 0x0) /usr/local/go/src/net/http/transport.go:1270 +0x6e net/http.(*noteEOFReader).Read(0xc20b070960, 0xc20b0f8000, 0x1000, 0x1000, 0xc209f7d200, 0x0, 0x0) :125 +0xd4 bufio.(*Reader).fill(0xc20af31740) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(*Reader).Peek(0xc20af31740, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:132 +0xf0 net/http.(*persistConn).readLoop(0xc20af96420) /usr/local/go/src/net/http/transport.go:842 +0xa4 created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:660 +0xc9f goroutine 80 [select, 22 minutes]: net/http.(*persistConn).writeLoop(0xc20af96420) /usr/local/go/src/net/http/transport.go:945 +0x41d created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:661 +0xcbc goroutine 99 [chan receive, 25 minutes]: github.com/couchbase/indexing/secondary/indexer.(*adminMgr).run(0xc20b17a920) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/admin_manager.go:51 +0x61 created by github.com/couchbase/indexing/secondary/indexer.NewAdminManager /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/admin_manager.go:36 +0xbb goroutine 100 [chan receive, 25 minutes]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0xc20b17e0f0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:154 +0x91 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:144 +0x106 goroutine 101 [chan receive, 2 minutes]: github.com/couchbase/indexing/secondary/logging.func·001() /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:196 +0x7d created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:200 +0x194 goroutine 102 [IO wait, 22 minutes]: net.(*pollDesc).Wait(0xc208056e60, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc208056e60, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).accept(0xc208056e00, 0x0, 0x7f5d6ba40e70, 0xc20bb21900) /usr/local/go/src/net/fd_unix.go:419 +0x40b net.(*TCPListener).AcceptTCP(0xc20802e1a8, 0x5c13fe, 0x0, 0x0) /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e net/http.tcpKeepAliveListener.Accept(0xc20802e1a8, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:1976 +0x4c net/http.(*Server).Serve(0xc209f78960, 0x7f5d6ba5c078, 0xc20802e1a8, 0x0, 0x0) /usr/local/go/src/net/http/server.go:1728 +0x92 net/http.(*Server).ListenAndServe(0xc209f78960, 0x0, 0x0) /usr/local/go/src/net/http/server.go:1718 +0x154 net/http.ListenAndServe(0xc20b17a950, 0x5, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:1808 +0xba github.com/couchbase/indexing/secondary/indexer.func·010() /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:275 +0x42 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:279 +0x26c6 goroutine 103 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0xc20800d680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:489 +0x229 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:391 +0x40 goroutine 104 [select, 22 minutes]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0xc20800d680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:430 +0x320 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:392 +0x5d goroutine 36 [IO wait]: net.(*pollDesc).Wait(0xc208010370, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc208010370, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc208010310, 0xc20b19c000, 0x1000, 0x1000, 0x0, 0x7f5d6ba40e70, 0xc255a9db10) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2080c6008, 0xc20b19c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc net/http.(*liveSwitchReader).Read(0xc20b0fe048, 0xc20b19c000, 0x1000, 0x1000, 0xc26f323898, 0x0, 0x0) /usr/local/go/src/net/http/server.go:214 +0xab io.(*LimitedReader).Read(0xc209f7a020, 0xc20b19c000, 0x1000, 0x1000, 0xc23d5fc400, 0x0, 0x0) /usr/local/go/src/io/io.go:408 +0xce bufio.(*Reader).fill(0xc20805d4a0) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(*Reader).ReadSlice(0xc20805d4a0, 0x7f5d6ba5c30a, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:295 +0x257 bufio.(*Reader).ReadLine(0xc20805d4a0, 0x0, 0x0, 0x0, 0xc269b47d00, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:324 +0x62 net/textproto.(*Reader).readLineSlice(0xc23df96990, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/textproto/reader.go:55 +0x9e net/textproto.(*Reader).ReadLine(0xc23df96990, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/textproto/reader.go:36 +0x4f net/http.ReadRequest(0xc20805d4a0, 0xc26f367930, 0x0, 0x0) /usr/local/go/src/net/http/request.go:598 +0xcb net/http.(*conn).readRequest(0xc20b0fe000, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:586 +0x26f net/http.(*conn).serve(0xc20b0fe000) /usr/local/go/src/net/http/server.go:1162 +0x69e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:1751 +0x35e goroutine 26 [IO wait]: net.(*pollDesc).Wait(0xc208057870, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc208057870, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc208057810, 0xc20b225000, 0x1000, 0x1000, 0x0, 0x7f5d6ba40e70, 0xc2263d8058) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20802e240, 0xc20b225000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc net/http.noteEOFReader.Read(0x7f5d6ba4c740, 0xc20802e240, 0xc20806e898, 0xc20b225000, 0x1000, 0x1000, 0xc13380, 0x0, 0x0) /usr/local/go/src/net/http/transport.go:1270 +0x6e net/http.(*noteEOFReader).Read(0xc20b2352e0, 0xc20b225000, 0x1000, 0x1000, 0xc209f87600, 0x0, 0x0) :125 +0xd4 bufio.(*Reader).fill(0xc209f79740) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(*Reader).Peek(0xc209f79740, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:132 +0xf0 net/http.(*persistConn).readLoop(0xc20806e840) /usr/local/go/src/net/http/transport.go:842 +0xa4 created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:660 +0xc9f goroutine 37 [IO wait]: net.(*pollDesc).Wait(0xc2080103e0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2080103e0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc208010380, 0xc20b23e000, 0x1000, 0x1000, 0x0, 0x7f5d6ba40e70, 0xc25793a170) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2080c6010, 0xc20b23e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc net/http.(*liveSwitchReader).Read(0xc20b0fe0e8, 0xc20b23e000, 0x1000, 0x1000, 0xc2080bd898, 0x0, 0x0) /usr/local/go/src/net/http/server.go:214 +0xab io.(*LimitedReader).Read(0xc209f7a080, 0xc20b23e000, 0x1000, 0x1000, 0xc24f3db400, 0x0, 0x0) /usr/local/go/src/io/io.go:408 +0xce bufio.(*Reader).fill(0xc20805d500) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(*Reader).ReadSlice(0xc20805d500, 0x7f5d6ba5c30a, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:295 +0x257 bufio.(*Reader).ReadLine(0xc20805d500, 0x0, 0x0, 0x0, 0xc26b5e8900, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:324 +0x62 net/textproto.(*Reader).readLineSlice(0xc23f69c960, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/textproto/reader.go:55 +0x9e net/textproto.(*Reader).ReadLine(0xc23f69c960, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/textproto/reader.go:36 +0x4f net/http.ReadRequest(0xc20805d500, 0xc211ea51e0, 0x0, 0x0) /usr/local/go/src/net/http/request.go:598 +0xcb net/http.(*conn).readRequest(0xc20b0fe0a0, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:586 +0x26f net/http.(*conn).serve(0xc20b0fe0a0) /usr/local/go/src/net/http/server.go:1162 +0x69e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:1751 +0x35e goroutine 27 [select]: net/http.(*persistConn).writeLoop(0xc20806e840) /usr/local/go/src/net/http/transport.go:945 +0x41d created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:661 +0xcbc goroutine 667 [chan receive]: github.com/couchbase/indexing/secondary/dcp.(*DcpFeed).genServer(0xc20bd5c840, 0xc20bd5c780, 0xabba) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:232 +0xea created by github.com/couchbase/indexing/secondary/dcp.(*Bucket).StartDcpFeedOver /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:158 +0x558 goroutine 655 [IO wait]: net.(*pollDesc).Wait(0xc20bd07e90, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bd07e90, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bd07e30, 0xc20bd543e0, 0x18, 0x18, 0x0, 0x7f5d6ba40e70, 0xc25793a1b8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20b045250, 0xc20bd543e0, 0x18, 0x18, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc io.ReadAtLeast(0x7f5d6ba4c740, 0xc20b045250, 0xc20bd543e0, 0x18, 0x18, 0x18, 0x0, 0x0, 0x0) /usr/local/go/src/io/io.go:298 +0xf1 io.ReadFull(0x7f5d6ba4c740, 0xc20b045250, 0xc20bd543e0, 0x18, 0x18, 0x1400, 0x0, 0x0) /usr/local/go/src/io/io.go:316 +0x6d github.com/couchbase/indexing/secondary/dcp/transport.(*MCRequest).Receive(0xc255188f60, 0x7f5d6ba4c740, 0xc20b045250, 0xc20bd543e0, 0x18, 0x18, 0x1418, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/mc_req.go:130 +0xee github.com/couchbase/indexing/secondary/dcp/transport/client.(*DcpFeed).doReceive(0xc20bd097a0, 0xc20bd09800, 0xc20bdac930) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/client/dcp_feed.go:882 +0x1b0 created by github.com/couchbase/indexing/secondary/dcp/transport/client.NewDcpFeed /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/client/dcp_feed.go:68 +0x4bf goroutine 690 [select]: github.com/couchbase/indexing/secondary/dcp/transport/client.(*DcpFeed).genServer(0xc20b33d2c0, 0xabba, 0xc20b33d200, 0xc20b33d260, 0xc20b33d320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/client/dcp_feed.go:165 +0x18cf created by github.com/couchbase/indexing/secondary/dcp/transport/client.NewDcpFeed /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/client/dcp_feed.go:67 +0x48e goroutine 618 [select, 2 minutes]: net/http.(*persistConn).writeLoop(0xc20806fa20) /usr/local/go/src/net/http/transport.go:945 +0x41d created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:661 +0xcbc goroutine 617 [IO wait, 2 minutes]: net.(*pollDesc).Wait(0xc20b4f7fe0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b4f7fe0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b4f7f80, 0xc20b2fe000, 0x1000, 0x1000, 0x0, 0x7f5d6ba40e70, 0xc25590ec90) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af4e850, 0xc20b2fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc net/http.noteEOFReader.Read(0x7f5d6ba4c740, 0xc20af4e850, 0xc20806fa78, 0xc20b2fe000, 0x1000, 0x1000, 0xc13380, 0x0, 0x0) /usr/local/go/src/net/http/transport.go:1270 +0x6e net/http.(*noteEOFReader).Read(0xc20b2f4300, 0xc20b2fe000, 0x1000, 0x1000, 0xc208012000, 0x0, 0x0) :125 +0xd4 bufio.(*Reader).fill(0xc20b5b3b60) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(*Reader).Peek(0xc20b5b3b60, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:132 +0xf0 net/http.(*persistConn).readLoop(0xc20806fa20) /usr/local/go/src/net/http/transport.go:842 +0xa4 created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:660 +0xc9f goroutine 9131251 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21632a5f0, 0xc2162bdda0, 0xc238d1d2c0, 0xc238d1d320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 12640 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2630e6500, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc256572000, 0x7f5d6ba4b5a8, 0xc21621d680, 0xc238f47b00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc256572000, 0x7f5d6ba4b5a8, 0xc21621d680, 0xc238f47b00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21621d680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 649 [IO wait, 22 minutes]: net.(*pollDesc).Wait(0xc20ba21f00, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20ba21f00, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20ba21ea0, 0xc20bc9bfd8, 0x8, 0x8, 0x0, 0x7f5d6ba40e70, 0xc20bd16010) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20b045110, 0xc20bc9bfd8, 0x8, 0x8, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/gometa/common.(*PeerPipe).readBytes(0xc20bc7b9e0, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:256 +0x102 github.com/couchbase/gometa/common.(*PeerPipe).doReceive(0xc20bc7b9e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:201 +0x7c created by github.com/couchbase/gometa/common.NewPeerPipe /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:60 +0x14b goroutine 691 [IO wait]: net.(*pollDesc).Wait(0xc20b337870, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b337870, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b337810, 0xc20bd8ef80, 0x18, 0x18, 0x0, 0x7f5d6ba40e70, 0xc257d1f4a8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2080c6ac0, 0xc20bd8ef80, 0x18, 0x18, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc io.ReadAtLeast(0x7f5d6ba4c740, 0xc2080c6ac0, 0xc20bd8ef80, 0x18, 0x18, 0x18, 0x0, 0x0, 0x0) /usr/local/go/src/io/io.go:298 +0xf1 io.ReadFull(0x7f5d6ba4c740, 0xc2080c6ac0, 0xc20bd8ef80, 0x18, 0x18, 0x1400, 0x0, 0x0) /usr/local/go/src/io/io.go:316 +0x6d github.com/couchbase/indexing/secondary/dcp/transport.(*MCRequest).Receive(0xc21a1bde00, 0x7f5d6ba4c740, 0xc2080c6ac0, 0xc20bd8ef80, 0x18, 0x18, 0x1418, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/mc_req.go:130 +0xee github.com/couchbase/indexing/secondary/dcp/transport/client.(*DcpFeed).doReceive(0xc20b33d2c0, 0xc20b33d320, 0xc20b339c50) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/client/dcp_feed.go:882 +0x1b0 created by github.com/couchbase/indexing/secondary/dcp/transport/client.NewDcpFeed /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/client/dcp_feed.go:68 +0x4bf goroutine 9130824 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b42d58, 0xc21613dec0, 0xc20b27f4a0, 0xc20b27f500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 676 [IO wait]: net.(*pollDesc).Wait(0xc20b9c35d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b9c35d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b9c3570, 0xc20bd96000, 0x1000, 0x1000, 0x0, 0x7f5d6ba40e70, 0xc257f818b8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20802ecf8, 0xc20bd96000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc net/http.noteEOFReader.Read(0x7f5d6ba4c740, 0xc20802ecf8, 0xc20bb24f78, 0xc20bd96000, 0x1000, 0x1000, 0xc13380, 0x0, 0x0) /usr/local/go/src/net/http/transport.go:1270 +0x6e net/http.(*noteEOFReader).Read(0xc20bd8eec0, 0xc20bd96000, 0x1000, 0x1000, 0xc209f7db00, 0x0, 0x0) :125 +0xd4 bufio.(*Reader).fill(0xc20b874900) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(*Reader).Peek(0xc20b874900, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:132 +0xf0 net/http.(*persistConn).readLoop(0xc20bb24f20) /usr/local/go/src/net/http/transport.go:842 +0xa4 created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:660 +0xc9f goroutine 233 [IO wait]: net.(*pollDesc).Wait(0xc20b93db10, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b93db10, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b93dab0, 0xc20bb74000, 0x1000, 0x1000, 0x0, 0x7f5d6ba40e70, 0xc26375d910) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20802ef28, 0xc20bb74000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc net/http.(*liveSwitchReader).Read(0xc20bb72048, 0xc20bb74000, 0x1000, 0x1000, 0xc215ec9898, 0x0, 0x0) /usr/local/go/src/net/http/server.go:214 +0xab io.(*LimitedReader).Read(0xc20bb70380, 0xc20bb74000, 0x1000, 0x1000, 0xc211782100, 0x0, 0x0) /usr/local/go/src/io/io.go:408 +0xce bufio.(*Reader).fill(0xc20baeea20) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(*Reader).ReadSlice(0xc20baeea20, 0x7f5d6ba5c30a, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:295 +0x257 bufio.(*Reader).ReadLine(0xc20baeea20, 0x0, 0x0, 0x0, 0xc26b942900, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:324 +0x62 net/textproto.(*Reader).readLineSlice(0xc20d1cb650, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/textproto/reader.go:55 +0x9e net/textproto.(*Reader).ReadLine(0xc20d1cb650, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/textproto/reader.go:36 +0x4f net/http.ReadRequest(0xc20baeea20, 0xc21884ad00, 0x0, 0x0) /usr/local/go/src/net/http/request.go:598 +0xcb net/http.(*conn).readRequest(0xc20bb72000, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:586 +0x26f net/http.(*conn).serve(0xc20bb72000) /usr/local/go/src/net/http/server.go:1162 +0x69e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:1751 +0x35e goroutine 9130150 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b77558, 0xc2161595c0, 0xc238f47320, 0xc238f47380) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 92981 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc245d8e000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d40d6d0, 0x7f5d6ba4b5a8, 0xc21361c738, 0xc26945e540) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d40d6d0, 0x7f5d6ba4b5a8, 0xc21361c738, 0xc26945e540) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21361c738) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 661 [select]: github.com/couchbase/indexing/secondary/indexer.(*fdbSlice).handleCommandsWorker(0xc20b42f080, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:274 +0x7d0 created by github.com/couchbase/indexing/secondary/indexer.NewForestDBSlice /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:137 +0x13a1 goroutine 656 [select]: github.com/couchbase/indexing/secondary/dcp/transport/client.(*DcpFeed).genServer(0xc20bd09c20, 0xabba, 0xc20bd09b60, 0xc20bd09bc0, 0xc20bd09c80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/client/dcp_feed.go:165 +0x18cf created by github.com/couchbase/indexing/secondary/dcp/transport/client.NewDcpFeed /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/client/dcp_feed.go:67 +0x48e goroutine 652 [select, 22 minutes]: github.com/couchbase/gometa/protocol.(*messageListener).start(0xc20bc7bc50) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:403 +0x5bf created by github.com/couchbase/gometa/protocol.(*Leader).AddWatcher /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:256 +0x5b7 goroutine 657 [IO wait]: net.(*pollDesc).Wait(0xc20bd07f70, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bd07f70, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bd07f10, 0xc20bdc5100, 0x18, 0x18, 0x0, 0x7f5d6ba40e70, 0xc257a21930) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20b045258, 0xc20bdc5100, 0x18, 0x18, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc io.ReadAtLeast(0x7f5d6ba4c740, 0xc20b045258, 0xc20bdc5100, 0x18, 0x18, 0x18, 0x0, 0x0, 0x0) /usr/local/go/src/io/io.go:298 +0xf1 io.ReadFull(0x7f5d6ba4c740, 0xc20b045258, 0xc20bdc5100, 0x18, 0x18, 0x1400, 0x0, 0x0) /usr/local/go/src/io/io.go:316 +0x6d github.com/couchbase/indexing/secondary/dcp/transport.(*MCRequest).Receive(0xc247ef80c0, 0x7f5d6ba4c740, 0xc20b045258, 0xc20bdc5100, 0x18, 0x18, 0x1418, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/mc_req.go:130 +0xee github.com/couchbase/indexing/secondary/dcp/transport/client.(*DcpFeed).doReceive(0xc20bd09c20, 0xc20bd09c80, 0xc20bdacab0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/client/dcp_feed.go:882 +0x1b0 created by github.com/couchbase/indexing/secondary/dcp/transport/client.NewDcpFeed /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/client/dcp_feed.go:68 +0x4bf goroutine 648 [chan receive, 21 minutes]: github.com/couchbase/gometa/common.(*PeerPipe).doSend(0xc20bc7b9e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:153 +0xb1 created by github.com/couchbase/gometa/common.NewPeerPipe /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:59 +0x131 goroutine 678 [IO wait, 22 minutes]: net.(*pollDesc).Wait(0xc20b9c3950, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b9c3950, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).accept(0xc20b9c38f0, 0x0, 0x7f5d6ba40e70, 0xc20d890398) /usr/local/go/src/net/fd_unix.go:419 +0x40b net.(*TCPListener).AcceptTCP(0xc20802ed28, 0xae16c0, 0x0, 0x0) /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e net.(*TCPListener).Accept(0xc20802ed28, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/tcpsock_posix.go:244 +0x4c github.com/couchbase/indexing/secondary/dataport.listener(0xc20be97e40, 0x18, 0x7f5d6ba531c0, 0xc20802ed28, 0xc20bee7200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:493 +0x3f created by github.com/couchbase/indexing/secondary/dataport.NewServer /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:171 +0x70a goroutine 9131015 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2131d1438, 0xc219a58de0, 0xc2612523c0, 0xc261252420) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 654 [select]: github.com/couchbase/indexing/secondary/dcp/transport/client.(*DcpFeed).genServer(0xc20bd097a0, 0xabba, 0xc20bd096e0, 0xc20bd09740, 0xc20bd09800) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/client/dcp_feed.go:165 +0x18cf created by github.com/couchbase/indexing/secondary/dcp/transport/client.NewDcpFeed /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/client/dcp_feed.go:67 +0x48e goroutine 9086277 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2bb80, 0xc2159a6300, 0xc2458d35c0, 0xc2458d3620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9084556 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bb598a8, 0xc20b906840, 0xc216b25620, 0xc216b25680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 665 [chan receive]: github.com/couchbase/indexing/secondary/dcp.(*DcpFeed).genServer(0xc20b33c9c0, 0xc20b33c900, 0xabba) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:232 +0xea created by github.com/couchbase/indexing/secondary/dcp.(*Bucket).StartDcpFeedOver /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:158 +0x558 goroutine 692 [select]: github.com/couchbase/indexing/secondary/dcp/transport/client.(*DcpFeed).genServer(0xc20b33d680, 0xabba, 0xc20b33d5c0, 0xc20b33d620, 0xc20b33d6e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/client/dcp_feed.go:165 +0x18cf created by github.com/couchbase/indexing/secondary/dcp/transport/client.NewDcpFeed /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/client/dcp_feed.go:67 +0x48e goroutine 591 [IO wait, 2 minutes]: net.(*pollDesc).Wait(0xc20bb1edf0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bb1edf0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bb1ed90, 0xc20b300000, 0x1000, 0x1000, 0x0, 0x7f5d6ba40e70, 0xc25590ec80) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20802eb80, 0xc20b300000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc net/http.(*liveSwitchReader).Read(0xc20b03f1c8, 0xc20b300000, 0x1000, 0x1000, 0xafbaa0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:214 +0xab io.(*LimitedReader).Read(0xc20bb232c0, 0xc20b300000, 0x1000, 0x1000, 0x7f5d6b8951d0, 0x0, 0x0) /usr/local/go/src/io/io.go:408 +0xce bufio.(*Reader).fill(0xc20ba110e0) /usr/local/go/src/bufio/bufio.go:97 +0x1ce bufio.(*Reader).ReadSlice(0xc20ba110e0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:295 +0x257 bufio.(*Reader).ReadLine(0xc20ba110e0, 0x0, 0x0, 0x0, 0xc204422a00, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:324 +0x62 net/textproto.(*Reader).readLineSlice(0xc243dd5920, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/textproto/reader.go:55 +0x9e net/textproto.(*Reader).ReadLine(0xc243dd5920, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/textproto/reader.go:36 +0x4f net/http.ReadRequest(0xc20ba110e0, 0xc21884b520, 0x0, 0x0) /usr/local/go/src/net/http/request.go:598 +0xcb net/http.(*conn).readRequest(0xc20b03f180, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:586 +0x26f net/http.(*conn).serve(0xc20b03f180) /usr/local/go/src/net/http/server.go:1162 +0x69e created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:1751 +0x35e goroutine 694 [chan receive]: github.com/couchbase/indexing/secondary/dcp.(*DcpFeed).genServer(0xc20bd5ca20, 0xc20bd5c960, 0xabba) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:232 +0xea created by github.com/couchbase/indexing/secondary/dcp.(*Bucket).StartDcpFeedOver /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:158 +0x558 goroutine 693 [IO wait]: net.(*pollDesc).Wait(0xc20bceb870, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bceb870, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bceb810, 0xc20bdc5340, 0x18, 0x18, 0x0, 0x7f5d6ba40e70, 0xc257a21958) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af1cdc0, 0xc20bdc5340, 0x18, 0x18, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc io.ReadAtLeast(0x7f5d6ba4c740, 0xc20af1cdc0, 0xc20bdc5340, 0x18, 0x18, 0x18, 0x0, 0x0, 0x0) /usr/local/go/src/io/io.go:298 +0xf1 io.ReadFull(0x7f5d6ba4c740, 0xc20af1cdc0, 0xc20bdc5340, 0x18, 0x18, 0x1400, 0x0, 0x0) /usr/local/go/src/io/io.go:316 +0x6d github.com/couchbase/indexing/secondary/dcp/transport.(*MCRequest).Receive(0xc247ef8180, 0x7f5d6ba4c740, 0xc20af1cdc0, 0xc20bdc5340, 0x18, 0x18, 0x1418, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/mc_req.go:130 +0xee github.com/couchbase/indexing/secondary/dcp/transport/client.(*DcpFeed).doReceive(0xc20b33d680, 0xc20b33d6e0, 0xc20bd5a600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/client/dcp_feed.go:882 +0x1b0 created by github.com/couchbase/indexing/secondary/dcp/transport/client.NewDcpFeed /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/transport/client/dcp_feed.go:68 +0x4bf goroutine 666 [chan receive]: github.com/couchbase/indexing/secondary/dcp.(*DcpFeed).genServer(0xc20bd5c5a0, 0xc20bd5c4e0, 0xabba) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:232 +0xea created by github.com/couchbase/indexing/secondary/dcp.(*Bucket).StartDcpFeedOver /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/upr.go:158 +0x558 goroutine 677 [select]: net/http.(*persistConn).writeLoop(0xc20bb24f20) /usr/local/go/src/net/http/transport.go:945 +0x41d created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:661 +0xcbc goroutine 695 [chan receive]: github.com/couchbase/indexing/secondary/common.(*vbSeqnosReader).Routine(0xc20bdcc3c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:71 +0x6c created by github.com/couchbase/indexing/secondary/common.newVbSeqnosReader /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:54 +0xb2 goroutine 668 [select]: github.com/couchbase/indexing/secondary/indexer.func·097() /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:2637 +0x12c created by github.com/couchbase/indexing/secondary/indexer.(*timekeeper).startTimer /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:2645 +0x478 goroutine 70695 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc262d843c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2628a2730, 0x7f5d6ba4b5a8, 0xc212f85368, 0xc24759e4e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2628a2730, 0x7f5d6ba4b5a8, 0xc212f85368, 0xc24759e4e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212f85368) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130829 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c7708, 0xc20bcded20, 0xc20b27fa40, 0xc20b27faa0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 679 [chan receive]: github.com/couchbase/indexing/secondary/dataport.(*Server).genServer(0xc20b9c3880, 0xc20bee7200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:295 +0x17e created by github.com/couchbase/indexing/secondary/dataport.NewServer /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:172 +0x730 goroutine 680 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationStreamReader).run(0xc20bd90380) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_reader.go:160 +0x406 created by github.com/couchbase/indexing/secondary/indexer.CreateMutationStreamReader /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_reader.go:119 +0x87c goroutine 681 [chan receive, 22 minutes]: github.com/couchbase/indexing/secondary/indexer.(*mutationStreamReader).listenSupvCmd(0xc20bd90380) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_reader.go:201 +0x77 created by github.com/couchbase/indexing/secondary/indexer.CreateMutationStreamReader /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_reader.go:120 +0x896 goroutine 682 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationStreamReader).syncWorker(0xc20bd90380) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_reader.go:683 +0x107 created by github.com/couchbase/indexing/secondary/indexer.CreateMutationStreamReader /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_reader.go:122 +0x8b0 goroutine 683 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationStreamReader).startMutationStreamWorker(0xc20bd90380, 0x0, 0xc20bee73e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_reader.go:360 +0x3fe created by github.com/couchbase/indexing/secondary/indexer.(*mutationStreamReader).startWorkers /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stream_reader.go:513 +0x147 goroutine 32241 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21510e0d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21510e0d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21510e070, 0xc228cf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25294d0b8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214de88d0, 0xc228cf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214de88d0, 0xc228cf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214de88d0, 0xc228cf0000, 0xfa000, 0xfa000, 0xc214d20000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2144f9200, 0x7f5d6b8a1528, 0xc214de88d0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214de88d0, 0xc214d24d20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 755 [IO wait]: net.(*pollDesc).Wait(0xc20bfe1170, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bfe1170, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bfe1110, 0xc20c3bc000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2561b0fe8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20b0454e8, 0xc20c3bc000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20b0454e8, 0xc20c3bc000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20b0454e8, 0xc20c3bc000, 0xfa000, 0xfa000, 0x6fc230000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20c30d3b0, 0x7f5d6b8a1528, 0xc20b0454e8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/dataport.doReceive(0xc20be97e40, 0x18, 0xc20c30d3e0, 0xfa000, 0x7530, 0xc20bee71a0, 0xc20bee7200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:538 +0x254 created by github.com/couchbase/indexing/secondary/dataport.(*Server).startWorker /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:380 +0x392 goroutine 64311 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21d06b040, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2518534a0, 0x7f5d6ba4b5a8, 0xc21332d190, 0xc262295c80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2518534a0, 0x7f5d6ba4b5a8, 0xc21332d190, 0xc262295c80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21332d190) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 67357 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241064c80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc213214d20, 0x7f5d6ba4b5a8, 0xc212f84828, 0xc23aca7080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc213214d20, 0x7f5d6ba4b5a8, 0xc212f84828, 0xc23aca7080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212f84828) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131269 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffae78, 0xc2104fe480, 0xc2633762a0, 0xc263376300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131127 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214fef858, 0xc2136b71a0, 0xc26945e420, 0xc26945e480) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32178 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc249735680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25b471e00, 0x7f5d6ba4b5a8, 0xc216170598, 0xc23b9d2720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25b471e00, 0x7f5d6ba4b5a8, 0xc216170598, 0xc23b9d2720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216170598) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 12726 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d667900, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc213e7bf90, 0x7f5d6ba4b5a8, 0xc215e72c88, 0xc23e1f1bc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc213e7bf90, 0x7f5d6ba4b5a8, 0xc215e72c88, 0xc23e1f1bc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e72c88) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50967 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc215bcb540, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265d6c910, 0x7f5d6ba4b5a8, 0xc213b04b08, 0xc20b281f80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265d6c910, 0x7f5d6ba4b5a8, 0xc213b04b08, 0xc20b281f80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213b04b08) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 20120 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc215658a00, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc215658a00, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2156589a0, 0xc22320c000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21e7cdb40) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215a2b338, 0xc22320c000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215a2b338, 0xc22320c000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215a2b338, 0xc22320c000, 0xfa000, 0xfa000, 0xc215ec0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20c0ab0b0, 0x7f5d6b8a1528, 0xc215a2b338, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2b338, 0xc215ec18c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32293 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b93d720, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b93d720, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b93d6c0, 0xc229af0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc248607ff8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214834b30, 0xc229af0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214834b30, 0xc229af0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214834b30, 0xc229af0000, 0xfa000, 0xfa000, 0xc2144e0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2139a2420, 0x7f5d6b8a1528, 0xc214834b30, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214834b30, 0xc2144e1bc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 40745 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f034dc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc254c7d540, 0x7f5d6ba4b5a8, 0xc213aad650, 0xc25a17a9c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc254c7d540, 0x7f5d6ba4b5a8, 0xc213aad650, 0xc25a17a9c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213aad650) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32194 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23dccab40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25b7d3ef0, 0x7f5d6ba4b5a8, 0xc21ff39aa0, 0xc265d913e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25b7d3ef0, 0x7f5d6ba4b5a8, 0xc21ff39aa0, 0xc265d913e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff39aa0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 82973 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc22b3eb870, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc22b3eb870, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc22b3eb810, 0xc236d12000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25561b6b8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af1c458, 0xc236d12000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20af1c458, 0xc236d12000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20af1c458, 0xc236d12000, 0xfa000, 0xfa000, 0xc20e020000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20b089fb0, 0x7f5d6b8a1528, 0xc20af1c458, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af1c458, 0xc20e02ee40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32193 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b4f7720, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b4f7720, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b4f76c0, 0xc2294f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc252bdff30) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214b72868, 0xc2294f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214b72868, 0xc2294f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214b72868, 0xc2294f0000, 0xfa000, 0xfa000, 0xc214830000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214262d80, 0x7f5d6b8a1528, 0xc214b72868, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b72868, 0xc21483e480) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32269 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b93df70, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b93df70, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b93df10, 0xc229ef0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc256ce4958) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214834070, 0xc229ef0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214834070, 0xc229ef0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214834070, 0xc229ef0000, 0xfa000, 0xfa000, 0xc213740000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2134df1d0, 0x7f5d6b8a1528, 0xc214834070, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214834070, 0xc213740de0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 64237 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc219c06990, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc219c06990, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc219c06930, 0xc231ef0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ef514a8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21324a1e0, 0xc231ef0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21324a1e0, 0xc231ef0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21324a1e0, 0xc231ef0000, 0xfa000, 0xfa000, 0xc212970000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20d5e12c0, 0x7f5d6b8a1528, 0xc21324a1e0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21324a1e0, 0xc21297c660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 50957 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21bdab2c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2551e70e0, 0x7f5d6ba4b5a8, 0xc214c773b0, 0xc2425817a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2551e70e0, 0x7f5d6ba4b5a8, 0xc214c773b0, 0xc2425817a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214c773b0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131228 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffb1f0, 0xc2162145a0, 0xc2611e6f60, 0xc2611e6fc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129003 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2147a19f0, 0xc213683bc0, 0xc20cfecfc0, 0xc20cfed020) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 51006 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc22b3ea450, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc22b3ea450, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc22b3ea3f0, 0xc22f4f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21fede150) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21437f5c0, 0xc22f4f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21437f5c0, 0xc22f4f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21437f5c0, 0xc22f4f0000, 0xfa000, 0xfa000, 0xc20e7b0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20b282bd0, 0x7f5d6b8a1528, 0xc21437f5c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21437f5c0, 0xc20e7bd500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 12554 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc249735040, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2628a20a0, 0x7f5d6ba4b5a8, 0xc2160c4050, 0xc23b9d2180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2628a20a0, 0x7f5d6ba4b5a8, 0xc2160c4050, 0xc23b9d2180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160c4050) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 45576 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc218849180, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2551e6c80, 0x7f5d6ba4b5a8, 0xc216266138, 0xc20b27f260) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2551e6c80, 0x7f5d6ba4b5a8, 0xc216266138, 0xc20b27f260) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216266138) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9501 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b336060, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b336060, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b336000, 0xc2246e0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bcc7c08) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2160c4020, 0xc2246e0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2160c4020, 0xc2246e0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2160c4020, 0xc2246e0000, 0xfa000, 0xfa000, 0xc215fd0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2153ee0c0, 0x7f5d6b8a1528, 0xc2160c4020, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160c4020, 0xc215fd0180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 45637 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bb1e1b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bb1e1b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bb1e150, 0xc22c4e4000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc256ce40d8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215c5ccb8, 0xc22c4e4000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215c5ccb8, 0xc22c4e4000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215c5ccb8, 0xc22c4e4000, 0xfa000, 0xfa000, 0xc20c050000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc215e4c000, 0x7f5d6b8a1528, 0xc215c5ccb8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215c5ccb8, 0xc20c0520c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32267 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c29fb80, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c29fb80, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c29fb20, 0xc229cf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc248607dd8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21462e898, 0xc229cf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21462e898, 0xc229cf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21462e898, 0xc229cf0000, 0xfa000, 0xfa000, 0xc213860000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc213540e10, 0x7f5d6b8a1528, 0xc21462e898, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21462e898, 0xc213860660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 50938 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241803cc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2661172c0, 0x7f5d6ba4b5a8, 0xc215cd34b8, 0xc264aea2a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2661172c0, 0x7f5d6ba4b5a8, 0xc215cd34b8, 0xc264aea2a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd34b8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131380 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b0457b0, 0xc215fc32c0, 0xc2150a7800, 0xc2150a7860) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 840 [IO wait]: net.(*pollDesc).Wait(0xc20c3b74f0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c3b74f0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c3b7490, 0xc20dd7e000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2574f0110) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20d8896b8, 0xc20dd7e000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20d8896b8, 0xc20dd7e000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20d8896b8, 0xc20dd7e000, 0xfa000, 0xfa000, 0x6fc230000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20db637d0, 0x7f5d6b8a1528, 0xc20d8896b8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/dataport.doReceive(0xc20be97e40, 0x18, 0xc20db63800, 0xfa000, 0x7530, 0xc20bee71a0, 0xc20bee7200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:538 +0x254 created by github.com/couchbase/indexing/secondary/dataport.(*Server).startWorker /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:380 +0x392 goroutine 62742 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20e598990, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20e598990, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20e598930, 0xc2314f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc219bb9ce0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21369e0d8, 0xc2314f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21369e0d8, 0xc2314f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21369e0d8, 0xc2314f0000, 0xfa000, 0xfa000, 0xc219e00000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc211423410, 0x7f5d6b8a1528, 0xc21369e0d8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21369e0d8, 0xc219e0ca80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32197 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23dccb540, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2560a8550, 0x7f5d6ba4b5a8, 0xc215fff788, 0xc265d91ce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2560a8550, 0x7f5d6ba4b5a8, 0xc215fff788, 0xc265d91ce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215fff788) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 12553 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2186503c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2618121e0, 0x7f5d6ba4b5a8, 0xc2160c4020, 0xc25fa2ac60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2618121e0, 0x7f5d6ba4b5a8, 0xc2160c4020, 0xc25fa2ac60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160c4020) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 23589 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23b198500, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc264376370, 0x7f5d6ba4b5a8, 0xc215cd2928, 0xc258bce3c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc264376370, 0x7f5d6ba4b5a8, 0xc215cd2928, 0xc258bce3c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd2928) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9088122 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139a5a40, 0xc20e9ed560, 0xc2459e8480, 0xc2459e84e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 72464 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc213163900, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d40d2c0, 0x7f5d6ba4b5a8, 0xc210df00c8, 0xc263f371a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d40d2c0, 0x7f5d6ba4b5a8, 0xc210df00c8, 0xc263f371a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210df00c8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9129758 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2bb10, 0xc215e07e00, 0xc23b94dbc0, 0xc23b94dc20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 17007 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bd06680, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bd06680, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bd06620, 0xc2264f8000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24a5637c8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af4eb30, 0xc2264f8000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20af4eb30, 0xc2264f8000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20af4eb30, 0xc2264f8000, 0xfa000, 0xfa000, 0xc216270000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e5df4a0, 0x7f5d6b8a1528, 0xc20af4eb30, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4eb30, 0xc216274480) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131117 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2154183a8, 0xc2141daa20, 0xc258bcea80, 0xc258bceae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 38856 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2421d9cc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2168f9180, 0x7f5d6ba4b5a8, 0xc213ed9080, 0xc248010000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2168f9180, 0x7f5d6ba4b5a8, 0xc213ed9080, 0xc248010000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ed9080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 47247 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc25de36780, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265d6d590, 0x7f5d6ba4b5a8, 0xc213ffb1f0, 0xc2611e6f60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265d6d590, 0x7f5d6ba4b5a8, 0xc213ffb1f0, 0xc2611e6f60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffb1f0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32231 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc244c48500, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc212153220, 0x7f5d6ba4b5a8, 0xc2152fb258, 0xc21758b560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc212153220, 0x7f5d6ba4b5a8, 0xc2152fb258, 0xc21758b560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2152fb258) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130860 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214321c68, 0xc21392e840, 0xc20b281d40, 0xc20b281da0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 51111 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21bdaa140, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25596a370, 0x7f5d6ba4b5a8, 0xc2143207c8, 0xc242581560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25596a370, 0x7f5d6ba4b5a8, 0xc2143207c8, 0xc242581560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2143207c8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 72465 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2338ec8b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2338ec8b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2338ec850, 0xc2346f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2186989b8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc210df00c8, 0xc2346f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc210df00c8, 0xc2346f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc210df00c8, 0xc2346f0000, 0xfa000, 0xfa000, 0xc20d600000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2338eaa20, 0x7f5d6b8a1528, 0xc210df00c8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210df00c8, 0xc20d603200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 74681 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc216f2c000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d40ceb0, 0x7f5d6ba4b5a8, 0xc215cd2d40, 0xc261250180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d40ceb0, 0x7f5d6ba4b5a8, 0xc215cd2d40, 0xc261250180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd2d40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 11551 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc208069e00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc218a3bb30, 0x7f5d6ba4b5a8, 0xc216171110, 0xc238f47560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc218a3bb30, 0x7f5d6ba4b5a8, 0xc216171110, 0xc238f47560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216171110) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 74817 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21d06bcc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc251853d10, 0x7f5d6ba4b5a8, 0xc21462e850, 0xc25fa2a6c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc251853d10, 0x7f5d6ba4b5a8, 0xc21462e850, 0xc25fa2a6c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21462e850) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 23620 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a045b10, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a045b10, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a045ab0, 0xc220e56000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24d673bb8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215fff7f0, 0xc220e56000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215fff7f0, 0xc220e56000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215fff7f0, 0xc220e56000, 0xfa000, 0xfa000, 0xc2143c0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e5bad80, 0x7f5d6b8a1528, 0xc215fff7f0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215fff7f0, 0xc2143cb380) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 65940 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20f02f950, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20f02f950, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20f02f8f0, 0xc231df0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bcc6ab8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21332d190, 0xc231df0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21332d190, 0xc231df0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21332d190, 0xc231df0000, 0xfa000, 0xfa000, 0xc213290000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21a7de4e0, 0x7f5d6b8a1528, 0xc21332d190, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21332d190, 0xc213292060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131072 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a75d98, 0xc21429e660, 0xc2150a7380, 0xc2150a73e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131049 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffbee0, 0xc20bd75c80, 0xc261250840, 0xc2612508a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 25172 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20af45090, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af45090, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af45030, 0xc220f50000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2444568a8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215b43c20, 0xc220f50000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215b43c20, 0xc220f50000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215b43c20, 0xc220f50000, 0xfa000, 0xfa000, 0xc213cb0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20b67e7e0, 0x7f5d6b8a1528, 0xc215b43c20, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b43c20, 0xc213cbcea0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9130158 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214de88d0, 0xc214d24d20, 0xc238f47c20, 0xc238f47c80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 49261 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b9c3790, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b9c3790, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b9c3730, 0xc22d484000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24a563398) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e324f8, 0xc22d484000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e324f8, 0xc22d484000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e324f8, 0xc22d484000, 0xfa000, 0xfa000, 0xc214a90000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21533f710, 0x7f5d6b8a1528, 0xc215e324f8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e324f8, 0xc214a9ae40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 70679 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a17eed0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a17eed0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a17ee70, 0xc232ef0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc252bdff78) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc212dec098, 0xc232ef0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc212dec098, 0xc232ef0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc212dec098, 0xc232ef0000, 0xfa000, 0xfa000, 0xc21ec90000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21ee77530, 0x7f5d6b8a1528, 0xc212dec098, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212dec098, 0xc21ec91440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 74815 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210f01e90, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210f01e90, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210f01e30, 0xc236254000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211d1b570) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214a17c00, 0xc236254000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214a17c00, 0xc236254000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214a17c00, 0xc236254000, 0xfa000, 0xfa000, 0xc215e00000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2160e5380, 0x7f5d6b8a1528, 0xc214a17c00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a17c00, 0xc215e07620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9129937 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd2ac0, 0xc215fd58c0, 0xc23e1f1980, 0xc23e1f19e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 74729 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc216f2cf00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc26480cc30, 0x7f5d6ba4b5a8, 0xc21463ae40, 0xc261250de0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc26480cc30, 0x7f5d6ba4b5a8, 0xc21463ae40, 0xc261250de0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21463ae40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 37428 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc24f819e00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25596a550, 0x7f5d6ba4b5a8, 0xc2142719f8, 0xc238d1b020) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25596a550, 0x7f5d6ba4b5a8, 0xc2142719f8, 0xc238d1b020) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2142719f8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 10416 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc249734f00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24566a7d0, 0x7f5d6ba4b5a8, 0xc20b044c58, 0xc23b9d2060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24566a7d0, 0x7f5d6ba4b5a8, 0xc20b044c58, 0xc23b9d2060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b044c58) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131075 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e32c70, 0xc210d5ed20, 0xc21c04ce40, 0xc21c04cea0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 81376 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20ba20760, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20ba20760, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20ba20700, 0xc236c18000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240b0a558) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2146170c0, 0xc236c18000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2146170c0, 0xc236c18000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2146170c0, 0xc236c18000, 0xfa000, 0xfa000, 0xc216100000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20b338450, 0x7f5d6b8a1528, 0xc2146170c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2146170c0, 0xc216103f20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 51044 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210f00920, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210f00920, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210f008c0, 0xc22eff0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25294c738) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2145e06d8, 0xc22eff0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2145e06d8, 0xc22eff0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2145e06d8, 0xc22eff0000, 0xfa000, 0xfa000, 0xc2111f0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20dcec630, 0x7f5d6b8a1528, 0xc2145e06d8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2145e06d8, 0xc2111f14a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 51061 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc244c49040, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2121538b0, 0x7f5d6ba4b5a8, 0xc21437ed40, 0xc21758bd40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2121538b0, 0x7f5d6ba4b5a8, 0xc21437ed40, 0xc21758bd40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21437ed40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 20074 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241065040, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc218a3b950, 0x7f5d6ba4b5a8, 0xc215e84900, 0xc21aab10e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc218a3b950, 0x7f5d6ba4b5a8, 0xc215e84900, 0xc21aab10e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e84900) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131054 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21463ae40, 0xc2153bd620, 0xc261250de0, 0xc261250e40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 86418 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bd06d80, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bd06d80, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bd06d20, 0xc2374f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25102c1b8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2140a6918, 0xc2374f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2140a6918, 0xc2374f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2140a6918, 0xc2374f0000, 0xfa000, 0xfa000, 0xc2100c0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc212d7ef00, 0x7f5d6b8a1528, 0xc2140a6918, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2140a6918, 0xc2100cd920) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72472 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21c4b4b40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2551e6d20, 0x7f5d6ba4b5a8, 0xc210ad43b0, 0xc238d1c9c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2551e6d20, 0x7f5d6ba4b5a8, 0xc210ad43b0, 0xc238d1c9c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210ad43b0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 30308 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23d416780, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2153c21e0, 0x7f5d6ba4b5a8, 0xc20af4eab8, 0xc21c04cae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2153c21e0, 0x7f5d6ba4b5a8, 0xc20af4eab8, 0xc21c04cae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4eab8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 35809 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc211823040, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25596aaf0, 0x7f5d6ba4b5a8, 0xc2139ce550, 0xc238d1b6e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25596aaf0, 0x7f5d6ba4b5a8, 0xc2139ce550, 0xc238d1b6e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139ce550) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 20124 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b9c3cd0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b9c3cd0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b9c3c70, 0xc2235f4000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23e4c1db0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21fac0268, 0xc2235f4000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21fac0268, 0xc2235f4000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21fac0268, 0xc2235f4000, 0xfa000, 0xfa000, 0xc215ec0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2137dc840, 0x7f5d6b8a1528, 0xc21fac0268, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21fac0268, 0xc215ec1bc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 12757 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219b0f7c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc266117c70, 0x7f5d6ba4b5a8, 0xc20b0457b0, 0xc2150a7800) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc266117c70, 0x7f5d6ba4b5a8, 0xc20b0457b0, 0xc2150a7800) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b0457b0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 17048 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bd066f0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bd066f0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bd06690, 0xc2240cc000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24458a5f8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af4f028, 0xc2240cc000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20af4f028, 0xc2240cc000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20af4f028, 0xc2240cc000, 0xfa000, 0xfa000, 0xc21b4c0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc215f020f0, 0x7f5d6b8a1528, 0xc20af4f028, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4f028, 0xc21b4c6ea0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 52678 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c2b19c0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c2b19c0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c2b1960, 0xc22fcf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23f914350) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20bb598a8, 0xc22fcf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20bb598a8, 0xc22fcf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20bb598a8, 0xc22fcf0000, 0xfa000, 0xfa000, 0xc20b900000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20c30de00, 0x7f5d6b8a1528, 0xc20bb598a8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bb598a8, 0xc20b906840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 59671 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc217385f70, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc217385f70, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc217385f10, 0xc230ef0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24d673988) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21ff38b08, 0xc230ef0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21ff38b08, 0xc230ef0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21ff38b08, 0xc230ef0000, 0xfa000, 0xfa000, 0xc20def0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21bb942a0, 0x7f5d6b8a1528, 0xc21ff38b08, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff38b08, 0xc20def6720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131043 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd2d40, 0xc2161b4360, 0xc261250180, 0xc2612501e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 70811 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21de34300, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21de34300, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21de342a0, 0xc2339f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24a563948) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2110561d8, 0xc2339f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2110561d8, 0xc2339f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2110561d8, 0xc2339f0000, 0xfa000, 0xfa000, 0xc20bb90000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc213b7b740, 0x7f5d6b8a1528, 0xc2110561d8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2110561d8, 0xc20bb90f00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 51033 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210f007d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210f007d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210f00770, 0xc22eef0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc244456e78) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2147a1cc8, 0xc22eef0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2147a1cc8, 0xc22eef0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2147a1cc8, 0xc22eef0000, 0xfa000, 0xfa000, 0xc211470000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ec12db0, 0x7f5d6b8a1528, 0xc2147a1cc8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2147a1cc8, 0xc21147b800) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 52724 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23b198640, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc264376460, 0x7f5d6ba4b5a8, 0xc213f0f9c8, 0xc258bce4e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc264376460, 0x7f5d6ba4b5a8, 0xc213f0f9c8, 0xc258bce4e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213f0f9c8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 25158 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a0fb5d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a0fb5d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a0fb570, 0xc221432000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bcc7278) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215d0c980, 0xc221432000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215d0c980, 0xc221432000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215d0c980, 0xc221432000, 0xfa000, 0xfa000, 0xc2141d0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20d9211a0, 0x7f5d6b8a1528, 0xc215d0c980, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215d0c980, 0xc2141dbf20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 67445 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21966df00, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21966df00, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21966dea0, 0xc2324f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25294cd28) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21361c148, 0xc2324f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21361c148, 0xc2324f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21361c148, 0xc2324f0000, 0xfa000, 0xfa000, 0xc21b440000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21beb1f20, 0x7f5d6b8a1528, 0xc21361c148, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21361c148, 0xc21b4475c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 70615 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc213068920, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc213068920, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2130688c0, 0xc232df0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23be9e028) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc212f9db60, 0xc232df0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc212f9db60, 0xc232df0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc212f9db60, 0xc232df0000, 0xfa000, 0xfa000, 0xc21f760000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21b5f41e0, 0x7f5d6b8a1528, 0xc212f9db60, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212f9db60, 0xc21f7692c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 83076 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219f31a40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d5f3590, 0x7f5d6ba4b5a8, 0xc20af1c458, 0xc238d1d740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d5f3590, 0x7f5d6ba4b5a8, 0xc20af1c458, 0xc238d1d740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af1c458) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 79686 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a0fabc0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a0fabc0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a0fab60, 0xc236736000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240d6b248) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215b42a50, 0xc236736000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215b42a50, 0xc236736000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215b42a50, 0xc236736000, 0xfa000, 0xfa000, 0xc21c800000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2144775c0, 0x7f5d6b8a1528, 0xc215b42a50, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b42a50, 0xc21c80d560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9130945 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e32748, 0xc214ab3ec0, 0xc264aea060, 0xc264aea0c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 70707 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20bbab680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23fa33400, 0x7f5d6ba4b5a8, 0xc21367e9c8, 0xc25fa6b860) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23fa33400, 0x7f5d6ba4b5a8, 0xc21367e9c8, 0xc25fa6b860) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21367e9c8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50994 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc212a14840, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc212a14840, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc212a147e0, 0xc22e424000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc210321920) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214c76118, 0xc22e424000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214c76118, 0xc22e424000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214c76118, 0xc22e424000, 0xfa000, 0xfa000, 0xc213640000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e5e2420, 0x7f5d6b8a1528, 0xc214c76118, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214c76118, 0xc213641c20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 56927 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc25d682140, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc259030ff0, 0x7f5d6ba4b5a8, 0xc213657010, 0xc263376180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc259030ff0, 0x7f5d6ba4b5a8, 0xc213657010, 0xc263376180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213657010) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 74850 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20eb43640, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20eb43640, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20eb435e0, 0xc236448000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240d6bb28) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21463ae40, 0xc236448000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21463ae40, 0xc236448000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21463ae40, 0xc236448000, 0xfa000, 0xfa000, 0xc2153b0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc215faec30, 0x7f5d6b8a1528, 0xc21463ae40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21463ae40, 0xc2153bd620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 42353 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21d06aa00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc26129b7c0, 0x7f5d6ba4b5a8, 0xc213ed8168, 0xc20d0db440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc26129b7c0, 0x7f5d6ba4b5a8, 0xc213ed8168, 0xc20d0db440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ed8168) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131106 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214835df0, 0xc2108634a0, 0xc24e0f1e00, 0xc24e0f1e60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 28300 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a0fbfe0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a0fbfe0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a0fbf80, 0xc221df6000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc252036148) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215d0d668, 0xc221df6000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215d0d668, 0xc221df6000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215d0d668, 0xc221df6000, 0xfa000, 0xfa000, 0xc2132f0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20c30db00, 0x7f5d6b8a1528, 0xc215d0d668, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215d0d668, 0xc2132ffec0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 38817 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bb1fe20, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bb1fe20, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bb1fdc0, 0xc22b1f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240b0a958) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214349d28, 0xc22b1f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214349d28, 0xc22b1f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214349d28, 0xc22b1f0000, 0xfa000, 0xfa000, 0xc20bd30000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc216442d50, 0x7f5d6b8a1528, 0xc214349d28, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214349d28, 0xc20bd33020) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 50873 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc213547330, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc213547330, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2135472d0, 0xc22d966000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2422e82b8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2162676c8, 0xc22d966000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2162676c8, 0xc22d966000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2162676c8, 0xc22d966000, 0xfa000, 0xfa000, 0xc214110000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21366f2c0, 0x7f5d6b8a1528, 0xc2162676c8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2162676c8, 0xc214116b40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 12770 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f415540, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d40d4f0, 0x7f5d6ba4b5a8, 0xc216266ef0, 0xc26945e300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d40d4f0, 0x7f5d6ba4b5a8, 0xc216266ef0, 0xc26945e300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216266ef0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50945 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bceb4f0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bceb4f0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bceb490, 0xc22e32a000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21fede648) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214f37198, 0xc22e32a000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214f37198, 0xc22e32a000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214f37198, 0xc22e32a000, 0xfa000, 0xfa000, 0xc213640000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21366fb30, 0x7f5d6b8a1528, 0xc214f37198, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f37198, 0xc213640c00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 40724 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc215bcb680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2590306e0, 0x7f5d6ba4b5a8, 0xc215a2a660, 0xc2611e60c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2590306e0, 0x7f5d6ba4b5a8, 0xc215a2a660, 0xc2611e60c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2a660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9129946 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2130a2f68, 0xc216010480, 0xc265d91740, 0xc265d917a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 23411 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21c4b4a00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d40cf50, 0x7f5d6ba4b5a8, 0xc215419d88, 0xc238d1c8a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d40cf50, 0x7f5d6ba4b5a8, 0xc215419d88, 0xc238d1c8a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215419d88) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 11486 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20af44140, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af44140, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af440e0, 0xc225be0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2422e8428) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215fff0e0, 0xc225be0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215fff0e0, 0xc225be0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215fff0e0, 0xc225be0000, 0xfa000, 0xfa000, 0xc216210000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214f45350, 0x7f5d6b8a1528, 0xc215fff0e0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215fff0e0, 0xc21621ecc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 50983 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21510f410, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21510f410, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21510f3b0, 0xc22db5a000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc244456728) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215cd34b8, 0xc22db5a000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215cd34b8, 0xc22db5a000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215cd34b8, 0xc22db5a000, 0xfa000, 0xfa000, 0xc214020000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20f67c540, 0x7f5d6b8a1528, 0xc215cd34b8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd34b8, 0xc21402e540) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 1898 [IO wait]: net.(*pollDesc).Wait(0xc20ba20060, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20ba20060, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20ba20000, 0xc21015c000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2495eaaf0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20e2d9f00, 0xc21015c000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20e2d9f00, 0xc21015c000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20e2d9f00, 0xc21015c000, 0xfa000, 0xfa000, 0x6fc230000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20f665a70, 0x7f5d6b8a1528, 0xc20e2d9f00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/dataport.doReceive(0xc20be97e40, 0x18, 0xc20f665aa0, 0xfa000, 0x7530, 0xc20bee71a0, 0xc20bee7200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:538 +0x254 created by github.com/couchbase/indexing/secondary/dataport.(*Server).startWorker /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:380 +0x392 goroutine 9129950 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215419c18, 0xc2161e3860, 0xc265d91bc0, 0xc265d91c20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 39075 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23d417a40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2153c28c0, 0x7f5d6ba4b5a8, 0xc21419b0b8, 0xc21c04d3e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2153c28c0, 0x7f5d6ba4b5a8, 0xc21419b0b8, 0xc21c04d3e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21419b0b8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 40681 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b26e530, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b26e530, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b26e4d0, 0xc22b9f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21ad5a5c0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213aadb70, 0xc22b9f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213aadb70, 0xc22b9f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213aadb70, 0xc22b9f0000, 0xfa000, 0xfa000, 0xc2118e0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21f5685a0, 0x7f5d6b8a1528, 0xc213aadb70, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213aadb70, 0xc2118e8780) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9130148 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff38a68, 0xc21d818840, 0xc238f470e0, 0xc238f47140) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 27043 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21ff7d040, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2560a8af0, 0x7f5d6ba4b5a8, 0xc21533c010, 0xc25ecba720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2560a8af0, 0x7f5d6ba4b5a8, 0xc21533c010, 0xc25ecba720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21533c010) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130821 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2138c0670, 0xc20dbdcae0, 0xc20b27f140, 0xc20b27f1a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 31159 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc262d84500, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2560a89b0, 0x7f5d6ba4b5a8, 0xc215e40af0, 0xc24759e600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2560a89b0, 0x7f5d6ba4b5a8, 0xc215e40af0, 0xc24759e600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e40af0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 1505 [IO wait]: net.(*pollDesc).Wait(0xc20c3b6610, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c3b6610, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c3b65b0, 0xc20eef6000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc252bc3f40) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20ec6df88, 0xc20eef6000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20ec6df88, 0xc20eef6000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20ec6df88, 0xc20eef6000, 0xfa000, 0xfa000, 0x6fc230000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20eecf710, 0x7f5d6b8a1528, 0xc20ec6df88, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/dataport.doReceive(0xc20be97e40, 0x18, 0xc20eecf740, 0xfa000, 0x7530, 0xc20bee71a0, 0xc20bee7200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:538 +0x254 created by github.com/couchbase/indexing/secondary/dataport.(*Server).startWorker /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:380 +0x392 goroutine 28444 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20ce6a060, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20ce6a060, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20ce6a000, 0xc2224cc000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24a563458) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2080c6828, 0xc2224cc000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2080c6828, 0xc2224cc000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2080c6828, 0xc2224cc000, 0xfa000, 0xfa000, 0xc2131e0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20af47a10, 0x7f5d6b8a1528, 0xc2080c6828, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c6828, 0xc2131eb320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 30478 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20ce6b950, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20ce6b950, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20ce6b8f0, 0xc2279f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ef505b8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21ff39828, 0xc2279f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21ff39828, 0xc2279f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21ff39828, 0xc2279f0000, 0xfa000, 0xfa000, 0xc21d3f0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214c79290, 0x7f5d6b8a1528, 0xc21ff39828, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff39828, 0xc21d3fbb00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 91707 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21816f3a0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21816f3a0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21816f340, 0xc2385f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc218698f98) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21361c738, 0xc2385f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21361c738, 0xc2385f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21361c738, 0xc2385f0000, 0xfa000, 0xfa000, 0xc21db70000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2109c37d0, 0x7f5d6b8a1528, 0xc21361c738, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21361c738, 0xc21db7f740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 55983 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23aabcc80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25596a280, 0x7f5d6ba4b5a8, 0xc2136fd060, 0xc265d8fc80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25596a280, 0x7f5d6ba4b5a8, 0xc2136fd060, 0xc265d8fc80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2136fd060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 72532 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc215a6c0d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc215a6c0d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc215a6c070, 0xc22072c000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240d6b7f8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20bfa2020, 0xc22072c000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20bfa2020, 0xc22072c000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20bfa2020, 0xc22072c000, 0xfa000, 0xfa000, 0xc219a60000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20c24a060, 0x7f5d6b8a1528, 0xc20bfa2020, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bfa2020, 0xc219a62000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32264 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23aabd180, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2551e6f50, 0x7f5d6ba4b5a8, 0xc214a08ab0, 0xc265d8fda0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2551e6f50, 0x7f5d6ba4b5a8, 0xc214a08ab0, 0xc265d8fda0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a08ab0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 88125 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2328ec140, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc256203540, 0x7f5d6ba4b5a8, 0xc213ed9a08, 0xc20cfecd80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc256203540, 0x7f5d6ba4b5a8, 0xc213ed9a08, 0xc20cfecd80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ed9a08) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 31168 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b4f6df0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b4f6df0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b4f6d90, 0xc227ff0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23d886ab0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215b43560, 0xc227ff0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215b43560, 0xc227ff0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215b43560, 0xc227ff0000, 0xfa000, 0xfa000, 0xc216180000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214e3b470, 0x7f5d6b8a1528, 0xc215b43560, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b43560, 0xc216181740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72563 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2630e6140, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc218a3bd10, 0x7f5d6ba4b5a8, 0xc212bbfb30, 0xc238f477a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc218a3bd10, 0x7f5d6ba4b5a8, 0xc212bbfb30, 0xc238f477a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212bbfb30) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 64238 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc211c7bcd0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc211c7bcd0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc211c7bc70, 0xc231ff0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23d8867f8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2131d0390, 0xc231ff0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2131d0390, 0xc231ff0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2131d0390, 0xc231ff0000, 0xfa000, 0xfa000, 0xc212970000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21fb1a780, 0x7f5d6b8a1528, 0xc2131d0390, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2131d0390, 0xc21297c6c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 21897 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc244c48a00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2121535e0, 0x7f5d6ba4b5a8, 0xc215edacb0, 0xc21758b9e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2121535e0, 0x7f5d6ba4b5a8, 0xc215edacb0, 0xc21758b9e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215edacb0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32325 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2328ec640, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc264ccfb80, 0x7f5d6ba4b5a8, 0xc2147a19f0, 0xc20cfecfc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc264ccfb80, 0x7f5d6ba4b5a8, 0xc2147a19f0, 0xc20cfecfc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2147a19f0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130792 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216267008, 0xc219316840, 0xc2646253e0, 0xc264625440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 10010 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20af1a1b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af1a1b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af1a150, 0xc226c88000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc212a6ab18) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20bb58bd8, 0xc226c88000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20bb58bd8, 0xc226c88000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20bb58bd8, 0xc226c88000, 0xfa000, 0xfa000, 0xc21d490000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ca13d10, 0x7f5d6b8a1528, 0xc20bb58bd8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bb58bd8, 0xc21d492000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72585 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc215a6cae0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc215a6cae0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc215a6ca80, 0xc23569c000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23faf8298) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e859a8, 0xc23569c000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e859a8, 0xc23569c000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e859a8, 0xc23569c000, 0xfa000, 0xfa000, 0xc21d3b0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc212526e10, 0x7f5d6b8a1528, 0xc215e859a8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e859a8, 0xc21d3bca20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 45613 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bf9ab50, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bf9ab50, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bf9aaf0, 0xc22cbba000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211dac708) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213ffabd0, 0xc22cbba000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213ffabd0, 0xc22cbba000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213ffabd0, 0xc22cbba000, 0xfa000, 0xfa000, 0xc216190000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2162ec900, 0x7f5d6b8a1528, 0xc213ffabd0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffabd0, 0xc216192180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131073 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e73288, 0xc215fd1aa0, 0xc2150a74a0, 0xc2150a7500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32290 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21bdaa000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24a7a63c0, 0x7f5d6ba4b5a8, 0xc214b76bd0, 0xc255c3e8a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24a7a63c0, 0x7f5d6ba4b5a8, 0xc214b76bd0, 0xc255c3e8a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b76bd0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 20200 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241a6f900, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24e2d2230, 0x7f5d6ba4b5a8, 0xc20af4fb10, 0xc263f37740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24e2d2230, 0x7f5d6ba4b5a8, 0xc20af4fb10, 0xc263f37740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4fb10) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 43956 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21d06b180, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc251853630, 0x7f5d6ba4b5a8, 0xc214321070, 0xc262295da0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc251853630, 0x7f5d6ba4b5a8, 0xc214321070, 0xc262295da0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214321070) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131255 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af1c458, 0xc20e02ee40, 0xc238d1d740, 0xc238d1d7a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32234 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c29f720, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c29f720, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c29f6c0, 0xc2286f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25102c2d8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215366a80, 0xc2286f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215366a80, 0xc2286f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215366a80, 0xc2286f0000, 0xfa000, 0xfa000, 0xc215eb0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21486d530, 0x7f5d6b8a1528, 0xc215366a80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215366a80, 0xc215eb0060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9130954 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215d0d868, 0xc215fc4cc0, 0xc238d1aba0, 0xc238d1ac00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131157 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff390c0, 0xc2161842a0, 0xc264aea4e0, 0xc264aea540) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 29416 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d25d680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24566b220, 0x7f5d6ba4b5a8, 0xc216266aa8, 0xc240bbaba0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24566b220, 0x7f5d6ba4b5a8, 0xc216266aa8, 0xc240bbaba0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216266aa8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 62743 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d25c500, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2551e7590, 0x7f5d6ba4b5a8, 0xc2134eec68, 0xc240bba3c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2551e7590, 0x7f5d6ba4b5a8, 0xc2134eec68, 0xc240bba3c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2134eec68) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32302 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21510eca0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21510eca0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21510ec40, 0xc22a1f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ac8bd08) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214617ed8, 0xc22a1f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214617ed8, 0xc22a1f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214617ed8, 0xc22a1f0000, 0xfa000, 0xfa000, 0xc213550000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2134df7a0, 0x7f5d6b8a1528, 0xc214617ed8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214617ed8, 0xc21355ed20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32238 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21c4b4dc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d5f20f0, 0x7f5d6ba4b5a8, 0xc214f1f1b0, 0xc238d1cc00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d5f20f0, 0x7f5d6ba4b5a8, 0xc214f1f1b0, 0xc238d1cc00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1f1b0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131250 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622e658, 0xc215fd2360, 0xc238d1d1a0, 0xc238d1d200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 72156 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2338ec370, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2338ec370, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2338ec310, 0xc233bf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240d6afd8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21149f588, 0xc233bf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21149f588, 0xc233bf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21149f588, 0xc233bf0000, 0xfa000, 0xfa000, 0xc20c610000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2338ea480, 0x7f5d6b8a1528, 0xc21149f588, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21149f588, 0xc20c616ea0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32291 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219b0f180, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc266117860, 0x7f5d6ba4b5a8, 0xc214b77058, 0xc2150a7260) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc266117860, 0x7f5d6ba4b5a8, 0xc214b77058, 0xc2150a7260) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b77058) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130957 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c6828, 0xc2131eb320, 0xc238d1af00, 0xc238d1af60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 70681 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc218848000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc238e18af0, 0x7f5d6ba4b5a8, 0xc212ded408, 0xc261b5ef00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc238e18af0, 0x7f5d6ba4b5a8, 0xc212ded408, 0xc261b5ef00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212ded408) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 37429 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c3b7410, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c3b7410, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c3b73b0, 0xc22adf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ac8baa8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215631b50, 0xc22adf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215631b50, 0xc22adf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215631b50, 0xc22adf0000, 0xfa000, 0xfa000, 0xc20bb00000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20b0306c0, 0x7f5d6b8a1528, 0xc215631b50, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215631b50, 0xc20bb009c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 11271 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc218650780, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2618124b0, 0x7f5d6ba4b5a8, 0xc216266c68, 0xc25fa2afc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2618124b0, 0x7f5d6ba4b5a8, 0xc216266c68, 0xc25fa2afc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216266c68) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9086276 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139a5030, 0xc20b906c00, 0xc2458d34a0, 0xc2458d3500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131315 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffb668, 0xc21619a420, 0xc263f37aa0, 0xc263f37b00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 64116 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20e598f40, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20e598f40, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20e598ee0, 0xc2319f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211d530d8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21369fd40, 0xc2319f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21369fd40, 0xc2319f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21369fd40, 0xc2319f0000, 0xfa000, 0xfa000, 0xc21afe0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20c10fdd0, 0x7f5d6b8a1528, 0xc21369fd40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21369fd40, 0xc21afe2720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 67385 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a17e990, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a17e990, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a17e930, 0xc2322f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211da1048) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213574b38, 0xc2322f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213574b38, 0xc2322f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213574b38, 0xc2322f0000, 0xfa000, 0xfa000, 0xc219a50000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc219942360, 0x7f5d6b8a1528, 0xc213574b38, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213574b38, 0xc219a5cde0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 70702 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c2b05a0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c2b05a0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c2b0540, 0xc234ecc000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ac8b1b8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21ff617f0, 0xc234ecc000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21ff617f0, 0xc234ecc000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21ff617f0, 0xc234ecc000, 0xfa000, 0xfa000, 0xc210920000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20b565320, 0x7f5d6b8a1528, 0xc21ff617f0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff617f0, 0xc2109226c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72159 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2421d9680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23f2ce550, 0x7f5d6ba4b5a8, 0xc211a49658, 0xc2150a63c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23f2ce550, 0x7f5d6ba4b5a8, 0xc211a49658, 0xc2150a63c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211a49658) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 49334 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bceb100, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bceb100, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bceb0a0, 0xc22d86c000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc244456d08) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc216170f70, 0xc22d86c000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc216170f70, 0xc22d86c000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc216170f70, 0xc22d86c000, 0xfa000, 0xfa000, 0xc20b040000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214c8fec0, 0x7f5d6b8a1528, 0xc216170f70, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216170f70, 0xc20b041bc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32257 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2630e6640, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2565720f0, 0x7f5d6ba4b5a8, 0xc214de88d0, 0xc238f47c20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2565720f0, 0x7f5d6ba4b5a8, 0xc214de88d0, 0xc238f47c20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214de88d0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 29301 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bcea290, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bcea290, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bcea230, 0xc2226c0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21606ecd8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2162662c8, 0xc2226c0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2162662c8, 0xc2226c0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2162662c8, 0xc2226c0000, 0xfa000, 0xfa000, 0xc213fa0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2162edf50, 0x7f5d6b8a1528, 0xc2162662c8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2162662c8, 0xc213fa1b60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131099 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212f85368, 0xc20c351440, 0xc24759e4e0, 0xc24759e540) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32192 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2421d9b80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2608299a0, 0x7f5d6ba4b5a8, 0xc214b72868, 0xc25e3c1ec0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2608299a0, 0x7f5d6ba4b5a8, 0xc214b72868, 0xc25e3c1ec0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b72868) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 69007 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f035b80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d5f2640, 0x7f5d6ba4b5a8, 0xc2131d17a0, 0xc264625ce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d5f2640, 0x7f5d6ba4b5a8, 0xc2131d17a0, 0xc264625ce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2131d17a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131050 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af1d400, 0xc2161c4960, 0xc261250960, 0xc2612509c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 50964 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc249734dc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2551e6be0, 0x7f5d6ba4b5a8, 0xc215e85b60, 0xc23aa51f20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2551e6be0, 0x7f5d6ba4b5a8, 0xc215e85b60, 0xc23aa51f20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e85b60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 70699 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c29e840, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c29e840, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c29e7e0, 0xc226858000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24f472d58) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc212f85ca0, 0xc226858000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc212f85ca0, 0xc226858000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc212f85ca0, 0xc226858000, 0xfa000, 0xfa000, 0xc217f40000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2158caf30, 0x7f5d6b8a1528, 0xc212f85ca0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212f85ca0, 0xc217f47c80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131129 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215edbb50, 0xc21544a300, 0xc26945e660, 0xc26945e6c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 18614 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b9c2ed0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b9c2ed0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b9c2e70, 0xc2242c0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211da0448) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20bfa3918, 0xc2242c0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20bfa3918, 0xc2242c0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20bfa3918, 0xc2242c0000, 0xfa000, 0xfa000, 0xc216270000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2153b7f80, 0x7f5d6b8a1528, 0xc20bfa3918, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bfa3918, 0xc216274ae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 49338 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21510f4f0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21510f4f0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21510f490, 0xc22df42000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc252036418) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21519e5d0, 0xc22df42000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21519e5d0, 0xc22df42000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21519e5d0, 0xc22df42000, 0xfa000, 0xfa000, 0xc2139a0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20f458c90, 0x7f5d6b8a1528, 0xc21519e5d0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21519e5d0, 0xc2139a1260) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 56897 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20e598760, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20e598760, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20e598700, 0xc230df0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23d886a28) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21369f030, 0xc230df0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21369f030, 0xc230df0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21369f030, 0xc230df0000, 0xfa000, 0xfa000, 0xc220300000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214c84930, 0x7f5d6b8a1528, 0xc21369f030, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21369f030, 0xc22030ad20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131125 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215188b88, 0xc2135aaa20, 0xc26945e1e0, 0xc26945e240) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32475 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210f003e0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210f003e0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210f00380, 0xc22a6f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240d6b648) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214514510, 0xc22a6f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214514510, 0xc22a6f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214514510, 0xc22a6f0000, 0xfa000, 0xfa000, 0xc213390000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21300ef60, 0x7f5d6b8a1528, 0xc214514510, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214514510, 0xc213398f60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 25171 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219b0eb40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2661173b0, 0x7f5d6ba4b5a8, 0xc215b43c20, 0xc2150a6cc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2661173b0, 0x7f5d6ba4b5a8, 0xc215b43c20, 0xc2150a6cc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b43c20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50937 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21ff7cb40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23f2cee10, 0x7f5d6ba4b5a8, 0xc215cd3028, 0xc264aea840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23f2cee10, 0x7f5d6ba4b5a8, 0xc215cd3028, 0xc264aea840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd3028) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130841 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bb58a88, 0xc21b4c6000, 0xc238d1c780, 0xc238d1c7e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32198 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b337f00, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b337f00, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b337ea0, 0xc2284f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2167a8e68) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2152fb258, 0xc2284f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2152fb258, 0xc2284f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2152fb258, 0xc2284f0000, 0xfa000, 0xfa000, 0xc215f40000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214933260, 0x7f5d6b8a1528, 0xc2152fb258, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2152fb258, 0xc215f43500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 70741 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21410a5a0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21410a5a0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21410a540, 0xc2332f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc217f93eb8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc212e76938, 0xc2332f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc212e76938, 0xc2332f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc212e76938, 0xc2332f0000, 0xfa000, 0xfa000, 0xc21ec90000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21edfcf90, 0x7f5d6b8a1528, 0xc212e76938, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212e76938, 0xc21ec94a80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9129008 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1e6d0, 0xc2162253e0, 0xc20cfed560, 0xc20cfed5c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 69090 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b139fe0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b139fe0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b139f80, 0xc232af0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24b6a1d90) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc212f84828, 0xc232af0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc212f84828, 0xc232af0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc212f84828, 0xc232af0000, 0xfa000, 0xfa000, 0xc219f00000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc218b710b0, 0x7f5d6b8a1528, 0xc212f84828, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212f84828, 0xc219f0fda0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 12555 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc218849680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24a3a9040, 0x7f5d6ba4b5a8, 0xc2160c4058, 0xc20b27f5c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24a3a9040, 0x7f5d6ba4b5a8, 0xc2160c4058, 0xc20b27f5c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160c4058) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 12666 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc218848780, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24566ae60, 0x7f5d6ba4b5a8, 0xc216027480, 0xc20b27fb60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24566ae60, 0x7f5d6ba4b5a8, 0xc216027480, 0xc20b27fb60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216027480) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 42233 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20ba21b10, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20ba21b10, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20ba21ab0, 0xc22baf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25bb9ee80) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213ed8168, 0xc22baf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213ed8168, 0xc22baf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213ed8168, 0xc22baf0000, 0xfa000, 0xfa000, 0xc210b70000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ba41b90, 0x7f5d6b8a1528, 0xc213ed8168, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ed8168, 0xc210b7a420) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 74761 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210f01bf0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210f01bf0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210f01b90, 0xc235e6c000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc212a6af48) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214de9480, 0xc235e6c000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214de9480, 0xc235e6c000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214de9480, 0xc235e6c000, 0xfa000, 0xfa000, 0xc216140000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20dc71800, 0x7f5d6b8a1528, 0xc214de9480, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214de9480, 0xc216142a20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32195 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc217384df0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc217384df0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc217384d90, 0xc227df0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23e4c1898) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2160c5c10, 0xc227df0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2160c5c10, 0xc227df0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2160c5c10, 0xc227df0000, 0xfa000, 0xfa000, 0xc216130000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214c44d20, 0x7f5d6b8a1528, 0xc2160c5c10, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160c5c10, 0xc21613a4e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131234 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210c75030, 0xc216224cc0, 0xc21aab1320, 0xc21aab1380) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 51079 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20bbabcc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc258644410, 0x7f5d6ba4b5a8, 0xc215630840, 0xc2612522a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc258644410, 0x7f5d6ba4b5a8, 0xc215630840, 0xc2612522a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215630840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50995 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc212a14a70, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc212a14a70, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc212a14a10, 0xc22e618000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23faf8888) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214c762d0, 0xc22e618000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214c762d0, 0xc22e618000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214c762d0, 0xc22e618000, 0xfa000, 0xfa000, 0xc2134b0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ed99170, 0x7f5d6b8a1528, 0xc214c762d0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214c762d0, 0xc2134b1e60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 51048 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc22b3ea300, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc22b3ea300, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc22b3ea2a0, 0xc22f3f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2167a95e8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21437ed40, 0xc22f3f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21437ed40, 0xc22f3f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21437ed40, 0xc22f3f0000, 0xfa000, 0xfa000, 0xc20f9d0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20bdcb020, 0x7f5d6b8a1528, 0xc21437ed40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21437ed40, 0xc20f9db080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 12639 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b4f6300, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b4f6300, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b4f62a0, 0xc224e04000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc256ce4208) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20802f398, 0xc224e04000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20802f398, 0xc224e04000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20802f398, 0xc224e04000, 0xfa000, 0xfa000, 0xc215fe0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214d20fc0, 0x7f5d6b8a1528, 0xc20802f398, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20802f398, 0xc215fead20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 70201 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc262d84640, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2121539a0, 0x7f5d6ba4b5a8, 0xc211e88110, 0xc24759f8c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2121539a0, 0x7f5d6ba4b5a8, 0xc211e88110, 0xc24759f8c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211e88110) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 52682 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc22b3eadf0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc22b3eadf0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc22b3ead90, 0xc22fff0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc213b0cb68) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2139a5030, 0xc22fff0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2139a5030, 0xc22fff0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2139a5030, 0xc22fff0000, 0xfa000, 0xfa000, 0xc20b900000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20b36a600, 0x7f5d6b8a1528, 0xc2139a5030, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139a5030, 0xc20b906c00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 62711 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2118228c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25596a730, 0x7f5d6ba4b5a8, 0xc21369fe18, 0xc238d1b260) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25596a730, 0x7f5d6ba4b5a8, 0xc21369fe18, 0xc238d1b260) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21369fe18) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 51091 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23938bb80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc224991040, 0x7f5d6ba4b5a8, 0xc20bfa2198, 0xc20d0dd440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc224991040, 0x7f5d6ba4b5a8, 0xc20bfa2198, 0xc20d0dd440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bfa2198) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131028 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e72120, 0xc216180b40, 0xc21758b320, 0xc21758b380) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 12758 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc218848500, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24566ac80, 0x7f5d6ba4b5a8, 0xc20b045888, 0xc20b27f920) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24566ac80, 0x7f5d6ba4b5a8, 0xc20b045888, 0xc20b27f920) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b045888) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9129947 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b72c60, 0xc21452e360, 0xc265d91860, 0xc265d918c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 23428 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20af1aa70, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af1aa70, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af1aa10, 0xc220d5c000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2539276c8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20bb59a40, 0xc220d5c000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20bb59a40, 0xc220d5c000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20bb59a40, 0xc220d5c000, 0xfa000, 0xfa000, 0xc214350000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e5ba8a0, 0x7f5d6b8a1528, 0xc20bb59a40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bb59a40, 0xc214350a80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 31162 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21bdaba40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24a3a9360, 0x7f5d6ba4b5a8, 0xc215e413f8, 0xc242581e60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24a3a9360, 0x7f5d6ba4b5a8, 0xc215e413f8, 0xc242581e60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e413f8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 17107 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc245d8e280, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d40d8b0, 0x7f5d6ba4b5a8, 0xc2080c6a00, 0xc26945e780) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d40d8b0, 0x7f5d6ba4b5a8, 0xc2080c6a00, 0xc26945e780) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c6a00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 23007 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23b198f00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc264376a00, 0x7f5d6ba4b5a8, 0xc215ffe528, 0xc258bceba0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc264376a00, 0x7f5d6ba4b5a8, 0xc215ffe528, 0xc258bceba0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215ffe528) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 27070 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bcea4c0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bcea4c0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bcea460, 0xc2228b4000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211d65a68) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc216266aa8, 0xc2228b4000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc216266aa8, 0xc2228b4000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc216266aa8, 0xc2228b4000, 0xfa000, 0xfa000, 0xc213a80000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21639d320, 0x7f5d6b8a1528, 0xc216266aa8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216266aa8, 0xc213a87140) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 20201 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b138b50, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b138b50, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b138af0, 0xc223018000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24d673688) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2080c7660, 0xc223018000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2080c7660, 0xc223018000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2080c7660, 0xc223018000, 0xfa000, 0xfa000, 0xc215dd0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214bece40, 0x7f5d6b8a1528, 0xc2080c7660, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c7660, 0xc215ddf680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32309 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc213547100, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc213547100, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2135470a0, 0xc2296f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc210321ee0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214a17048, 0xc2296f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214a17048, 0xc2296f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214a17048, 0xc2296f0000, 0xfa000, 0xfa000, 0xc214660000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214180f30, 0x7f5d6b8a1528, 0xc214a17048, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a17048, 0xc21466f740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131138 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210b73e50, 0xc21f766cc0, 0xc23cc70000, 0xc23cc70060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 11274 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c29e530, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c29e530, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c29e4d0, 0xc2257ca000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211da1a48) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21632a5f0, 0xc2257ca000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21632a5f0, 0xc2257ca000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21632a5f0, 0xc2257ca000, 0xfa000, 0xfa000, 0xc2162b0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214f20870, 0x7f5d6b8a1528, 0xc21632a5f0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21632a5f0, 0xc2162bdda0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 50978 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23aabcf00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc262d5c640, 0x7f5d6ba4b5a8, 0xc2162676c8, 0xc23cc70900) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc262d5c640, 0x7f5d6ba4b5a8, 0xc2162676c8, 0xc23cc70900) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2162676c8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 86048 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2328eca00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc256203950, 0x7f5d6ba4b5a8, 0xc2140a6918, 0xc20cfed320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc256203950, 0x7f5d6ba4b5a8, 0xc2140a6918, 0xc20cfed320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2140a6918) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32232 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc217385100, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc217385100, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2173850a0, 0xc2285f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25294c6a0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215a05e78, 0xc2285f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215a05e78, 0xc2285f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215a05e78, 0xc2285f0000, 0xfa000, 0xfa000, 0xc215ed0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214933f80, 0x7f5d6b8a1528, 0xc215a05e78, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a05e78, 0xc215ed45a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131222 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e402a8, 0xc2162154a0, 0xc2611e68a0, 0xc2611e6900) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32324 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20eb43b80, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20eb43b80, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20eb43b20, 0xc229ff0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211d52f68) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21463a720, 0xc229ff0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21463a720, 0xc229ff0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21463a720, 0xc229ff0000, 0xfa000, 0xfa000, 0xc213680000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2134dd050, 0x7f5d6b8a1528, 0xc21463a720, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21463a720, 0xc21368c240) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 30476 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c29eed0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c29eed0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c29ee70, 0xc2275f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc213d1c820) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e40af0, 0xc2275f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e40af0, 0xc2275f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e40af0, 0xc2275f0000, 0xfa000, 0xfa000, 0xc21d3f0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214f446c0, 0x7f5d6b8a1528, 0xc215e40af0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e40af0, 0xc21d3fb8c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 52709 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc212a15720, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc212a15720, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc212a156c0, 0xc2301f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2557b3e98) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2143207c8, 0xc2301f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2143207c8, 0xc2301f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2143207c8, 0xc2301f0000, 0xfa000, 0xfa000, 0xc214040000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e14bc50, 0x7f5d6b8a1528, 0xc2143207c8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2143207c8, 0xc214042f60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131013 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1e8c0, 0xc21362ee40, 0xc261252180, 0xc2612521e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32089 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc217385c60, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc217385c60, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc217385c00, 0xc2289f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2167a9178) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215edb418, 0xc2289f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215edb418, 0xc2289f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215edb418, 0xc2289f0000, 0xfa000, 0xfa000, 0xc215440000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2147cb140, 0x7f5d6b8a1528, 0xc215edb418, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215edb418, 0xc21544baa0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32511 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2170d2d80, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2170d2d80, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2170d2d20, 0xc22a8f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23d8866c8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e32c70, 0xc22a8f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e32c70, 0xc22a8f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e32c70, 0xc22a8f0000, 0xfa000, 0xfa000, 0xc210d50000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e7cde30, 0x7f5d6b8a1528, 0xc215e32c70, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e32c70, 0xc210d5ed20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 26705 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2156598e0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2156598e0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc215659880, 0xc221cfc000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21fede288) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215188b88, 0xc221cfc000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215188b88, 0xc221cfc000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215188b88, 0xc221cfc000, 0xfa000, 0xfa000, 0xc2135a0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20baf0870, 0x7f5d6b8a1528, 0xc215188b88, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215188b88, 0xc2135aaa20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 51052 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc22b3ea4c0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc22b3ea4c0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc22b3ea460, 0xc22f6f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23e4c1ad8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2139ce670, 0xc22f6f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2139ce670, 0xc22f6f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2139ce670, 0xc22f6f0000, 0xfa000, 0xfa000, 0xc20ed60000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20dd37470, 0x7f5d6b8a1528, 0xc2139ce670, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139ce670, 0xc20ed69c20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 50940 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21e709a40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc258644dc0, 0x7f5d6ba4b5a8, 0xc21519e5d0, 0xc261252cc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc258644dc0, 0x7f5d6ba4b5a8, 0xc21519e5d0, 0xc261252cc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21519e5d0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 70575 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21de34530, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21de34530, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21de344d0, 0xc233ff0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bcc76f8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc211057e28, 0xc233ff0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc211057e28, 0xc233ff0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc211057e28, 0xc233ff0000, 0xfa000, 0xfa000, 0xc216020000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc213b7b8f0, 0x7f5d6b8a1528, 0xc211057e28, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211057e28, 0xc2160212c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9130154 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212bbfb30, 0xc21a14c2a0, 0xc238f477a0, 0xc238f47800) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 51003 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc22b3ea290, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc22b3ea290, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc22b3ea230, 0xc22f2f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc219bb9e28) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21437e2e8, 0xc22f2f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21437e2e8, 0xc22f2f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21437e2e8, 0xc22f2f0000, 0xfa000, 0xfa000, 0xc20fdb0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20cb7fc50, 0x7f5d6b8a1528, 0xc21437e2e8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21437e2e8, 0xc20fdb3320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 31167 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c29f1e0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c29f1e0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c29f180, 0xc227cf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc244456208) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e72120, 0xc227cf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e72120, 0xc227cf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e72120, 0xc227cf0000, 0xfa000, 0xfa000, 0xc216180000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214c44480, 0x7f5d6b8a1528, 0xc215e72120, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e72120, 0xc216180b40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 50939 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc216f2c3c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc26480c320, 0x7f5d6ba4b5a8, 0xc215cd3e40, 0xc2612503c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc26480c320, 0x7f5d6ba4b5a8, 0xc215cd3e40, 0xc2612503c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd3e40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9129001 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ed9a08, 0xc20dbdc780, 0xc20cfecd80, 0xc20cfecde0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 78045 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a045410, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a045410, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a0453b0, 0xc23663c000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc256ce4588) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215a2a588, 0xc23663c000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215a2a588, 0xc23663c000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215a2a588, 0xc23663c000, 0xfa000, 0xfa000, 0xc20e510000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2153af380, 0x7f5d6b8a1528, 0xc215a2a588, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2a588, 0xc20e516180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32229 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2421d97c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc251853590, 0x7f5d6ba4b5a8, 0xc2152fa770, 0xc2150a64e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc251853590, 0x7f5d6ba4b5a8, 0xc2152fa770, 0xc2150a64e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2152fa770) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50865 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219b0f040, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc266117770, 0x7f5d6ba4b5a8, 0xc2147a1cc8, 0xc2150a7140) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc266117770, 0x7f5d6ba4b5a8, 0xc2147a1cc8, 0xc2150a7140) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2147a1cc8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131235 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210e641a8, 0xc20bb91440, 0xc21aab1440, 0xc21aab14a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 49344 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21966c680, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21966c680, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21966c620, 0xc22ebf4000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211d529d8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2149df558, 0xc22ebf4000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2149df558, 0xc22ebf4000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2149df558, 0xc22ebf4000, 0xfa000, 0xfa000, 0xc211e90000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e68ef60, 0x7f5d6b8a1528, 0xc2149df558, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2149df558, 0xc211e9cde0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 69091 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc24b9dd400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24a7a64b0, 0x7f5d6ba4b5a8, 0xc2132eb6d0, 0xc2187fd5c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24a7a64b0, 0x7f5d6ba4b5a8, 0xc2132eb6d0, 0xc2187fd5c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2132eb6d0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32474 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc216f2cb40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc26480c960, 0x7f5d6ba4b5a8, 0xc214514510, 0xc261250a80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc26480c960, 0x7f5d6ba4b5a8, 0xc214514510, 0xc261250a80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214514510) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 74851 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc218848a00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24566b040, 0x7f5d6ba4b5a8, 0xc2145e0078, 0xc20b27fda0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24566b040, 0x7f5d6ba4b5a8, 0xc2145e0078, 0xc20b27fda0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2145e0078) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131047 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21fac1640, 0xc21487ef00, 0xc261250600, 0xc261250660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 23591 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc245500500, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24081a050, 0x7f5d6ba4b5a8, 0xc215cd3de8, 0xc21aab1b00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24081a050, 0x7f5d6ba4b5a8, 0xc215cd3de8, 0xc21aab1b00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd3de8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9086446 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e85d58, 0xc214c4d500, 0xc260845680, 0xc2608456e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 49301 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210e95170, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210e95170, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210e95110, 0xc22d196000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc212a6ac38) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e402a8, 0xc22d196000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e402a8, 0xc22d196000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e402a8, 0xc22d196000, 0xfa000, 0xfa000, 0xc216210000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214eaf020, 0x7f5d6b8a1528, 0xc215e402a8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e402a8, 0xc2162154a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32305 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2132481b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2132481b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc213248150, 0xc22a5f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc218698ca8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21437eb68, 0xc22a5f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21437eb68, 0xc22a5f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21437eb68, 0xc22a5f0000, 0xfa000, 0xfa000, 0xc213420000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2131e6630, 0x7f5d6b8a1528, 0xc21437eb68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21437eb68, 0xc213421860) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131061 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2149dee48, 0xc212478fc0, 0xc2150a6720, 0xc2150a6780) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 50876 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc213547f00, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc213547f00, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc213547ea0, 0xc22e136000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24f4733a8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215188738, 0xc22e136000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215188738, 0xc22e136000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215188738, 0xc22e136000, 0xfa000, 0xfa000, 0xc2136e0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e575380, 0x7f5d6b8a1528, 0xc215188738, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215188738, 0xc2136e0480) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 51001 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210f00d10, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210f00d10, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210f00cb0, 0xc22f1f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2520362b8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2145e18a8, 0xc22f1f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2145e18a8, 0xc22f1f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2145e18a8, 0xc22f1f0000, 0xfa000, 0xfa000, 0xc20fea0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20cb7ee40, 0x7f5d6b8a1528, 0xc2145e18a8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2145e18a8, 0xc20fead740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32271 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21bdabe00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24a3a9630, 0x7f5d6ba4b5a8, 0xc214617ed8, 0xc26243e1e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24a3a9630, 0x7f5d6ba4b5a8, 0xc214617ed8, 0xc26243e1e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214617ed8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 28588 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21510e220, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21510e220, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21510e1c0, 0xc221fea000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240b0ac58) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214f1e6a0, 0xc221fea000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214f1e6a0, 0xc221fea000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214f1e6a0, 0xc221fea000, 0xfa000, 0xfa000, 0xc213280000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc211bebd10, 0x7f5d6b8a1528, 0xc214f1e6a0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1e6a0, 0xc213288ae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 92502 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a387480, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a387480, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a387420, 0xc2383f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240d6b300) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213ffbee0, 0xc2383f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213ffbee0, 0xc2383f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213ffbee0, 0xc2383f0000, 0xfa000, 0xfa000, 0xc20bd70000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21fd60240, 0x7f5d6b8a1528, 0xc213ffbee0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffbee0, 0xc20bd75c80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 28599 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b4f6220, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b4f6220, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b4f61c0, 0xc2225c6000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2167a9068) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc216170148, 0xc2225c6000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc216170148, 0xc2225c6000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc216170148, 0xc2225c6000, 0xfa000, 0xfa000, 0xc215d40000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20bdacd80, 0x7f5d6b8a1528, 0xc216170148, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216170148, 0xc215d41080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 21872 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f415e00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc251853a40, 0x7f5d6ba4b5a8, 0xc215d0c980, 0xc24ec8bf80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc251853a40, 0x7f5d6ba4b5a8, 0xc215d0c980, 0xc24ec8bf80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215d0c980) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131172 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b42a50, 0xc21c80d560, 0xc242581c20, 0xc242581c80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 91704 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc25d682280, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2590310e0, 0x7f5d6ba4b5a8, 0xc213ffae78, 0xc2633762a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2590310e0, 0x7f5d6ba4b5a8, 0xc213ffae78, 0xc2633762a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffae78) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32260 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc213546df0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc213546df0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc213546d90, 0xc228ff0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc252cfa698) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2160fe1a8, 0xc228ff0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2160fe1a8, 0xc228ff0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2160fe1a8, 0xc228ff0000, 0xfa000, 0xfa000, 0xc214c60000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214438f00, 0x7f5d6b8a1528, 0xc2160fe1a8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160fe1a8, 0xc214c604e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72539 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2170d2680, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2170d2680, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2170d2620, 0xc2351ba000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23d887248) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21622fc00, 0xc2351ba000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21622fc00, 0xc2351ba000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21622fc00, 0xc2351ba000, 0xfa000, 0xfa000, 0xc20bb80000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc22c1ea750, 0x7f5d6b8a1528, 0xc21622fc00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622fc00, 0xc20bb873e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 50860 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc249735180, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25b4710e0, 0x7f5d6ba4b5a8, 0xc21533d600, 0xc23b9d22a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25b4710e0, 0x7f5d6ba4b5a8, 0xc21533d600, 0xc23b9d22a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21533d600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32292 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2328ec8c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc256203860, 0x7f5d6ba4b5a8, 0xc214834b30, 0xc20cfed200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc256203860, 0x7f5d6ba4b5a8, 0xc214834b30, 0xc20cfed200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214834b30) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131014 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215630840, 0xc20f00baa0, 0xc2612522a0, 0xc261252300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 23009 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21e708a00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2586449b0, 0x7f5d6ba4b5a8, 0xc215ffee78, 0xc261252720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2586449b0, 0x7f5d6ba4b5a8, 0xc215ffee78, 0xc261252720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215ffee78) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50986 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21510f480, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21510f480, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21510f420, 0xc22de48000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240d6aaa0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215cd3e40, 0xc22de48000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215cd3e40, 0xc22de48000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215cd3e40, 0xc22de48000, 0xfa000, 0xfa000, 0xc213b20000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20f66e1e0, 0x7f5d6b8a1528, 0xc215cd3e40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd3e40, 0xc213b20660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131098 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2138c10d0, 0xc213b63980, 0xc24759e3c0, 0xc24759e420) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129942 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214c76118, 0xc213641c20, 0xc265d912c0, 0xc265d91320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 12697 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21bdabcc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24a3a9540, 0x7f5d6ba4b5a8, 0xc215e32168, 0xc26243e0c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24a3a9540, 0x7f5d6ba4b5a8, 0xc215e32168, 0xc26243e0c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e32168) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9086315 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212e76938, 0xc21ec94a80, 0xc2608469c0, 0xc260846a20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 39074 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23c815cc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc212152f50, 0x7f5d6ba4b5a8, 0xc21419a1d8, 0xc21758b200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc212152f50, 0x7f5d6ba4b5a8, 0xc21419a1d8, 0xc21758b200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21419a1d8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130862 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213b04b08, 0xc219e19320, 0xc20b281f80, 0xc2611e6000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 51076 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21e708f00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc258644cd0, 0x7f5d6ba4b5a8, 0xc2145e18a8, 0xc261252ba0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc258644cd0, 0x7f5d6ba4b5a8, 0xc2145e18a8, 0xc261252ba0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2145e18a8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9129888 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4fb10, 0xc21545cb40, 0xc263f37740, 0xc263f377a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 50998 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bceba30, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bceba30, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bceb9d0, 0xc22ea00000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc212a6a9d8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214a16db0, 0xc22ea00000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214a16db0, 0xc22ea00000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214a16db0, 0xc22ea00000, 0xfa000, 0xfa000, 0xc212f90000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e5e0b40, 0x7f5d6b8a1528, 0xc214a16db0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a16db0, 0xc212f95680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32210 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2188488c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24566af50, 0x7f5d6ba4b5a8, 0xc215e72b18, 0xc20b27fc80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24566af50, 0x7f5d6ba4b5a8, 0xc215e72b18, 0xc20b27fc80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e72b18) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32300 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b4f7950, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b4f7950, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b4f78f0, 0xc22a0f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc219bb9b78) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2147a19f0, 0xc22a0f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2147a19f0, 0xc22a0f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2147a19f0, 0xc22a0f0000, 0xfa000, 0xfa000, 0xc213680000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2134b5800, 0x7f5d6b8a1528, 0xc2147a19f0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2147a19f0, 0xc213683bc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 55985 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23d4168c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24566a640, 0x7f5d6ba4b5a8, 0xc2136fcfa0, 0xc21c04cc00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24566a640, 0x7f5d6ba4b5a8, 0xc2136fcfa0, 0xc21c04cc00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2136fcfa0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 11673 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21966c4c0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21966c4c0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21966c460, 0xc220538000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc213d1c678) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc216266310, 0xc220538000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc216266310, 0xc220538000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc216266310, 0xc220538000, 0xfa000, 0xfa000, 0xc218a90000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214d1a960, 0x7f5d6b8a1528, 0xc216266310, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216266310, 0xc218a91920) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 62780 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2328ec780, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc256203770, 0x7f5d6ba4b5a8, 0xc213656840, 0xc20cfed0e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc256203770, 0x7f5d6ba4b5a8, 0xc213656840, 0xc20cfed0e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213656840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 27060 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21510e3e0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21510e3e0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21510e380, 0xc2220e4000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc219bb9f88) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214f1f530, 0xc2220e4000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214f1f530, 0xc2220e4000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214f1f530, 0xc2220e4000, 0xfa000, 0xfa000, 0xc213620000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2195b76e0, 0x7f5d6b8a1528, 0xc214f1f530, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1f530, 0xc21362ede0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 42298 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b26e680, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b26e680, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b26e620, 0xc22bbf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24fbac278) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213aad650, 0xc22bbf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213aad650, 0xc22bbf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213aad650, 0xc22bbf0000, 0xfa000, 0xfa000, 0xc20f2c0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21edea480, 0x7f5d6b8a1528, 0xc213aad650, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213aad650, 0xc20f2c57a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32187 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c29f5d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c29f5d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c29f570, 0xc2280f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211d64e38) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e72b18, 0xc2280f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e72b18, 0xc2280f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e72b18, 0xc2280f0000, 0xfa000, 0xfa000, 0xc215fd0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214a70120, 0x7f5d6b8a1528, 0xc215e72b18, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e72b18, 0xc215fd2fc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 74814 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2186508c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc261812640, 0x7f5d6ba4b5a8, 0xc214a17c00, 0xc25fa2b0e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc261812640, 0x7f5d6ba4b5a8, 0xc214a17c00, 0xc25fa2b0e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a17c00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32319 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219f30f00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc258644910, 0x7f5d6ba4b5a8, 0xc2145f5cc0, 0xc238d1d080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc258644910, 0x7f5d6ba4b5a8, 0xc2145f5cc0, 0xc238d1d080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2145f5cc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 74813 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21be37870, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21be37870, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21be37810, 0xc23615a000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23e4c1ec8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214a75738, 0xc23615a000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214a75738, 0xc23615a000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214a75738, 0xc23615a000, 0xfa000, 0xfa000, 0xc215e90000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc215fde120, 0x7f5d6b8a1528, 0xc214a75738, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a75738, 0xc215e92c00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72534 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2170d2060, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2170d2060, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2170d2000, 0xc234ae4000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24f473268) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc212bbfeb0, 0xc234ae4000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc212bbfeb0, 0xc234ae4000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc212bbfeb0, 0xc234ae4000, 0xfa000, 0xfa000, 0xc210b70000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc213384e70, 0x7f5d6b8a1528, 0xc212bbfeb0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212bbfeb0, 0xc210b78000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 70577 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc219c07250, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc219c07250, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc219c071f0, 0xc2341f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bcc7d28) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2109374a0, 0xc2341f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2109374a0, 0xc2341f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2109374a0, 0xc2341f0000, 0xfa000, 0xfa000, 0xc216020000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21b5e6ea0, 0x7f5d6b8a1528, 0xc2109374a0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2109374a0, 0xc216021320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 64191 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc212f3b100, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc212f3b100, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc212f3b0a0, 0xc2326f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc219bb9c68) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21306a2c0, 0xc2326f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21306a2c0, 0xc2326f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21306a2c0, 0xc2326f0000, 0xfa000, 0xfa000, 0xc21afe0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc210b481b0, 0x7f5d6b8a1528, 0xc21306a2c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21306a2c0, 0xc21afe90e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 79345 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21c4b4c80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23f2ce5f0, 0x7f5d6ba4b5a8, 0xc2080c71a8, 0xc238d1cae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23f2ce5f0, 0x7f5d6ba4b5a8, 0xc2080c71a8, 0xc238d1cae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c71a8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50871 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219b0ef00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc266117680, 0x7f5d6ba4b5a8, 0xc216170f70, 0xc2150a7020) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc266117680, 0x7f5d6ba4b5a8, 0xc216170f70, 0xc2150a7020) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216170f70) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9129873 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21632a3f0, 0xc215ddec60, 0xc265d8fec0, 0xc265d8ff20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129297 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139cf280, 0xc20cb31140, 0xc24ec8bd40, 0xc24ec8bda0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 37442 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20ec06d10, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20ec06d10, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20ec06cb0, 0xc22acf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ef50170) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2142719f8, 0xc22acf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2142719f8, 0xc22acf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2142719f8, 0xc22acf0000, 0xfa000, 0xfa000, 0xc20ba60000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20b955650, 0x7f5d6b8a1528, 0xc2142719f8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2142719f8, 0xc20ba6fb00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 45550 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20ce6a4c0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20ce6a4c0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20ce6a460, 0xc22c9c6000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24f472270) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215ffe0d8, 0xc22c9c6000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215ffe0d8, 0xc22c9c6000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215ffe0d8, 0xc22c9c6000, 0xfa000, 0xfa000, 0xc216190000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20c196d50, 0x7f5d6b8a1528, 0xc215ffe0d8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215ffe0d8, 0xc21619a3c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 50997 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2328ecf00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc256203d10, 0x7f5d6ba4b5a8, 0xc214b73c98, 0xc20cfed7a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc256203d10, 0x7f5d6ba4b5a8, 0xc214b73c98, 0xc20cfed7a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b73c98) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 47629 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b9c3720, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b9c3720, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b9c36c0, 0xc22d290000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc219bb9c10) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2160ffa38, 0xc22d290000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2160ffa38, 0xc22d290000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2160ffa38, 0xc22d290000, 0xfa000, 0xfa000, 0xc216190000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc213b6f3e0, 0x7f5d6b8a1528, 0xc2160ffa38, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160ffa38, 0xc21619bc80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9087904 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213aac9a8, 0xc2118e8720, 0xc23aca5020, 0xc23aca5080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 55865 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d25c640, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2551e7680, 0x7f5d6ba4b5a8, 0xc21622eb98, 0xc240bba4e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2551e7680, 0x7f5d6ba4b5a8, 0xc21622eb98, 0xc240bba4e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622eb98) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 72353 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc244c48b40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2121536d0, 0x7f5d6ba4b5a8, 0xc215e84248, 0xc21758bb00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2121536d0, 0x7f5d6ba4b5a8, 0xc215e84248, 0xc21758bb00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e84248) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 29415 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bf9a290, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bf9a290, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bf9a230, 0xc2227ba000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211d64008) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21519f7e8, 0xc2227ba000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21519f7e8, 0xc2227ba000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21519f7e8, 0xc2227ba000, 0xfa000, 0xfa000, 0xc21b4c0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21639d290, 0x7f5d6b8a1528, 0xc21519f7e8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21519f7e8, 0xc21b4c69c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 67458 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20bbabe00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23f2ce690, 0x7f5d6ba4b5a8, 0xc2131d1438, 0xc2612523c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23f2ce690, 0x7f5d6ba4b5a8, 0xc2131d1438, 0xc2612523c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2131d1438) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9088102 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21367e9c8, 0xc2160210e0, 0xc25fa6b860, 0xc25fa6b8c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32261 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20bbab900, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23fa334f0, 0x7f5d6ba4b5a8, 0xc214de8478, 0xc25fa6b980) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23fa334f0, 0x7f5d6ba4b5a8, 0xc214de8478, 0xc25fa6b980) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214de8478) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131016 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b045f58, 0xc213a86060, 0xc2612524e0, 0xc261252540) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 69166 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a17f100, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a17f100, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a17f0a0, 0xc232ff0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24e63d428) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc212ded408, 0xc232ff0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc212ded408, 0xc232ff0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc212ded408, 0xc232ff0000, 0xfa000, 0xfa000, 0xc213830000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21cf14750, 0x7f5d6b8a1528, 0xc212ded408, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212ded408, 0xc2138324e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 37419 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc244c48640, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc212153310, 0x7f5d6ba4b5a8, 0xc21fac0950, 0xc21758b680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc212153310, 0x7f5d6ba4b5a8, 0xc21fac0950, 0xc21758b680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21fac0950) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 15269 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20af1a140, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af1a140, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af1a0e0, 0xc2121d4000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bcc6a50) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20bb58a88, 0xc2121d4000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20bb58a88, 0xc2121d4000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20bb58a88, 0xc2121d4000, 0xfa000, 0xfa000, 0xc21b4c0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21366f410, 0x7f5d6b8a1528, 0xc20bb58a88, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bb58a88, 0xc21b4c6000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 47009 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21c4b4f00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d5f2320, 0x7f5d6ba4b5a8, 0xc213efb4b0, 0xc238d1cd20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d5f2320, 0x7f5d6ba4b5a8, 0xc213efb4b0, 0xc238d1cd20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213efb4b0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32211 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219b0f400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc266117a40, 0x7f5d6ba4b5a8, 0xc215e73288, 0xc2150a74a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc266117a40, 0x7f5d6ba4b5a8, 0xc215e73288, 0xc2150a74a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e73288) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 31166 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23c815e00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc266117040, 0x7f5d6ba4b5a8, 0xc215e72120, 0xc21758b320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc266117040, 0x7f5d6ba4b5a8, 0xc215e72120, 0xc21758b320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e72120) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130801 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e73b10, 0xc214362f00, 0xc264625e00, 0xc264625e60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130153 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139cec00, 0xc219e91e00, 0xc238f47680, 0xc238f476e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 13721 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bb1fcd0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bb1fcd0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bb1fc70, 0xc224ff8000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25294cf98) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21621d680, 0xc224ff8000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21621d680, 0xc224ff8000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21621d680, 0xc224ff8000, 0xfa000, 0xfa000, 0xc215e20000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21480e810, 0x7f5d6b8a1528, 0xc21621d680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21621d680, 0xc215e2e660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 40677 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20af45d40, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af45d40, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af45ce0, 0xc22b3f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bcc73e8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215a2a658, 0xc22b3f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215a2a658, 0xc22b3f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215a2a658, 0xc22b3f0000, 0xfa000, 0xfa000, 0xc2118e0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2130c2d50, 0x7f5d6b8a1528, 0xc215a2a658, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2a658, 0xc2118e8600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131218 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21632a260, 0xc218a915c0, 0xc2611e6420, 0xc2611e6480) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130793 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210df0220, 0xc2161c0240, 0xc264625500, 0xc264625560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9088045 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212f84828, 0xc219f0fda0, 0xc23aca7080, 0xc23aca70e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 74760 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b1381b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b1381b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b138150, 0xc235a84000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc218698538) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215cd2d40, 0xc235a84000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215cd2d40, 0xc235a84000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215cd2d40, 0xc235a84000, 0xfa000, 0xfa000, 0xc2161b0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2162024e0, 0x7f5d6b8a1528, 0xc215cd2d40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd2d40, 0xc2161b4360) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 73777 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc213547020, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc213547020, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc213546fc0, 0xc235796000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21e7cd808) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc210df0220, 0xc235796000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc210df0220, 0xc235796000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc210df0220, 0xc235796000, 0xfa000, 0xfa000, 0xc2161c0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20da522a0, 0x7f5d6b8a1528, 0xc210df0220, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210df0220, 0xc2161c0240) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131042 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b0456c8, 0xc2136b72c0, 0xc261250060, 0xc2612500c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 72573 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2170d26f0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2170d26f0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2170d2690, 0xc2352b4000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc212a6a5f8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc216026888, 0xc2352b4000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc216026888, 0xc2352b4000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc216026888, 0xc2352b4000, 0xfa000, 0xfa000, 0xc211570000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2158ef4d0, 0x7f5d6b8a1528, 0xc216026888, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216026888, 0xc211579200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 91705 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21410ad80, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21410ad80, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21410ad20, 0xc2381f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21606f128) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214349d20, 0xc2381f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214349d20, 0xc2381f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214349d20, 0xc2381f0000, 0xfa000, 0xfa000, 0xc21db70000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20cf65bf0, 0x7f5d6b8a1528, 0xc214349d20, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214349d20, 0xc21db7f620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131058 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211a49658, 0xc21f766900, 0xc2150a63c0, 0xc2150a6420) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32175 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d667e00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2628a2370, 0x7f5d6ba4b5a8, 0xc215edaa68, 0xc24759e060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2628a2370, 0x7f5d6ba4b5a8, 0xc215edaa68, 0xc24759e060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215edaa68) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131110 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622f160, 0xc216142e40, 0xc258bce2a0, 0xc258bce300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131035 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e84248, 0xc218c52ea0, 0xc21758bb00, 0xc21758bb60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131029 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21369e0d8, 0xc219e0ca80, 0xc21758b440, 0xc21758b4a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32283 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219b0f2c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc266117950, 0x7f5d6ba4b5a8, 0xc214a75d98, 0xc2150a7380) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc266117950, 0x7f5d6ba4b5a8, 0xc214a75d98, 0xc2150a7380) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a75d98) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32256 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc218650280, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2618120f0, 0x7f5d6ba4b5a8, 0xc215edbf68, 0xc25fa2ab40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2618120f0, 0x7f5d6ba4b5a8, 0xc215edbf68, 0xc25fa2ab40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215edbf68) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9088100 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211f10b90, 0xc21ec918c0, 0xc25fa6b620, 0xc25fa6b680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 52710 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23aabdb80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc262d5c370, 0x7f5d6ba4b5a8, 0xc2142efc58, 0xc23cc705a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc262d5c370, 0x7f5d6ba4b5a8, 0xc2142efc58, 0xc23cc705a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2142efc58) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 59670 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241803e00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23f2cea50, 0x7f5d6ba4b5a8, 0xc21ff38b08, 0xc264aea3c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23f2cea50, 0x7f5d6ba4b5a8, 0xc21ff38b08, 0xc264aea3c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff38b08) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50980 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21510f3a0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21510f3a0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21510f340, 0xc22da60000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24d673db8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215cd3028, 0xc22da60000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215cd3028, 0xc22da60000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215cd3028, 0xc22da60000, 0xfa000, 0xfa000, 0xc214020000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21366f350, 0x7f5d6b8a1528, 0xc215cd3028, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd3028, 0xc21402e3c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 49242 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d25ca00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc264ccfc20, 0x7f5d6ba4b5a8, 0xc2160ffa38, 0xc23b94de00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc264ccfc20, 0x7f5d6ba4b5a8, 0xc2160ffa38, 0xc23b94de00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160ffa38) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 16622 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d25c3c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc212153130, 0x7f5d6ba4b5a8, 0xc2080c6000, 0xc23b94dce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc212153130, 0x7f5d6ba4b5a8, 0xc2080c6000, 0xc23b94dce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c6000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 20075 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c2b08b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c2b08b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c2b0850, 0xc222e24000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24f472a88) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21632b828, 0xc222e24000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21632b828, 0xc222e24000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21632b828, 0xc222e24000, 0xfa000, 0xfa000, 0xc215490000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2144382d0, 0x7f5d6b8a1528, 0xc21632b828, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21632b828, 0xc215498000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9130958 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2142719f8, 0xc20ba6fb00, 0xc238d1b020, 0xc238d1b080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130798 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20802fc88, 0xc215fea7e0, 0xc264625aa0, 0xc264625b00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32275 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c29fa30, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c29fa30, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c29f9d0, 0xc2290f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24e412988) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e85d58, 0xc2290f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e85d58, 0xc2290f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e85d58, 0xc2290f0000, 0xfa000, 0xfa000, 0xc214c40000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21438b1a0, 0x7f5d6b8a1528, 0xc215e85d58, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e85d58, 0xc214c4d500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72466 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20e599e20, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20e599e20, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20e599dc0, 0xc2336f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc255c96b08) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc211e88110, 0xc2336f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc211e88110, 0xc2336f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc211e88110, 0xc2336f0000, 0xfa000, 0xfa000, 0xc21f760000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20fe6baa0, 0x7f5d6b8a1528, 0xc211e88110, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211e88110, 0xc21f7665a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 69034 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c29e140, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c29e140, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c29e0e0, 0xc2349ea000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240b0ae58) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc212f85368, 0xc2349ea000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc212f85368, 0xc2349ea000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc212f85368, 0xc2349ea000, 0xfa000, 0xfa000, 0xc20c350000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20c282d80, 0x7f5d6b8a1528, 0xc212f85368, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212f85368, 0xc20c351440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 23525 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc249735540, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25b471d10, 0x7f5d6ba4b5a8, 0xc21ff38558, 0xc23b9d2600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25b471d10, 0x7f5d6ba4b5a8, 0xc21ff38558, 0xc23b9d2600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff38558) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9086206 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ed8168, 0xc210b7a420, 0xc20d0db440, 0xc20d0db4a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 29414 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241065180, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2684838b0, 0x7f5d6ba4b5a8, 0xc2162662c8, 0xc21aab1200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2684838b0, 0x7f5d6ba4b5a8, 0xc2162662c8, 0xc21aab1200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2162662c8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131030 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2152fb258, 0xc215f43500, 0xc21758b560, 0xc21758b5c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 72462 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc244c48f00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2121537c0, 0x7f5d6ba4b5a8, 0xc210df0838, 0xc21758bc20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2121537c0, 0x7f5d6ba4b5a8, 0xc210df0838, 0xc21758bc20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210df0838) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32513 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2328ecdc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc256203c20, 0x7f5d6ba4b5a8, 0xc21621d058, 0xc20cfed680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc256203c20, 0x7f5d6ba4b5a8, 0xc21621d058, 0xc20cfed680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21621d058) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 21804 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219b0e280, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265651e00, 0x7f5d6ba4b5a8, 0xc20bb59a40, 0xc2150a6a80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265651e00, 0x7f5d6ba4b5a8, 0xc20bb59a40, 0xc2150a6a80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bb59a40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131191 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b73168, 0xc2133e2f00, 0xc240bba600, 0xc240bba660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 28303 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b93dbf0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b93dbf0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b93db90, 0xc2221de000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ac8b848) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214c76a30, 0xc2221de000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214c76a30, 0xc2221de000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214c76a30, 0xc2221de000, 0xfa000, 0xfa000, 0xc2132f0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2177ec0f0, 0x7f5d6b8a1528, 0xc214c76a30, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214c76a30, 0xc2132fe360) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 11671 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219f31540, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d5f27d0, 0x7f5d6ba4b5a8, 0xc21632a5f0, 0xc238d1d2c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d5f27d0, 0x7f5d6ba4b5a8, 0xc21632a5f0, 0xc238d1d2c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21632a5f0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9088103 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214de8478, 0xc214bdcf60, 0xc25fa6b980, 0xc25fa6b9e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 23291 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21e709cc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc258644fa0, 0x7f5d6ba4b5a8, 0xc215631338, 0xc261252f00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc258644fa0, 0x7f5d6ba4b5a8, 0xc215631338, 0xc261252f00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215631338) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131093 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1ed18, 0xc216198cc0, 0xc23e1f1e00, 0xc23e1f1e60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 63098 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21ff7cc80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23f2cef00, 0x7f5d6ba4b5a8, 0xc21406b1a8, 0xc264aea960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23f2cef00, 0x7f5d6ba4b5a8, 0xc21406b1a8, 0xc264aea960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21406b1a8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 16885 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20ce6a530, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20ce6a530, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20ce6a4d0, 0xc223dde000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc213d1c418) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e842a0, 0xc223dde000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e842a0, 0xc223dde000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e842a0, 0xc223dde000, 0xfa000, 0xfa000, 0xc214ad0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21486df20, 0x7f5d6b8a1528, 0xc215e842a0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e842a0, 0xc214ad5bc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72610 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d25d400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2551e7a40, 0x7f5d6ba4b5a8, 0xc215e859a8, 0xc240bba960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2551e7a40, 0x7f5d6ba4b5a8, 0xc215e859a8, 0xc240bba960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e859a8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130846 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213efb4b0, 0xc216246e40, 0xc238d1cd20, 0xc238d1cd80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 70678 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23938bcc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc260829a40, 0x7f5d6ba4b5a8, 0xc212dec098, 0xc20d0dd560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc260829a40, 0x7f5d6ba4b5a8, 0xc212dec098, 0xc20d0dd560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212dec098) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9087905 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2132ea640, 0xc21e91c7e0, 0xc23aca5140, 0xc23aca51a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131069 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216170f70, 0xc20b041bc0, 0xc2150a7020, 0xc2150a7080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 23292 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b4f6a70, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b4f6a70, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b4f6a10, 0xc22152c000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2520366e8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215631338, 0xc22152c000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215631338, 0xc22152c000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215631338, 0xc22152c000, 0xfa000, 0xfa000, 0xc214360000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20f1354d0, 0x7f5d6b8a1528, 0xc215631338, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215631338, 0xc214363020) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 14827 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241a6fe00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24e2d25f0, 0x7f5d6ba4b5a8, 0xc216171a90, 0xc263f37bc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24e2d25f0, 0x7f5d6ba4b5a8, 0xc216171a90, 0xc263f37bc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216171a90) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 21970 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a044df0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a044df0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a044d90, 0xc2236ee000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc218699978) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215ffe078, 0xc2236ee000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215ffe078, 0xc2236ee000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215ffe078, 0xc2236ee000, 0xfa000, 0xfa000, 0xc215480000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc213712390, 0x7f5d6b8a1528, 0xc215ffe078, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215ffe078, 0xc215480de0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 85729 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc218849400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d5f2280, 0x7f5d6ba4b5a8, 0xc20af1c830, 0xc20b27f380) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d5f2280, 0x7f5d6ba4b5a8, 0xc20af1c830, 0xc20b27f380) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af1c830) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 26702 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210dfc370, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210dfc370, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210dfc310, 0xc221a0e000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21fedee48) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21533d350, 0xc221a0e000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21533d350, 0xc221a0e000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21533d350, 0xc221a0e000, 0xfa000, 0xfa000, 0xc213680000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e2da6c0, 0x7f5d6b8a1528, 0xc21533d350, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21533d350, 0xc213685ce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131362 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21369fd40, 0xc21afe2720, 0xc23b9d2960, 0xc23b9d29c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 18520 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219f31400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d5f26e0, 0x7f5d6ba4b5a8, 0xc21622e658, 0xc238d1d1a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d5f26e0, 0x7f5d6ba4b5a8, 0xc21622e658, 0xc238d1d1a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622e658) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130864 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213718c28, 0xc21d8147e0, 0xc2611e61e0, 0xc2611e6240) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 51066 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc22b3ea6f0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc22b3ea6f0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc22b3ea690, 0xc22f7f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21e7cd4d8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2139cf2a8, 0xc22f7f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2139cf2a8, 0xc22f7f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2139cf2a8, 0xc22f7f0000, 0xfa000, 0xfa000, 0xc20ed30000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20db62de0, 0x7f5d6b8a1528, 0xc2139cf2a8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139cf2a8, 0xc20ed39d40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9129952 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e842a0, 0xc214ad5bc0, 0xc265d91e00, 0xc265d91e60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32285 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21510ea70, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21510ea70, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21510ea10, 0xc229df0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21e7cdc20) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2146172b8, 0xc229df0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2146172b8, 0xc229df0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2146172b8, 0xc229df0000, 0xfa000, 0xfa000, 0xc213980000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2136d0f60, 0x7f5d6b8a1528, 0xc2146172b8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2146172b8, 0xc213982600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131126 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216266ef0, 0xc21625da40, 0xc26945e300, 0xc26945e360) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 27072 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2328ec500, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc212153090, 0x7f5d6ba4b5a8, 0xc20b045bd8, 0xc20cfecea0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc212153090, 0x7f5d6ba4b5a8, 0xc20b045bd8, 0xc20cfecea0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b045bd8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 30293 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc208056610, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc208056610, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2080565b0, 0xc2272f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2167a8d08) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20b045bd8, 0xc2272f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20b045bd8, 0xc2272f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20b045bd8, 0xc2272f0000, 0xfa000, 0xfa000, 0xc2142a0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21639d3e0, 0x7f5d6b8a1528, 0xc20b045bd8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b045bd8, 0xc2142a95c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72572 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc24f819a40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265d6cb90, 0x7f5d6ba4b5a8, 0xc216026888, 0xc238d1acc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265d6cb90, 0x7f5d6ba4b5a8, 0xc216026888, 0xc238d1acc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216026888) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 72458 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21816f090, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21816f090, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21816f030, 0xc233af0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24458a810) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc211a48d60, 0xc233af0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc211a48d60, 0xc233af0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc211a48d60, 0xc233af0000, 0xfa000, 0xfa000, 0xc20d600000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21a45f620, 0x7f5d6b8a1528, 0xc211a48d60, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211a48d60, 0xc20d602d20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 43961 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc216f2c140, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc26480c190, 0x7f5d6ba4b5a8, 0xc214320a48, 0xc2612502a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc26480c190, 0x7f5d6ba4b5a8, 0xc214320a48, 0xc2612502a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214320a48) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 82976 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21c4b43c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc242ae7db0, 0x7f5d6ba4b5a8, 0xc215a2bb80, 0xc2458d35c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc242ae7db0, 0x7f5d6ba4b5a8, 0xc215a2bb80, 0xc2458d35c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2bb80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 55867 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc249734780, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25b470d20, 0x7f5d6ba4b5a8, 0xc21622efd8, 0xc23aa51e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25b470d20, 0x7f5d6ba4b5a8, 0xc21622efd8, 0xc23aa51e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622efd8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 28658 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21e708dc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc258644be0, 0x7f5d6ba4b5a8, 0xc215d0d668, 0xc261252a80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc258644be0, 0x7f5d6ba4b5a8, 0xc215d0d668, 0xc261252a80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215d0d668) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 62803 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a83a840, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a83a840, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a83a7e0, 0xc2318f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21606f578) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21621dc18, 0xc2318f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21621dc18, 0xc2318f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21621dc18, 0xc2318f0000, 0xfa000, 0xfa000, 0xc20f560000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2104414a0, 0x7f5d6b8a1528, 0xc21621dc18, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21621dc18, 0xc20f56ea20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 69037 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2170d25a0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2170d25a0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2170d2540, 0xc2350c0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24a563aa8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21622f258, 0xc2350c0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21622f258, 0xc2350c0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21622f258, 0xc2350c0000, 0xfa000, 0xfa000, 0xc219a50000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2157a7170, 0x7f5d6b8a1528, 0xc21622f258, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622f258, 0xc219a5e540) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9130848 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213574b38, 0xc219a5cde0, 0xc238d1cf60, 0xc238d1cfc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 86047 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2328ed180, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc256203ef0, 0x7f5d6ba4b5a8, 0xc215d0d6c8, 0xc20cfed9e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc256203ef0, 0x7f5d6ba4b5a8, 0xc215d0d6c8, 0xc20cfed9e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215d0d6c8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 13657 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21d06b400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc251853720, 0x7f5d6ba4b5a8, 0xc215d0d7d8, 0xc262295ec0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc251853720, 0x7f5d6ba4b5a8, 0xc215d0d7d8, 0xc262295ec0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215d0d7d8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9086445 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160fefc8, 0xc214a8a480, 0xc260845560, 0xc2608455c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130152 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216171110, 0xc2161c4900, 0xc238f47560, 0xc238f475c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131107 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2b338, 0xc215ec18c0, 0xc24e0f1f20, 0xc24e0f1f80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 49339 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f415680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d40d5e0, 0x7f5d6ba4b5a8, 0xc214fef858, 0xc26945e420) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d40d5e0, 0x7f5d6ba4b5a8, 0xc214fef858, 0xc26945e420) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214fef858) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 51075 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc212a14fb0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc212a14fb0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc212a14f50, 0xc22f0f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24f4722b8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214835df0, 0xc22f0f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214835df0, 0xc22f0f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214835df0, 0xc22f0f0000, 0xfa000, 0xfa000, 0xc210860000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20b127bf0, 0x7f5d6b8a1528, 0xc214835df0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214835df0, 0xc2108634a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 70698 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23b198b40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc264376730, 0x7f5d6ba4b5a8, 0xc212f85ca0, 0xc258bce840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc264376730, 0x7f5d6ba4b5a8, 0xc212f85ca0, 0xc258bce840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212f85ca0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 52634 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a0453a0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a0453a0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a045340, 0xc2305f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc212a6a1d8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213b04b08, 0xc2305f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213b04b08, 0xc2305f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213b04b08, 0xc2305f0000, 0xfa000, 0xfa000, 0xc219e10000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc211711320, 0x7f5d6b8a1528, 0xc213b04b08, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213b04b08, 0xc219e19320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 16886 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21c4b48c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc251853400, 0x7f5d6ba4b5a8, 0xc20bb58a88, 0xc238d1c780) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc251853400, 0x7f5d6ba4b5a8, 0xc20bb58a88, 0xc238d1c780) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bb58a88) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 91703 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a387330, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a387330, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a3872d0, 0xc237ff0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bcc72e0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213ffaaf8, 0xc237ff0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213ffaaf8, 0xc237ff0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213ffaaf8, 0xc237ff0000, 0xfa000, 0xfa000, 0xc21db70000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2109c3320, 0x7f5d6b8a1528, 0xc213ffaaf8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffaaf8, 0xc21db7f4a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 20195 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23dccb7c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2560a8730, 0x7f5d6ba4b5a8, 0xc20af4f7b8, 0xc265d91f20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2560a8730, 0x7f5d6ba4b5a8, 0xc20af4f7b8, 0xc265d91f20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4f7b8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 18394 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f035400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25b470e10, 0x7f5d6ba4b5a8, 0xc20af4f028, 0xc264625620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25b470e10, 0x7f5d6ba4b5a8, 0xc20af4f028, 0xc264625620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4f028) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 15421 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc24b9dd540, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2551e6dc0, 0x7f5d6ba4b5a8, 0xc215a2bb10, 0xc23b94dbc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2551e6dc0, 0x7f5d6ba4b5a8, 0xc215a2bb10, 0xc23b94dbc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2bb10) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 56241 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21816e140, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21816e140, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21816e0e0, 0xc2310f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2539278b8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21ff396c8, 0xc2310f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21ff396c8, 0xc2310f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21ff396c8, 0xc2310f0000, 0xfa000, 0xfa000, 0xc2152e0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214c84a50, 0x7f5d6b8a1528, 0xc21ff396c8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff396c8, 0xc2152e0f00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72474 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2338ec610, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2338ec610, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2338ec5b0, 0xc2345f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2167a94e8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc210df0838, 0xc2345f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc210df0838, 0xc2345f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc210df0838, 0xc2345f0000, 0xfa000, 0xfa000, 0xc21f760000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2338ea870, 0x7f5d6b8a1528, 0xc210df0838, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210df0838, 0xc21f766c60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 91702 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21d06b900, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc251853ae0, 0x7f5d6ba4b5a8, 0xc213ffaaf8, 0xc25fa2a360) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc251853ae0, 0x7f5d6ba4b5a8, 0xc213ffaaf8, 0xc25fa2a360) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffaaf8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9087884 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214515f30, 0xc20e02f080, 0xc25a17aae0, 0xc25a17ab40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130799 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214de80a8, 0xc214c9c180, 0xc264625bc0, 0xc264625c20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 72471 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21de34760, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21de34760, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21de34700, 0xc2340f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21ddd61c8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc211056b20, 0xc2340f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc211056b20, 0xc2340f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc211056b20, 0xc2340f0000, 0xfa000, 0xfa000, 0xc21f760000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc217f122a0, 0x7f5d6b8a1528, 0xc211056b20, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211056b20, 0xc21f766960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 18393 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d25cdc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2551e72c0, 0x7f5d6ba4b5a8, 0xc20af4eb30, 0xc240bba060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2551e72c0, 0x7f5d6ba4b5a8, 0xc20af4eb30, 0xc240bba060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4eb30) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 10428 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc216f2ca00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc26480c870, 0x7f5d6ba4b5a8, 0xc20af1d400, 0xc261250960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc26480c870, 0x7f5d6ba4b5a8, 0xc20af1d400, 0xc261250960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af1d400) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 11269 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20af1bc60, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af1bc60, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af1bc00, 0xc225ec0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211dbfb08) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20b044c58, 0xc225ec0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20b044c58, 0xc225ec0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20b044c58, 0xc225ec0000, 0xfa000, 0xfa000, 0xc2162b0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214f7eb70, 0x7f5d6b8a1528, 0xc20b044c58, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b044c58, 0xc2162bd8c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 25178 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc24f819cc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2551e6ff0, 0x7f5d6ba4b5a8, 0xc2080c6828, 0xc238d1af00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2551e6ff0, 0x7f5d6ba4b5a8, 0xc2080c6828, 0xc238d1af00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c6828) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131020 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2162669c0, 0xc210b7aea0, 0xc261252960, 0xc2612529c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32200 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc217385170, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc217385170, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc217385110, 0xc2287f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240b0aa58) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215edaa68, 0xc2287f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215edaa68, 0xc2287f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215edaa68, 0xc2287f0000, 0xfa000, 0xfa000, 0xc215e70000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2147e6d20, 0x7f5d6b8a1528, 0xc215edaa68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215edaa68, 0xc215e77ce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32090 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc213546ca0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc213546ca0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc213546c40, 0xc228df0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21606eb60) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20bb59378, 0xc228df0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20bb59378, 0xc228df0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20bb59378, 0xc228df0000, 0xfa000, 0xfa000, 0xc214f80000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2144f32f0, 0x7f5d6b8a1528, 0xc20bb59378, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bb59378, 0xc214f80a80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 23433 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21ff7c140, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23f2cec30, 0x7f5d6ba4b5a8, 0xc215fff7f0, 0xc264aea600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23f2cec30, 0x7f5d6ba4b5a8, 0xc215fff7f0, 0xc264aea600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215fff7f0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130800 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2131d17a0, 0xc21f773c20, 0xc264625ce0, 0xc264625d40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 43964 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc213249410, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc213249410, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2132493b0, 0xc22c0f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240d6b090) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2139a4f40, 0xc22c0f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2139a4f40, 0xc22c0f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2139a4f40, 0xc22c0f0000, 0xfa000, 0xfa000, 0xc212ca0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc22c1ea000, 0x7f5d6b8a1528, 0xc2139a4f40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139a4f40, 0xc212caa900) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 67358 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20e599aa0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20e599aa0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20e599a40, 0xc232bf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bfff478) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2132eb6d0, 0xc232bf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2132eb6d0, 0xc232bf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2132eb6d0, 0xc232bf0000, 0xfa000, 0xfa000, 0xc214690000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21e570810, 0x7f5d6b8a1528, 0xc2132eb6d0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2132eb6d0, 0xc21469f3e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 74351 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2410652c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2684839a0, 0x7f5d6ba4b5a8, 0xc210c75030, 0xc21aab1320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2684839a0, 0x7f5d6ba4b5a8, 0xc210c75030, 0xc21aab1320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210c75030) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 64271 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23fbc3a40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21c03f4f0, 0x7f5d6ba4b5a8, 0xc2132ea640, 0xc23aca5140) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21c03f4f0, 0x7f5d6ba4b5a8, 0xc2132ea640, 0xc23aca5140) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2132ea640) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 70814 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21d06be00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc251853e00, 0x7f5d6ba4b5a8, 0xc211057e28, 0xc25fa2a7e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc251853e00, 0x7f5d6ba4b5a8, 0xc211057e28, 0xc25fa2a7e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211057e28) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 23387 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a045020, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a045020, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a044fc0, 0xc2237e8000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24f473138) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215ffe528, 0xc2237e8000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215ffe528, 0xc2237e8000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215ffe528, 0xc2237e8000, 0xfa000, 0xfa000, 0xc214350000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc213868ae0, 0x7f5d6b8a1528, 0xc215ffe528, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215ffe528, 0xc21435b5c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 18643 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20ba20c30, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20ba20c30, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20ba20bd0, 0xc2243ba000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24f4726a8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21622f160, 0xc2243ba000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21622f160, 0xc2243ba000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21622f160, 0xc2243ba000, 0xfa000, 0xfa000, 0xc216140000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21d7256e0, 0x7f5d6b8a1528, 0xc21622f160, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622f160, 0xc216142e40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 54932 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc213248060, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc213248060, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc213248000, 0xc230bf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211dbf580) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2136fcfa0, 0xc230bf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2136fcfa0, 0xc230bf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2136fcfa0, 0xc230bf0000, 0xfa000, 0xfa000, 0xc21b510000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ccbf8f0, 0x7f5d6b8a1528, 0xc2136fcfa0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2136fcfa0, 0xc21b513320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72568 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2170d2530, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2170d2530, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2170d24d0, 0xc234fc6000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2557b3ba8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21622e9b0, 0xc234fc6000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21622e9b0, 0xc234fc6000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21622e9b0, 0xc234fc6000, 0xfa000, 0xfa000, 0xc21c190000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc213381b00, 0x7f5d6b8a1528, 0xc21622e9b0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622e9b0, 0xc21c195ec0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131065 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21369f030, 0xc22030ad20, 0xc2150a6ba0, 0xc2150a6c00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131091 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e72c88, 0xc216104ae0, 0xc23e1f1bc0, 0xc23e1f1c20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131140 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bb59378, 0xc214f80a80, 0xc23cc70240, 0xc23cc702a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 51068 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f415b80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d40d180, 0x7f5d6ba4b5a8, 0xc2139cf280, 0xc24ec8bd40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d40d180, 0x7f5d6ba4b5a8, 0xc2139cf280, 0xc24ec8bd40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139cf280) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130543 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b42c10, 0xc215eb1b60, 0xc25fa2a120, 0xc25fa2a180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 21898 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc245d8e140, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d40d7c0, 0x7f5d6ba4b5a8, 0xc215edbb50, 0xc26945e660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d40d7c0, 0x7f5d6ba4b5a8, 0xc215edbb50, 0xc26945e660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215edbb50) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131092 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2147a1318, 0xc211744960, 0xc23e1f1ce0, 0xc23e1f1d40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130545 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffaaf8, 0xc21db7f4a0, 0xc25fa2a360, 0xc25fa2a3c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 84785 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210e95950, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210e95950, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210e958f0, 0xc2373f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bcc7998) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215c5d060, 0xc2373f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215c5d060, 0xc2373f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215c5d060, 0xc2373f0000, 0xfa000, 0xfa000, 0xc2100c0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e986cf0, 0x7f5d6b8a1528, 0xc215c5d060, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215c5d060, 0xc2100cd8c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 47242 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc25d682000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc259030f00, 0x7f5d6ba4b5a8, 0xc213ffa248, 0xc263376060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc259030f00, 0x7f5d6ba4b5a8, 0xc213ffa248, 0xc263376060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffa248) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 16750 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2630e63c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc218a3bef0, 0x7f5d6ba4b5a8, 0xc20bfa2400, 0xc238f479e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc218a3bef0, 0x7f5d6ba4b5a8, 0xc20bfa2400, 0xc238f479e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bfa2400) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32295 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20eb431e0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20eb431e0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20eb43180, 0xc229bf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc244457108) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214a75d98, 0xc229bf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214a75d98, 0xc229bf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214a75d98, 0xc229bf0000, 0xfa000, 0xfa000, 0xc214290000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2139a36e0, 0x7f5d6b8a1528, 0xc214a75d98, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a75d98, 0xc21429e660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 23552 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219f31680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d5f32c0, 0x7f5d6ba4b5a8, 0xc215e73238, 0xc238d1d3e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d5f32c0, 0x7f5d6ba4b5a8, 0xc215e73238, 0xc238d1d3e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e73238) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 72241 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23b199040, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc264376af0, 0x7f5d6ba4b5a8, 0xc212bbfeb0, 0xc258bcecc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc264376af0, 0x7f5d6ba4b5a8, 0xc212bbfeb0, 0xc258bcecc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212bbfeb0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 61019 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a3866f0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a3866f0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a386690, 0xc2312f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24458a778) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213718c28, 0xc2312f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213718c28, 0xc2312f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213718c28, 0xc2312f0000, 0xfa000, 0xfa000, 0xc21d810000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21b354e10, 0x7f5d6b8a1528, 0xc213718c28, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213718c28, 0xc21d8147e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 63102 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23d417540, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2153c2500, 0x7f5d6ba4b5a8, 0xc2131d0390, 0xc21c04cf60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2153c2500, 0x7f5d6ba4b5a8, 0xc2131d0390, 0xc21c04cf60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2131d0390) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 25079 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21d06b7c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc251853950, 0x7f5d6ba4b5a8, 0xc215c5ce70, 0xc25fa2a240) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc251853950, 0x7f5d6ba4b5a8, 0xc215c5ce70, 0xc25fa2a240) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215c5ce70) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 52708 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc215bcb400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265d6c820, 0x7f5d6ba4b5a8, 0xc2142ee640, 0xc20b281e60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265d6c820, 0x7f5d6ba4b5a8, 0xc2142ee640, 0xc20b281e60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2142ee640) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 72574 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23d4172c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc26480c280, 0x7f5d6ba4b5a8, 0xc2160269c8, 0xc21c04cd20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc26480c280, 0x7f5d6ba4b5a8, 0xc2160269c8, 0xc21c04cd20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160269c8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 11646 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc208057020, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc208057020, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc208056fc0, 0xc226e70000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240d6b478) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af1d400, 0xc226e70000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20af1d400, 0xc226e70000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20af1d400, 0xc226e70000, 0xfa000, 0xfa000, 0xc2161c0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc215fbaa80, 0x7f5d6b8a1528, 0xc20af1d400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af1d400, 0xc2161c4960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 26700 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc215659640, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc215659640, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2156595e0, 0xc22181a000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24f4727d8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215cd2928, 0xc22181a000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215cd2928, 0xc22181a000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215cd2928, 0xc22181a000, 0xfa000, 0xfa000, 0xc213680000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ce118f0, 0x7f5d6b8a1528, 0xc215cd2928, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd2928, 0xc213685b60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 25080 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2170d3640, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2170d3640, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2170d35e0, 0xc221626000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bcc7180) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215c5ce70, 0xc221626000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215c5ce70, 0xc221626000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215c5ce70, 0xc221626000, 0xfa000, 0xfa000, 0xc213fd0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20d181440, 0x7f5d6b8a1528, 0xc215c5ce70, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215c5ce70, 0xc213fd80c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 12526 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d25d180, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2551e7860, 0x7f5d6ba4b5a8, 0xc215b42588, 0xc240bba720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2551e7860, 0x7f5d6ba4b5a8, 0xc215b42588, 0xc240bba720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b42588) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 86230 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241803680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23f2ce2d0, 0x7f5d6ba4b5a8, 0xc21519f698, 0xc264c39ce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23f2ce2d0, 0x7f5d6ba4b5a8, 0xc21519f698, 0xc264c39ce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21519f698) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32226 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c29f640, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c29f640, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c29f5e0, 0xc2281f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc244457358) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e73288, 0xc2281f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e73288, 0xc2281f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e73288, 0xc2281f0000, 0xfa000, 0xfa000, 0xc215fd0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214a1c870, 0x7f5d6b8a1528, 0xc215e73288, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e73288, 0xc215fd1aa0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9087791 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b42f68, 0xc21e91c660, 0xc217977bc0, 0xc217977c20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 23553 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2080574f0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2080574f0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc208057490, 0xc22104a000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25561b390) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e73238, 0xc22104a000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e73238, 0xc22104a000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e73238, 0xc22104a000, 0xfa000, 0xfa000, 0xc214310000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ec129c0, 0x7f5d6b8a1528, 0xc215e73238, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e73238, 0xc21431cfc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 17059 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219f317c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d5f33b0, 0x7f5d6ba4b5a8, 0xc21538e510, 0xc238d1d500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d5f33b0, 0x7f5d6ba4b5a8, 0xc21538e510, 0xc238d1d500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21538e510) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 89662 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21e709e00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc258645090, 0x7f5d6ba4b5a8, 0xc2136fc418, 0xc261253020) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc258645090, 0x7f5d6ba4b5a8, 0xc2136fc418, 0xc261253020) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2136fc418) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 67306 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21418e060, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21418e060, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21418e000, 0xc2323f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24d672830) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2131d1438, 0xc2323f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2131d1438, 0xc2323f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2131d1438, 0xc2323f0000, 0xfa000, 0xfa000, 0xc219a50000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21baaf1a0, 0x7f5d6b8a1528, 0xc2131d1438, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2131d1438, 0xc219a58de0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 79552 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc25e920000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2153c2c80, 0x7f5d6ba4b5a8, 0xc2145f59f0, 0xc21c04d860) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2153c2c80, 0x7f5d6ba4b5a8, 0xc2145f59f0, 0xc21c04d860) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2145f59f0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 11644 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bfe0220, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bfe0220, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bfe01c0, 0xc226ff6000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25294c7e0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc216171110, 0xc226ff6000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc216171110, 0xc226ff6000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc216171110, 0xc226ff6000, 0xfa000, 0xfa000, 0xc2161c0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2161bb950, 0x7f5d6b8a1528, 0xc216171110, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216171110, 0xc2161c4900) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32220 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc249734500, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23df6ad20, 0x7f5d6ba4b5a8, 0xc215e85d58, 0xc260845680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23df6ad20, 0x7f5d6ba4b5a8, 0xc215e85d58, 0xc260845680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e85d58) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 26909 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc213163b80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc264ccff40, 0x7f5d6ba4b5a8, 0xc214f1f530, 0xc263f373e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc264ccff40, 0x7f5d6ba4b5a8, 0xc214f1f530, 0xc263f373e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1f530) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 34621 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21bdabb80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24a3a9450, 0x7f5d6ba4b5a8, 0xc215631b50, 0xc242581f80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24a3a9450, 0x7f5d6ba4b5a8, 0xc215631b50, 0xc242581f80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215631b50) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 35429 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210dfce60, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210dfce60, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210dfce00, 0xc22aaf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25102c558) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21621d058, 0xc22aaf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21621d058, 0xc22aaf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21621d058, 0xc22aaf0000, 0xfa000, 0xfa000, 0xc20f9d0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20dcecab0, 0x7f5d6b8a1528, 0xc21621d058, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21621d058, 0xc20f9dbf20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131285 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff391f0, 0xc20def6840, 0xc238d1b800, 0xc238d1b860) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131204 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21462e850, 0xc215499500, 0xc25fa2a6c0, 0xc25fa2a720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 82975 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bceaae0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bceaae0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bceaa80, 0xc236e0c000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc257500100) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214515f30, 0xc236e0c000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214515f30, 0xc236e0c000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214515f30, 0xc236e0c000, 0xfa000, 0xfa000, 0xc20e020000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20bf7d9b0, 0x7f5d6b8a1528, 0xc214515f30, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214515f30, 0xc20e02f080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 13779 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc211c7a6f0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc211c7a6f0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc211c7a690, 0xc2250f2000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2557b3c40) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215b43f70, 0xc2250f2000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215b43f70, 0xc2250f2000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215b43f70, 0xc2250f2000, 0xfa000, 0xfa000, 0xc2162a0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21467c360, 0x7f5d6b8a1528, 0xc215b43f70, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b43f70, 0xc2162ab260) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 13658 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc211c7a060, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc211c7a060, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc211c7a000, 0xc224b16000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211dbfb90) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215b42c10, 0xc224b16000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215b42c10, 0xc224b16000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215b42c10, 0xc224b16000, 0xfa000, 0xfa000, 0xc215eb0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2144e5410, 0x7f5d6b8a1528, 0xc215b42c10, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b42c10, 0xc215eb1b60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131190 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622eb98, 0xc214440240, 0xc240bba4e0, 0xc240bba540) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32094 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc213546ed0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc213546ed0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc213546e70, 0xc2292f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24fe8f278) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2160fefc8, 0xc2292f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2160fefc8, 0xc2292f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2160fefc8, 0xc2292f0000, 0xfa000, 0xfa000, 0xc214a80000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2142cce10, 0x7f5d6b8a1528, 0xc2160fefc8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160fefc8, 0xc214a8a480) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131170 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21149f588, 0xc20c616ea0, 0xc2425819e0, 0xc242581a40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 19999 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c2b07d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c2b07d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c2b0770, 0xc2245ae000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc212a6a680) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21632a3f0, 0xc2245ae000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21632a3f0, 0xc2245ae000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21632a3f0, 0xc2245ae000, 0xfa000, 0xfa000, 0xc215dd0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214bec660, 0x7f5d6b8a1528, 0xc21632a3f0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21632a3f0, 0xc215ddec60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32306 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b93d560, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b93d560, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b93d500, 0xc2295f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bfff348) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214b76bd0, 0xc2295f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214b76bd0, 0xc2295f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214b76bd0, 0xc2295f0000, 0xfa000, 0xfa000, 0xc214790000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214261e90, 0x7f5d6b8a1528, 0xc214b76bd0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b76bd0, 0xc214797680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32088 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b337db0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b337db0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b337d50, 0xc2283f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bcc6be8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2152fa770, 0xc2283f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2152fa770, 0xc2283f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2152fa770, 0xc2283f0000, 0xfa000, 0xfa000, 0xc215fb0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214932660, 0x7f5d6b8a1528, 0xc2152fa770, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2152fa770, 0xc215fb33e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131059 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2152fa770, 0xc215fb33e0, 0xc2150a64e0, 0xc2150a6540) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 79553 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210ed36b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210ed36b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210ed3650, 0xc236a24000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24d6727a8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2080c71a8, 0xc236a24000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2080c71a8, 0xc236a24000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2080c71a8, 0xc236a24000, 0xfa000, 0xfa000, 0xc20bcd0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2136310b0, 0x7f5d6b8a1528, 0xc2080c71a8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c71a8, 0xc20bcdec00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32239 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20eb42a00, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20eb42a00, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20eb429a0, 0xc228af0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211da0a88) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214f1f1b0, 0xc228af0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214f1f1b0, 0xc228af0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214f1f1b0, 0xc228af0000, 0xfa000, 0xfa000, 0xc2151b0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21460acc0, 0x7f5d6b8a1528, 0xc214f1f1b0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1f1b0, 0xc2151bd0e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 16887 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc215bcbe00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265d6cfa0, 0x7f5d6ba4b5a8, 0xc20bb58bd8, 0xc2611e6780) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265d6cfa0, 0x7f5d6ba4b5a8, 0xc20bb58bd8, 0xc2611e6780) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bb58bd8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 13723 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d667540, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25596a140, 0x7f5d6ba4b5a8, 0xc215b43f70, 0xc23e1f1860) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25596a140, 0x7f5d6ba4b5a8, 0xc215b43f70, 0xc23e1f1860) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b43f70) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 74807 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20eb434f0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20eb434f0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20eb43490, 0xc235d72000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240b0a858) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214f1ed18, 0xc235d72000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214f1ed18, 0xc235d72000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214f1ed18, 0xc235d72000, 0xfa000, 0xfa000, 0xc216190000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2162032c0, 0x7f5d6b8a1528, 0xc214f1ed18, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1ed18, 0xc216198cc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 26703 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc215659870, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc215659870, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc215659810, 0xc221c02000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21606f898) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215cd3de8, 0xc221c02000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215cd3de8, 0xc221c02000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215cd3de8, 0xc221c02000, 0xfa000, 0xfa000, 0xc213680000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e2da960, 0x7f5d6b8a1528, 0xc215cd3de8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd3de8, 0xc213685e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 37414 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219b0edc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc266117590, 0x7f5d6ba4b5a8, 0xc21fac0610, 0xc2150a6f00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc266117590, 0x7f5d6ba4b5a8, 0xc21fac0610, 0xc2150a6f00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21fac0610) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32323 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23dccb040, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2560a80f0, 0x7f5d6ba4b5a8, 0xc214b72c60, 0xc265d91860) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2560a80f0, 0x7f5d6ba4b5a8, 0xc214b72c60, 0xc265d91860) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b72c60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32172 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f035040, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc259030460, 0x7f5d6ba4b5a8, 0xc2160c5c10, 0xc2646252c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc259030460, 0x7f5d6ba4b5a8, 0xc2160c5c10, 0xc2646252c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160c5c10) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 59674 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23aabd540, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265651f90, 0x7f5d6ba4b5a8, 0xc21ff396c8, 0xc23cc70120) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265651f90, 0x7f5d6ba4b5a8, 0xc21ff396c8, 0xc23cc70120) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff396c8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131053 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21621d790, 0xc20ca78420, 0xc261250cc0, 0xc261250d20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 72153 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc218650640, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2618123c0, 0x7f5d6ba4b5a8, 0xc211a48640, 0xc25fa2aea0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2618123c0, 0x7f5d6ba4b5a8, 0xc211a48640, 0xc25fa2aea0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211a48640) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 29303 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f415cc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24566a960, 0x7f5d6ba4b5a8, 0xc21519f7e8, 0xc24ec8be60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24566a960, 0x7f5d6ba4b5a8, 0xc21519f7e8, 0xc24ec8be60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21519f7e8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131162 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bb593a8, 0xc2143ca6c0, 0xc264aeaa80, 0xc264aeaae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9502 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b336680, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b336680, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b336620, 0xc224808000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240b0a758) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2160c4050, 0xc224808000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2160c4050, 0xc224808000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2160c4050, 0xc224808000, 0xfa000, 0xfa000, 0xc215fd0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21544c270, 0x7f5d6b8a1528, 0xc2160c4050, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160c4050, 0xc215fd01e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 31163 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20ce6bbf0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20ce6bbf0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20ce6bb90, 0xc227af0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc210321a18) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21ff39aa0, 0xc227af0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21ff39aa0, 0xc227af0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21ff39aa0, 0xc227af0000, 0xfa000, 0xfa000, 0xc216440000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214f449c0, 0x7f5d6b8a1528, 0xc21ff39aa0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff39aa0, 0xc216449f80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131096 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21340e4b8, 0xc21c80dec0, 0xc24759e180, 0xc24759e1e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 30876 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219b0ec80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2661174a0, 0x7f5d6ba4b5a8, 0xc21ff38aa0, 0xc2150a6de0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2661174a0, 0x7f5d6ba4b5a8, 0xc21ff38aa0, 0xc2150a6de0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff38aa0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 28643 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc208010300, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc208010300, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2080102a0, 0xc2223d2000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc213d1de68) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21533c010, 0xc2223d2000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21533c010, 0xc2223d2000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21533c010, 0xc2223d2000, 0xfa000, 0xfa000, 0xc213000000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20bdbc0f0, 0x7f5d6b8a1528, 0xc21533c010, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21533c010, 0xc213006000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 15458 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc208011480, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc208011480, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc208011420, 0xc223af0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24a563160) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215a2bb10, 0xc223af0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215a2bb10, 0xc223af0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215a2bb10, 0xc223af0000, 0xfa000, 0xfa000, 0xc215e00000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21415ede0, 0x7f5d6b8a1528, 0xc215a2bb10, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2bb10, 0xc215e07e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9084289 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212ded408, 0xc2138324e0, 0xc261b5ef00, 0xc261b5ef60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 72344 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc216f2cc80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc26480ca50, 0x7f5d6ba4b5a8, 0xc20bfa2020, 0xc261250ba0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc26480ca50, 0x7f5d6ba4b5a8, 0xc20bfa2020, 0xc261250ba0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bfa2020) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 62781 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21f69b100, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21f69b100, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21f69b0a0, 0xc2313f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc248607e70) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213656840, 0xc2313f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213656840, 0xc2313f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213656840, 0xc2313f0000, 0xfa000, 0xfa000, 0xc219310000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2157a6450, 0x7f5d6b8a1528, 0xc213656840, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213656840, 0xc2193163c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131171 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a08128, 0xc215fb3800, 0xc242581b00, 0xc242581b60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 88121 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2418037c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23f2ce3c0, 0x7f5d6ba4b5a8, 0xc213ed91b0, 0xc264c39e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23f2ce3c0, 0x7f5d6ba4b5a8, 0xc213ed91b0, 0xc264c39e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ed91b0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 55868 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b93d790, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b93d790, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b93d730, 0xc2309f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24458a468) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21622efd8, 0xc2309f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21622efd8, 0xc2309f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21622efd8, 0xc2309f0000, 0xfa000, 0xfa000, 0xc214440000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc210dd2a20, 0x7f5d6b8a1528, 0xc21622efd8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622efd8, 0xc2144402a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9130149 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21621d7a8, 0xc2120bd4a0, 0xc238f47200, 0xc238f47260) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 37420 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a0fba30, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a0fba30, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a0fb9d0, 0xc22aff0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2167a8f68) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21fac0950, 0xc22aff0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21fac0950, 0xc22aff0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21fac0950, 0xc22aff0000, 0xfa000, 0xfa000, 0xc20ed30000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc215db24b0, 0x7f5d6b8a1528, 0xc21fac0950, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21fac0950, 0xc20ed38960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 65930 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20e599640, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20e599640, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20e5995e0, 0xc2329f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23d5d4298) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2132ea640, 0xc2329f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2132ea640, 0xc2329f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2132ea640, 0xc2329f0000, 0xfa000, 0xfa000, 0xc21e910000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ea55020, 0x7f5d6b8a1528, 0xc2132ea640, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2132ea640, 0xc21e91c7e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 11485 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23aabd040, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc262d5c730, 0x7f5d6ba4b5a8, 0xc215fff0e0, 0xc23cc70a20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc262d5c730, 0x7f5d6ba4b5a8, 0xc215fff0e0, 0xc23cc70a20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215fff0e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9129945 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160fea18, 0xc215fccc60, 0xc265d91620, 0xc265d91680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130955 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216026888, 0xc211579200, 0xc238d1acc0, 0xc238d1ad20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131001 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e85b60, 0xc20b535d40, 0xc23aa51f20, 0xc23aa51f80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 11275 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23dccbb80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc258645180, 0x7f5d6ba4b5a8, 0xc216267ed8, 0xc25a4502a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc258645180, 0x7f5d6ba4b5a8, 0xc216267ed8, 0xc25a4502a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216267ed8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131210 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2109374a0, 0xc216021320, 0xc25fa2ad80, 0xc25fa2ade0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 20119 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc238e03e00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2586446e0, 0x7f5d6ba4b5a8, 0xc215a2b338, 0xc24e0f1f20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2586446e0, 0x7f5d6ba4b5a8, 0xc215a2b338, 0xc24e0f1f20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2b338) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131130 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c6a00, 0xc21d3bd200, 0xc26945e780, 0xc26945e7e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 74352 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210f01100, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210f01100, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210f010a0, 0xc23598a000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21606ee58) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc210c75030, 0xc23598a000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc210c75030, 0xc23598a000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc210c75030, 0xc23598a000, 0xfa000, 0xfa000, 0xc216220000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2162023f0, 0x7f5d6b8a1528, 0xc210c75030, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210c75030, 0xc216224cc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32272 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc213547480, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc213547480, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc213547420, 0xc22a4f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21e7cdd48) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2145f5cc0, 0xc22a4f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2145f5cc0, 0xc22a4f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2145f5cc0, 0xc22a4f0000, 0xfa000, 0xfa000, 0xc213440000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2131e83c0, 0x7f5d6b8a1528, 0xc2145f5cc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2145f5cc0, 0xc213445080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 88122 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b4f7250, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b4f7250, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b4f71f0, 0xc237bf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24d672538) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213ed91b0, 0xc237bf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213ed91b0, 0xc237bf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213ed91b0, 0xc237bf0000, 0xfa000, 0xfa000, 0xc213190000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc210e92810, 0x7f5d6b8a1528, 0xc213ed91b0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ed91b0, 0xc213191f80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 58164 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219b0e3c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2153c2640, 0x7f5d6ba4b5a8, 0xc21369f030, 0xc2150a6ba0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2153c2640, 0x7f5d6ba4b5a8, 0xc21369f030, 0xc2150a6ba0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21369f030) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131317 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214834070, 0xc213740de0, 0xc263f37ce0, 0xc263f37d40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131331 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af1dde0, 0xc213661620, 0xc25a450180, 0xc25a4501e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 43923 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2421d9e00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc266116d70, 0x7f5d6ba4b5a8, 0xc213efa080, 0xc2150a6180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc266116d70, 0x7f5d6ba4b5a8, 0xc213efa080, 0xc2150a6180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213efa080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 12414 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a0440d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a0440d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a044070, 0xc2252e6000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240b0a658) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e72c88, 0xc2252e6000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e72c88, 0xc2252e6000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e72c88, 0xc2252e6000, 0xfa000, 0xfa000, 0xc216100000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2142ad1a0, 0x7f5d6b8a1528, 0xc215e72c88, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e72c88, 0xc216104ae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9129885 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1f530, 0xc21362ede0, 0xc263f373e0, 0xc263f37440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 88178 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc215e04300, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc215e04300, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc215e042a0, 0xc2377f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211da0db0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af1c830, 0xc2377f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20af1c830, 0xc2377f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20af1c830, 0xc2377f0000, 0xfa000, 0xfa000, 0xc20e7f0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20d1ec6c0, 0x7f5d6b8a1528, 0xc20af1c830, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af1c830, 0xc20e7fd1a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131176 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e32168, 0xc215fd5740, 0xc26243e0c0, 0xc26243e120) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 28302 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21bdab900, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24a3a9270, 0x7f5d6ba4b5a8, 0xc214c76a30, 0xc242581d40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24a3a9270, 0x7f5d6ba4b5a8, 0xc214c76a30, 0xc242581d40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214c76a30) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50953 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23dccaa00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25b7d3e00, 0x7f5d6ba4b5a8, 0xc214c76118, 0xc265d912c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25b7d3e00, 0x7f5d6ba4b5a8, 0xc214c76118, 0xc265d912c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214c76118) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 65996 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc219c06d80, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc219c06d80, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc219c06d20, 0xc2320f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25102c7a8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21324b3b0, 0xc2320f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21324b3b0, 0xc2320f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21324b3b0, 0xc2320f0000, 0xfa000, 0xfa000, 0xc212970000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20c5c2de0, 0x7f5d6b8a1528, 0xc21324b3b0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21324b3b0, 0xc21297d1a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131114 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215fff568, 0xc20c5f9800, 0xc258bce720, 0xc258bce780) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129944 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e32460, 0xc20f9dbbc0, 0xc265d91500, 0xc265d91560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 29417 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b93c3e0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b93c3e0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b93c380, 0xc2270f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ef50718) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af4e9e8, 0xc2270f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20af4e9e8, 0xc2270f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20af4e9e8, 0xc2270f0000, 0xfa000, 0xfa000, 0xc21b4c0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc215fd60f0, 0x7f5d6b8a1528, 0xc20af4e9e8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4e9e8, 0xc21b4c6a20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72348 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc215a6c1b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc215a6c1b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc215a6c150, 0xc234dd2000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24458a198) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc216267008, 0xc234dd2000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc216267008, 0xc234dd2000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc216267008, 0xc234dd2000, 0xfa000, 0xfa000, 0xc219310000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21da612f0, 0x7f5d6b8a1528, 0xc216267008, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216267008, 0xc219316840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9130826 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a05e78, 0xc215ed45a0, 0xc20b27f6e0, 0xc20b27f740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 12745 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d667680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc213e7b2c0, 0x7f5d6ba4b5a8, 0xc215cd2ac0, 0xc23e1f1980) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc213e7b2c0, 0x7f5d6ba4b5a8, 0xc215cd2ac0, 0xc23e1f1980) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd2ac0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130541 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215d0d7d8, 0xc215fc41e0, 0xc262295ec0, 0xc262295f20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 74684 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc208069a40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc218a3b810, 0x7f5d6ba4b5a8, 0xc214b77558, 0xc238f47320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc218a3b810, 0x7f5d6ba4b5a8, 0xc214b77558, 0xc238f47320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b77558) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 49300 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc245500000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc268483c70, 0x7f5d6ba4b5a8, 0xc215e72de8, 0xc21aab1680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc268483c70, 0x7f5d6ba4b5a8, 0xc215e72de8, 0xc21aab1680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e72de8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 64097 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20fd44fb0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20fd44fb0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20fd44f50, 0xc231af0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240b0ab58) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21340e4b8, 0xc231af0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21340e4b8, 0xc231af0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21340e4b8, 0xc231af0000, 0xfa000, 0xfa000, 0xc21c800000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21916d1a0, 0x7f5d6b8a1528, 0xc21340e4b8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21340e4b8, 0xc21c80dec0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131268 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213657010, 0xc20def0720, 0xc263376180, 0xc2633761e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131206 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21538e430, 0xc215ddef00, 0xc25fa2a900, 0xc25fa2a960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 13715 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21c4b4780, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24b217d60, 0x7f5d6ba4b5a8, 0xc215b42c20, 0xc238d1c660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24b217d60, 0x7f5d6ba4b5a8, 0xc215b42c20, 0xc238d1c660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b42c20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 72470 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21816f330, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21816f330, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21816f2d0, 0xc233df0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24d672740) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc211a49658, 0xc233df0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc211a49658, 0xc233df0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc211a49658, 0xc233df0000, 0xfa000, 0xfa000, 0xc21f760000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2338ea660, 0x7f5d6b8a1528, 0xc211a49658, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211a49658, 0xc21f766900) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72453 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc213249f70, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc213249f70, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc213249f10, 0xc2337f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc256ce4458) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21149e030, 0xc2337f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21149e030, 0xc2337f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21149e030, 0xc2337f0000, 0xfa000, 0xfa000, 0xc20d600000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2338ea000, 0x7f5d6b8a1528, 0xc21149e030, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21149e030, 0xc20d6028a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 15201 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23d417cc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2153c2aa0, 0x7f5d6ba4b5a8, 0xc21538e7a8, 0xc21c04d620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2153c2aa0, 0x7f5d6ba4b5a8, 0xc21538e7a8, 0xc21c04d620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21538e7a8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 11669 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c29e1b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c29e1b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c29e150, 0xc21644a000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc212a6a6d8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21632a260, 0xc21644a000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21632a260, 0xc21644a000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21632a260, 0xc21644a000, 0xfa000, 0xfa000, 0xc218a90000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214f7ecc0, 0x7f5d6b8a1528, 0xc21632a260, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21632a260, 0xc218a915c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 12738 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23dccadc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265d6caa0, 0x7f5d6ba4b5a8, 0xc2160fea18, 0xc265d91620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265d6caa0, 0x7f5d6ba4b5a8, 0xc2160fea18, 0xc265d91620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160fea18) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131270 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2144377a8, 0xc2134ca240, 0xc2633763c0, 0xc263376420) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 12740 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21966c6f0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21966c6f0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21966c690, 0xc226126000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc218698d30) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc216266ef0, 0xc226126000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc216266ef0, 0xc226126000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc216266ef0, 0xc226126000, 0xfa000, 0xfa000, 0xc216250000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2154a9920, 0x7f5d6b8a1528, 0xc216266ef0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216266ef0, 0xc21625da40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 13762 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21e709b80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc258644eb0, 0x7f5d6ba4b5a8, 0xc215fffa48, 0xc261252de0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc258644eb0, 0x7f5d6ba4b5a8, 0xc215fffa48, 0xc261252de0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215fffa48) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 74805 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20eb43480, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20eb43480, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20eb43420, 0xc235c78000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25102c418) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214f1e6d0, 0xc235c78000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214f1e6d0, 0xc235c78000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214f1e6d0, 0xc235c78000, 0xfa000, 0xfa000, 0xc216220000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc216202870, 0x7f5d6b8a1528, 0xc214f1e6d0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1e6d0, 0xc2162253e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9130847 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2b7d0, 0xc21487ef60, 0xc238d1ce40, 0xc238d1cea0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 12739 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2156581b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2156581b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc215658150, 0xc22602c000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25102ca18) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20b045268, 0xc22602c000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20b045268, 0xc22602c000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20b045268, 0xc22602c000, 0xfa000, 0xfa000, 0xc216250000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2154a9770, 0x7f5d6b8a1528, 0xc20b045268, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b045268, 0xc21625d8c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 11668 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc215bcba40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265d6ccd0, 0x7f5d6ba4b5a8, 0xc21632a260, 0xc2611e6420) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265d6ccd0, 0x7f5d6ba4b5a8, 0xc21632a260, 0xc2611e6420) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21632a260) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 49243 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2421d9900, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2551e6eb0, 0x7f5d6ba4b5a8, 0xc215e324f8, 0xc2150a6600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2551e6eb0, 0x7f5d6ba4b5a8, 0xc215e324f8, 0xc2150a6600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e324f8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131187 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2110561d8, 0xc20bb90f00, 0xc240bba180, 0xc240bba1e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32170 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c29f090, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c29f090, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c29f030, 0xc2278f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ac8b988) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e413f8, 0xc2278f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e413f8, 0xc2278f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e413f8, 0xc2278f0000, 0xfa000, 0xfa000, 0xc216180000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214f20c60, 0x7f5d6b8a1528, 0xc215e413f8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e413f8, 0xc216184300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 21907 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b9c3640, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b9c3640, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b9c35e0, 0xc223400000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ac8b4a8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21fac1640, 0xc223400000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21fac1640, 0xc223400000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21fac1640, 0xc223400000, 0xfa000, 0xfa000, 0xc214870000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2137dc5d0, 0x7f5d6b8a1528, 0xc21fac1640, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21fac1640, 0xc21487ef00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 21609 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bd076b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bd076b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bd07650, 0xc223112000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2167a9278) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215edacb0, 0xc223112000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215edacb0, 0xc223112000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215edacb0, 0xc223112000, 0xfa000, 0xfa000, 0xc216400000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2139a2d50, 0x7f5d6b8a1528, 0xc215edacb0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215edacb0, 0xc216403080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 51027 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21510fe90, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21510fe90, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21510fe30, 0xc22e80c000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25102c688) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214b73c98, 0xc22e80c000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214b73c98, 0xc22e80c000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214b73c98, 0xc22e80c000, 0xfa000, 0xfa000, 0xc2131e0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20eafc5d0, 0x7f5d6b8a1528, 0xc214b73c98, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b73c98, 0xc2131e4660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 20191 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bd07720, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bd07720, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bd076c0, 0xc223306000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2186990c8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215edbb50, 0xc223306000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215edbb50, 0xc223306000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215edbb50, 0xc223306000, 0xfa000, 0xfa000, 0xc215440000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc213b03560, 0x7f5d6b8a1528, 0xc215edbb50, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215edbb50, 0xc21544a300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 51008 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc22b3ea990, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc22b3ea990, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc22b3ea930, 0xc22f9f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2186987a8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2139cf280, 0xc22f9f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2139cf280, 0xc22f9f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2139cf280, 0xc22f9f0000, 0xfa000, 0xfa000, 0xc20cb30000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20b4cfad0, 0x7f5d6b8a1528, 0xc2139cf280, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139cf280, 0xc20cb31140) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 20123 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21c4b5040, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d5f2410, 0x7f5d6ba4b5a8, 0xc215a2b7d0, 0xc238d1ce40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d5f2410, 0x7f5d6ba4b5a8, 0xc215a2b7d0, 0xc238d1ce40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2b7d0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 12760 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2170d2220, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2170d2220, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2170d21c0, 0xc225cda000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bcc6e18) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215d0d7d8, 0xc225cda000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215d0d7d8, 0xc225cda000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215d0d7d8, 0xc225cda000, 0xfa000, 0xfa000, 0xc215fc0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2144e5320, 0x7f5d6b8a1528, 0xc215d0d7d8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215d0d7d8, 0xc215fc41e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131115 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212f85ca0, 0xc217f47c80, 0xc258bce840, 0xc258bce8a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131282 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4e9e8, 0xc21b4c6a20, 0xc238d1b4a0, 0xc238d1b500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130953 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21533dc58, 0xc2162251a0, 0xc238d1aa80, 0xc238d1aae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131012 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139cf2a8, 0xc20ed39d40, 0xc261252060, 0xc2612520c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 25020 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23dccba40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2560a8910, 0x7f5d6ba4b5a8, 0xc20af1dde0, 0xc25a450180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2560a8910, 0x7f5d6ba4b5a8, 0xc20af1dde0, 0xc25a450180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af1dde0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 21801 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21ff7cdc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23f2ceff0, 0x7f5d6ba4b5a8, 0xc20bb593a8, 0xc264aeaa80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23f2ceff0, 0x7f5d6ba4b5a8, 0xc20bb593a8, 0xc264aeaa80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bb593a8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 26560 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20af1bdb0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af1bdb0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af1bd50, 0xc221720000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240d6a630) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20b0456c8, 0xc221720000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20b0456c8, 0xc221720000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20b0456c8, 0xc221720000, 0xfa000, 0xfa000, 0xc2136b0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ce117d0, 0x7f5d6b8a1528, 0xc20b0456c8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b0456c8, 0xc2136b72c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 37338 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21966c610, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21966c610, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21966c5b0, 0xc22b2f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23d886e08) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21419b0b8, 0xc22b2f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21419b0b8, 0xc22b2f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21419b0b8, 0xc22b2f0000, 0xfa000, 0xfa000, 0xc20b8f0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20b0eeff0, 0x7f5d6b8a1528, 0xc21419b0b8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21419b0b8, 0xc20b8fc120) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 20192 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc216f2c640, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24a3a8fa0, 0x7f5d6ba4b5a8, 0xc21fac1640, 0xc261250600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24a3a8fa0, 0x7f5d6ba4b5a8, 0xc21fac1640, 0xc261250600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21fac1640) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 70682 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc211742e60, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc211742e60, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc211742e00, 0xc2330f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21e6e87b0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc211f10378, 0xc2330f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc211f10378, 0xc2330f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc211f10378, 0xc2330f0000, 0xfa000, 0xfa000, 0xc21ec90000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21ee77620, 0x7f5d6b8a1528, 0xc211f10378, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211f10378, 0xc21ec91680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 10009 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c29e760, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c29e760, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c29e700, 0xc2258f2000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2422e8158) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21632bfd0, 0xc2258f2000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21632bfd0, 0xc2258f2000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21632bfd0, 0xc2258f2000, 0xfa000, 0xfa000, 0xc215e70000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214f7f290, 0x7f5d6b8a1528, 0xc21632bfd0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21632bfd0, 0xc215e76300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9130863 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2a660, 0xc2119b2e40, 0xc2611e60c0, 0xc2611e6120) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131148 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210937968, 0xc20d603440, 0xc23cc70b40, 0xc23cc70ba0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 12755 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2328ed2c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc241c0c000, 0x7f5d6ba4b5a8, 0xc20b045268, 0xc20cfedb00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc241c0c000, 0x7f5d6ba4b5a8, 0xc20b045268, 0xc20cfedb00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b045268) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 30879 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc211822a00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25596a820, 0x7f5d6ba4b5a8, 0xc21ff39828, 0xc238d1b380) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25596a820, 0x7f5d6ba4b5a8, 0xc21ff39828, 0xc238d1b380) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff39828) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 72540 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c2b06f0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c2b06f0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c2b0690, 0xc2355a2000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24f472bc8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215fff568, 0xc2355a2000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215fff568, 0xc2355a2000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215fff568, 0xc2355a2000, 0xfa000, 0xfa000, 0xc20c5f0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21e6c6630, 0x7f5d6b8a1528, 0xc215fff568, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215fff568, 0xc20c5f9800) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131119 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212bbfeb0, 0xc210b78000, 0xc258bcecc0, 0xc258bced20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 70333 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21418e3e0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21418e3e0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21418e380, 0xc2333f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211da1278) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2131d17a0, 0xc2333f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2131d17a0, 0xc2333f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2131d17a0, 0xc2333f0000, 0xfa000, 0xfa000, 0xc21f770000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2117705a0, 0x7f5d6b8a1528, 0xc2131d17a0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2131d17a0, 0xc21f773c20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 47628 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc25de36000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265d6d090, 0x7f5d6ba4b5a8, 0xc215e402a8, 0xc2611e68a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265d6d090, 0x7f5d6ba4b5a8, 0xc215e402a8, 0xc2611e68a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e402a8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 27016 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc244c48780, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc212153400, 0x7f5d6ba4b5a8, 0xc216170148, 0xc21758b7a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc212153400, 0x7f5d6ba4b5a8, 0xc216170148, 0xc21758b7a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216170148) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 70808 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2131637c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d5f21e0, 0x7f5d6ba4b5a8, 0xc21317f2f8, 0xc263f37080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d5f21e0, 0x7f5d6ba4b5a8, 0xc21317f2f8, 0xc263f37080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21317f2f8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 45551 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bf9b020, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bf9b020, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bf9afc0, 0xc22cdae000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc256ce46b8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213ffb668, 0xc22cdae000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213ffb668, 0xc22cdae000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213ffb668, 0xc22cdae000, 0xfa000, 0xfa000, 0xc216190000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc212fd7ec0, 0x7f5d6b8a1528, 0xc213ffb668, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffb668, 0xc21619a420) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 18658 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc25de36280, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265d6d270, 0x7f5d6ba4b5a8, 0xc21622f740, 0xc2611e6ae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265d6d270, 0x7f5d6ba4b5a8, 0xc21622f740, 0xc2611e6ae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622f740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32298 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241a6f040, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24e2d26e0, 0x7f5d6ba4b5a8, 0xc214834070, 0xc263f37ce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24e2d26e0, 0x7f5d6ba4b5a8, 0xc214834070, 0xc263f37ce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214834070) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131301 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b045268, 0xc21625d8c0, 0xc20cfedb00, 0xc20cfedb60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 25117 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2080575d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2080575d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc208057570, 0xc22123e000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211dac2c0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e739b8, 0xc22123e000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e739b8, 0xc22123e000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e739b8, 0xc22123e000, 0xfa000, 0xfa000, 0xc213b20000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20d180de0, 0x7f5d6b8a1528, 0xc215e739b8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e739b8, 0xc213b205a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9129886 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215c5ccb8, 0xc20c0520c0, 0xc263f37500, 0xc263f37560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32279 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b93d170, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b93d170, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b93d110, 0xc2293f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23d5d40f0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214b765c8, 0xc2293f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214b765c8, 0xc2293f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214b765c8, 0xc2293f0000, 0xfa000, 0xfa000, 0xc2149a0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2142ad3e0, 0x7f5d6b8a1528, 0xc214b765c8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b765c8, 0xc2149a04e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 47362 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20af44680, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af44680, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af44620, 0xc22cac0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211da0df8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213efb4b0, 0xc22cac0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213efb4b0, 0xc22cac0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213efb4b0, 0xc22cac0000, 0xfa000, 0xfa000, 0xc216240000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ca13530, 0x7f5d6b8a1528, 0xc213efb4b0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213efb4b0, 0xc216246e40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 16830 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23dccb680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2560a8640, 0x7f5d6ba4b5a8, 0xc215e842a0, 0xc265d91e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2560a8640, 0x7f5d6ba4b5a8, 0xc215e842a0, 0xc265d91e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e842a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 93007 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc262d84280, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2628a2640, 0x7f5d6ba4b5a8, 0xc2138c10d0, 0xc24759e3c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2628a2640, 0x7f5d6ba4b5a8, 0xc2138c10d0, 0xc24759e3c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2138c10d0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 12727 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b93cd80, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b93cd80, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b93cd20, 0xc2253e0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211dac890) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215cd2ac0, 0xc2253e0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215cd2ac0, 0xc2253e0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215cd2ac0, 0xc2253e0000, 0xfa000, 0xfa000, 0xc215fd0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214263470, 0x7f5d6b8a1528, 0xc215cd2ac0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd2ac0, 0xc215fd58c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 30528 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2497343c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2632b1db0, 0x7f5d6ba4b5a8, 0xc2160fefc8, 0xc260845560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2632b1db0, 0x7f5d6ba4b5a8, 0xc2160fefc8, 0xc260845560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160fefc8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50966 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b26e6f0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b26e6f0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b26e690, 0xc2304f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24f472968) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213f0f9c8, 0xc2304f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213f0f9c8, 0xc2304f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213f0f9c8, 0xc2304f0000, 0xfa000, 0xfa000, 0xc20af50000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc227aef1a0, 0x7f5d6b8a1528, 0xc213f0f9c8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213f0f9c8, 0xc20af5a120) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 30985 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc217384a00, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc217384a00, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2173849a0, 0xc2273f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21fede0e8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2160c4048, 0xc2273f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2160c4048, 0xc2273f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2160c4048, 0xc2273f0000, 0xfa000, 0xfa000, 0xc216340000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2162034d0, 0x7f5d6b8a1528, 0xc2160c4048, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160c4048, 0xc21634ccc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 83123 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bcead10, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bcead10, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bceacb0, 0xc236f06000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24e412888) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214426d00, 0xc236f06000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214426d00, 0xc236f06000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214426d00, 0xc236f06000, 0xfa000, 0xfa000, 0xc215f40000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20aef5b90, 0x7f5d6b8a1528, 0xc214426d00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214426d00, 0xc215f46ba0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 84609 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23c815b80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24ee6d5e0, 0x7f5d6ba4b5a8, 0xc20af4ffd0, 0xc218a334a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24ee6d5e0, 0x7f5d6ba4b5a8, 0xc20af4ffd0, 0xc218a334a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4ffd0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 10076 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc211c7b100, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc211c7b100, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc211c7b0a0, 0xc223ce4000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc213d1c1a8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215419c18, 0xc223ce4000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215419c18, 0xc223ce4000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215419c18, 0xc223ce4000, 0xfa000, 0xfa000, 0xc2161e0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21486ddd0, 0x7f5d6b8a1528, 0xc215419c18, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215419c18, 0xc2161e3860) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 12763 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2170d2300, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2170d2300, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2170d22a0, 0xc224d0a000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2557b3a58) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215d0d868, 0xc224d0a000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215d0d868, 0xc224d0a000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215d0d868, 0xc224d0a000, 0xfa000, 0xfa000, 0xc215fc0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21480e4e0, 0x7f5d6b8a1528, 0xc215d0d868, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215d0d868, 0xc215fc4cc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 11280 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b93c920, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b93c920, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b93c8c0, 0xc220632000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc212a6a4c8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2160fea18, 0xc220632000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2160fea18, 0xc220632000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2160fea18, 0xc220632000, 0xfa000, 0xfa000, 0xc215fc0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214f20990, 0x7f5d6b8a1528, 0xc2160fea18, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160fea18, 0xc215fccc60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 11277 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2170d20d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2170d20d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2170d2070, 0xc226a50000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211d64ce8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc216027480, 0xc226a50000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc216027480, 0xc226a50000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc216027480, 0xc226a50000, 0xfa000, 0xfa000, 0xc215fc0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214d1a870, 0x7f5d6b8a1528, 0xc216027480, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216027480, 0xc215fcc7e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 18529 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23b1983c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc264376280, 0x7f5d6ba4b5a8, 0xc21622f160, 0xc258bce2a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc264376280, 0x7f5d6ba4b5a8, 0xc21622f160, 0xc258bce2a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622f160) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 49331 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b9c3db0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b9c3db0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b9c3d50, 0xc22d678000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24458abe8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e32f20, 0xc22d678000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e32f20, 0xc22d678000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e32f20, 0xc22d678000, 0xfa000, 0xfa000, 0xc20b040000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214c8fce0, 0x7f5d6b8a1528, 0xc215e32f20, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e32f20, 0xc20b041980) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131021 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215d0d668, 0xc2132ffec0, 0xc261252a80, 0xc261252ae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 11276 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21966c290, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21966c290, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21966c230, 0xc226956000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2520369c8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc216267ed8, 0xc226956000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc216267ed8, 0xc226956000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc216267ed8, 0xc226956000, 0xfa000, 0xfa000, 0xc2162b0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214d1a780, 0x7f5d6b8a1528, 0xc216267ed8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216267ed8, 0xc2162bde00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131078 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b43560, 0xc216181740, 0xc21c04d1a0, 0xc21c04d200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 11278 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23dccb900, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2560a8820, 0x7f5d6ba4b5a8, 0xc216266310, 0xc25a450060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2560a8820, 0x7f5d6ba4b5a8, 0xc216266310, 0xc25a450060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216266310) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 30245 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21ff7c280, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23f2ced20, 0x7f5d6ba4b5a8, 0xc214c77e00, 0xc264aea720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23f2ced20, 0x7f5d6ba4b5a8, 0xc214c77e00, 0xc264aea720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214c77e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131175 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215631b50, 0xc20bb009c0, 0xc242581f80, 0xc26243e000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 64236 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2118232c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25596acd0, 0x7f5d6ba4b5a8, 0xc21324a1e0, 0xc238d1b920) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25596acd0, 0x7f5d6ba4b5a8, 0xc21324a1e0, 0xc238d1b920) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21324a1e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 45546 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2630e6000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc218a3bc20, 0x7f5d6ba4b5a8, 0xc2139cec00, 0xc238f47680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc218a3bc20, 0x7f5d6ba4b5a8, 0xc2139cec00, 0xc238f47680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139cec00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9129002 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b045bd8, 0xc2142a95c0, 0xc20cfecea0, 0xc20cfecf00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 69035 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c29ea70, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c29ea70, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c29ea10, 0xc226220000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25294c110) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21ff38a68, 0xc226220000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21ff38a68, 0xc226220000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21ff38a68, 0xc226220000, 0xfa000, 0xfa000, 0xc21d810000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc213ae27b0, 0x7f5d6b8a1528, 0xc21ff38a68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff38a68, 0xc21d818840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72569 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d25d040, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2551e74a0, 0x7f5d6ba4b5a8, 0xc21622f258, 0xc240bba2a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2551e74a0, 0x7f5d6ba4b5a8, 0xc21622f258, 0xc240bba2a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622f258) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 74763 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f035680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc259030870, 0x7f5d6ba4b5a8, 0xc214a75738, 0xc264625860) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc259030870, 0x7f5d6ba4b5a8, 0xc214a75738, 0xc264625860) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a75738) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 72517 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23aabd400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265651ea0, 0x7f5d6ba4b5a8, 0xc210b73e50, 0xc23cc70000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265651ea0, 0x7f5d6ba4b5a8, 0xc210b73e50, 0xc23cc70000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210b73e50) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 21871 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21966cfb0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21966cfb0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21966cf50, 0xc221144000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211d52b78) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21ff38558, 0xc221144000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21ff38558, 0xc221144000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21ff38558, 0xc221144000, 0xfa000, 0xfa000, 0xc216000000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ec12ab0, 0x7f5d6b8a1528, 0xc21ff38558, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff38558, 0xc216007020) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72455 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21816efb0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21816efb0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21816ef50, 0xc2338f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bcc7f08) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc211a48640, 0xc2338f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc211a48640, 0xc2338f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc211a48640, 0xc2338f0000, 0xfa000, 0xfa000, 0xc20d600000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2338ea180, 0x7f5d6b8a1528, 0xc211a48640, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211a48640, 0xc20d602ae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 52680 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2170d3100, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2170d3100, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2170d30a0, 0xc22fef0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24b6a1d08) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af1d460, 0xc22fef0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20af1d460, 0xc22fef0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20af1d460, 0xc22fef0000, 0xfa000, 0xfa000, 0xc20b900000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e14b920, 0x7f5d6b8a1528, 0xc20af1d460, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af1d460, 0xc20b906900) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9088123 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213aac030, 0xc20f2c5560, 0xc2459e85a0, 0xc2459e8600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 20241 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc245d8e500, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d40da90, 0x7f5d6ba4b5a8, 0xc215ffe078, 0xc26945e9c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d40da90, 0x7f5d6ba4b5a8, 0xc215ffe078, 0xc26945e9c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215ffe078) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131205 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211057e28, 0xc2160212c0, 0xc25fa2a7e0, 0xc25fa2a840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131051 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214514510, 0xc213398f60, 0xc261250a80, 0xc261250ae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 43953 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc211c7a4c0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc211c7a4c0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc211c7a460, 0xc22bcf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bcc6cd0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214321070, 0xc22bcf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214321070, 0xc22bcf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214321070, 0xc22bcf0000, 0xfa000, 0xfa000, 0xc212cb0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20d8f31d0, 0x7f5d6b8a1528, 0xc214321070, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214321070, 0xc212cb0360) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 69169 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d667180, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2621e1f40, 0x7f5d6ba4b5a8, 0xc212e76938, 0xc2608469c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2621e1f40, 0x7f5d6ba4b5a8, 0xc212e76938, 0xc2608469c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212e76938) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 70204 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23dccb2c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2560a8370, 0x7f5d6ba4b5a8, 0xc210e65bf0, 0xc265d91aa0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2560a8370, 0x7f5d6ba4b5a8, 0xc210e65bf0, 0xc265d91aa0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210e65bf0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32280 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b93d5d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b93d5d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b93d570, 0xc2299f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc244456fd8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214b77058, 0xc2299f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214b77058, 0xc2299f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214b77058, 0xc2299f0000, 0xfa000, 0xfa000, 0xc214550000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214032bd0, 0x7f5d6b8a1528, 0xc214b77058, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b77058, 0xc214556c00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32168 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b4f67d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b4f67d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b4f6770, 0xc2274f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211d52d58) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc216170598, 0xc2274f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc216170598, 0xc2274f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc216170598, 0xc2274f0000, 0xfa000, 0xfa000, 0xc216180000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214e3a420, 0x7f5d6b8a1528, 0xc216170598, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216170598, 0xc216184240) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 12742 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc25de36500, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265d6d450, 0x7f5d6ba4b5a8, 0xc2160fe938, 0xc2611e6d20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265d6d450, 0x7f5d6ba4b5a8, 0xc2160fe938, 0xc2611e6d20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160fe938) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 83112 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20bbab540, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2448131d0, 0x7f5d6ba4b5a8, 0xc214427700, 0xc25fa6b740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2448131d0, 0x7f5d6ba4b5a8, 0xc214427700, 0xc25fa6b740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214427700) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 12744 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc215658290, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc215658290, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc215658230, 0xc22675e000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc244457c08) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20b0457b0, 0xc22675e000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20b0457b0, 0xc22675e000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20b0457b0, 0xc22675e000, 0xfa000, 0xfa000, 0xc215fc0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214d1ac90, 0x7f5d6b8a1528, 0xc20b0457b0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b0457b0, 0xc215fc32c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 25019 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210dfc1b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210dfc1b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210dfc150, 0xc221914000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc218698760) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20b045f58, 0xc221914000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20b045f58, 0xc221914000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20b045f58, 0xc221914000, 0xfa000, 0xfa000, 0xc213a80000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20baf05a0, 0x7f5d6b8a1528, 0xc20b045f58, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b045f58, 0xc213a86060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 12743 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b93cca0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b93cca0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b93cc40, 0xc226664000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc212a6b048) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2160fe938, 0xc226664000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2160fe938, 0xc226664000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2160fe938, 0xc226664000, 0xfa000, 0xfa000, 0xc215fc0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2154a9aa0, 0x7f5d6b8a1528, 0xc2160fe938, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160fe938, 0xc215fc30e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32164 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc213163680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2590309b0, 0x7f5d6ba4b5a8, 0xc2160c4048, 0xc263f36f60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2590309b0, 0x7f5d6ba4b5a8, 0xc2160c4048, 0xc263f36f60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160c4048) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 17015 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b138530, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b138530, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b1384d0, 0xc2203dc000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc218699258) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2080c6a00, 0xc2203dc000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2080c6a00, 0xc2203dc000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2080c6a00, 0xc2203dc000, 0xfa000, 0xfa000, 0xc21d3b0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20b338cc0, 0x7f5d6b8a1528, 0xc2080c6a00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c6a00, 0xc21d3bd200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9087975 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211f10378, 0xc21ec91680, 0xc2187fd4a0, 0xc2187fd500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130820 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622e9b0, 0xc21c195ec0, 0xc20b27f020, 0xc20b27f080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 58044 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc208056fb0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc208056fb0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc208056f50, 0xc230cf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23faf8198) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215630510, 0xc230cf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215630510, 0xc230cf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215630510, 0xc230cf0000, 0xfa000, 0xfa000, 0xc21e550000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20dc0cfc0, 0x7f5d6b8a1528, 0xc215630510, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215630510, 0xc21e55d260) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 50862 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc215bcbb80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265d6cdc0, 0x7f5d6ba4b5a8, 0xc2147a0900, 0xc2611e6540) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265d6cdc0, 0x7f5d6ba4b5a8, 0xc2147a0900, 0xc2611e6540) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2147a0900) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 17049 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f035900, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24a3a8d20, 0x7f5d6ba4b5a8, 0xc20802fc88, 0xc264625aa0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24a3a8d20, 0x7f5d6ba4b5a8, 0xc20802fc88, 0xc264625aa0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20802fc88) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 45577 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21e708c80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc258644af0, 0x7f5d6ba4b5a8, 0xc2162669c0, 0xc261252960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc258644af0, 0x7f5d6ba4b5a8, 0xc2162669c0, 0xc261252960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2162669c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 62802 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21be37cd0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21be37cd0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21be37c70, 0xc2315f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24a563c18) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2134eec68, 0xc2315f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2134eec68, 0xc2315f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2134eec68, 0xc2315f0000, 0xfa000, 0xfa000, 0xc20f560000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21ca13290, 0x7f5d6b8a1528, 0xc2134eec68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2134eec68, 0xc20f56e9c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 47243 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bf9a840, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bf9a840, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bf9a7e0, 0xc22c8cc000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21fede788) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213ffa248, 0xc22c8cc000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213ffa248, 0xc22c8cc000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213ffa248, 0xc22c8cc000, 0xfa000, 0xfa000, 0xc216210000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e5de4e0, 0x7f5d6b8a1528, 0xc213ffa248, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffa248, 0xc2162142a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 11272 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21966c060, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21966c060, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21966c000, 0xc2256d0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211d1a188) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc216266c68, 0xc2256d0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc216266c68, 0xc2256d0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc216266c68, 0xc2256d0000, 0xfa000, 0xfa000, 0xc2162b0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214d1a3c0, 0x7f5d6b8a1528, 0xc216266c68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216266c68, 0xc2162bda40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 20044 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc218650000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc251853ef0, 0x7f5d6ba4b5a8, 0xc21538e430, 0xc25fa2a900) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc251853ef0, 0x7f5d6ba4b5a8, 0xc21538e430, 0xc25fa2a900) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21538e430) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 12724 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b3360d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b3360d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b336070, 0xc22559c000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ac8b510) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2160c4058, 0xc22559c000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2160c4058, 0xc22559c000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2160c4058, 0xc22559c000, 0xfa000, 0xfa000, 0xc215fd0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc216203da0, 0x7f5d6b8a1528, 0xc2160c4058, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160c4058, 0xc215fd5500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 12725 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bcea300, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bcea300, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bcea2a0, 0xc2263fe000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ac8bbd8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e32168, 0xc2263fe000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e32168, 0xc2263fe000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e32168, 0xc2263fe000, 0xfa000, 0xfa000, 0xc215fd0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc215fd6e70, 0x7f5d6b8a1528, 0xc215e32168, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e32168, 0xc215fd5740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9130157 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21621d680, 0xc215e2e660, 0xc238f47b00, 0xc238f47b60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9087997 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214426d00, 0xc215f46ba0, 0xc253848900, 0xc253848960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 93003 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc218849040, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23f2ce4b0, 0x7f5d6ba4b5a8, 0xc2138c0670, 0xc20b27f140) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23f2ce4b0, 0x7f5d6ba4b5a8, 0xc2138c0670, 0xc20b27f140) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2138c0670) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 25187 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d6672c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc213e7aff0, 0x7f5d6ba4b5a8, 0xc215e739b8, 0xc23e1f1620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc213e7aff0, 0x7f5d6ba4b5a8, 0xc215e739b8, 0xc23e1f1620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e739b8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32312 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b4f7800, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b4f7800, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b4f77a0, 0xc2298f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc210321d38) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214b72c60, 0xc2298f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214b72c60, 0xc2298f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214b72c60, 0xc2298f0000, 0xfa000, 0xfa000, 0xc214520000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21402c540, 0x7f5d6b8a1528, 0xc214b72c60, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b72c60, 0xc21452e360) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 17031 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23aabd2c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265d6cc30, 0x7f5d6ba4b5a8, 0xc21632a3f0, 0xc265d8fec0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265d6cc30, 0x7f5d6ba4b5a8, 0xc21632a3f0, 0xc265d8fec0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21632a3f0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 51074 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21e708b40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc218a3ba90, 0x7f5d6ba4b5a8, 0xc2145e06d8, 0xc261252840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc218a3ba90, 0x7f5d6ba4b5a8, 0xc2145e06d8, 0xc261252840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2145e06d8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 23396 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20af1a990, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af1a990, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af1a930, 0xc220b68000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21ddd6058) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20bb593a8, 0xc220b68000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20bb593a8, 0xc220b68000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20bb593a8, 0xc220b68000, 0xfa000, 0xfa000, 0xc2143c0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e5ba2a0, 0x7f5d6b8a1528, 0xc20bb593a8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bb593a8, 0xc2143ca6c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9130155 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21361c148, 0xc21b4475c0, 0xc238f478c0, 0xc238f47920) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 37353 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc208069900, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc218a3b720, 0x7f5d6ba4b5a8, 0xc21621d7a8, 0xc238f47200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc218a3b720, 0x7f5d6ba4b5a8, 0xc21621d7a8, 0xc238f47200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21621d7a8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131046 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ed9f38, 0xc20dbdc900, 0xc2612504e0, 0xc261250540) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 30524 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23aabd680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2684837c0, 0x7f5d6ba4b5a8, 0xc20bb59378, 0xc23cc70240) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2684837c0, 0x7f5d6ba4b5a8, 0xc20bb59378, 0xc23cc70240) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bb59378) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131266 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f37198, 0xc213640c00, 0xc264625f20, 0xc264625f80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 51013 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2421d9a40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2661170e0, 0x7f5d6ba4b5a8, 0xc2149dee48, 0xc2150a6720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2661170e0, 0x7f5d6ba4b5a8, 0xc2149dee48, 0xc2150a6720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2149dee48) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32093 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21510e7d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21510e7d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21510e770, 0xc2291f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24efea528) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214de8478, 0xc2291f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214de8478, 0xc2291f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214de8478, 0xc2291f0000, 0xfa000, 0xfa000, 0xc214bd0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21436d260, 0x7f5d6b8a1528, 0xc214de8478, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214de8478, 0xc214bdcf60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 83109 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f034f00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24152fef0, 0x7f5d6ba4b5a8, 0xc214515f30, 0xc25a17aae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24152fef0, 0x7f5d6ba4b5a8, 0xc214515f30, 0xc25a17aae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214515f30) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 70684 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc211742ed0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc211742ed0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc211742e70, 0xc2331f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24efea270) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc211f10b90, 0xc2331f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc211f10b90, 0xc2331f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc211f10b90, 0xc2331f0000, 0xfa000, 0xfa000, 0xc21ec90000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21b5f4420, 0x7f5d6b8a1528, 0xc211f10b90, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211f10b90, 0xc21ec918c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 50942 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210f00530, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210f00530, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210f004d0, 0xc22e03c000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24458aa78) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21533d600, 0xc22e03c000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21533d600, 0xc22e03c000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21533d600, 0xc22e03c000, 0xfa000, 0xfa000, 0xc213990000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20f2df9b0, 0x7f5d6b8a1528, 0xc21533d600, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21533d600, 0xc213999ce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 92998 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a387410, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a387410, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a3873b0, 0xc2380f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21fedeb78) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213ffae78, 0xc2380f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213ffae78, 0xc2380f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213ffae78, 0xc2380f0000, 0xfa000, 0xfa000, 0xc2104f0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20cf65b00, 0x7f5d6b8a1528, 0xc213ffae78, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffae78, 0xc2104fe480) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 50855 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc212a14300, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc212a14300, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc212a142a0, 0xc22d772000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24f473018) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2154183a8, 0xc22d772000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2154183a8, 0xc22d772000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2154183a8, 0xc22d772000, 0xfa000, 0xfa000, 0xc2141d0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2135d8870, 0x7f5d6b8a1528, 0xc2154183a8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2154183a8, 0xc2141daa20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 13661 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc24f819540, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc249c03f90, 0x7f5d6ba4b5a8, 0xc215d0d868, 0xc238d1aba0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc249c03f90, 0x7f5d6ba4b5a8, 0xc215d0d868, 0xc238d1aba0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215d0d868) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 64193 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc215659b80, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc215659b80, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc215659b20, 0xc2328f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc218303258) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21367e4f0, 0xc2328f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21367e4f0, 0xc2328f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21367e4f0, 0xc2328f0000, 0xfa000, 0xfa000, 0xc21afe0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ea5a1e0, 0x7f5d6b8a1528, 0xc21367e4f0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21367e4f0, 0xc21afe9320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 26773 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc25d682500, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2590312c0, 0x7f5d6ba4b5a8, 0xc21533d350, 0xc2633764e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2590312c0, 0x7f5d6ba4b5a8, 0xc21533d350, 0xc2633764e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21533d350) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 12525 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c2b12c0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c2b12c0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c2b1260, 0xc2238e2000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2186985b0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215419d88, 0xc2238e2000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215419d88, 0xc2238e2000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215419d88, 0xc2238e2000, 0xfa000, 0xfa000, 0xc216300000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc212dc4210, 0x7f5d6b8a1528, 0xc215419d88, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215419d88, 0xc21630c2a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 20388 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bd07480, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bd07480, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bd07420, 0xc222f1e000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc256ce4338) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af4fb10, 0xc222f1e000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20af4fb10, 0xc222f1e000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20af4fb10, 0xc222f1e000, 0xfa000, 0xfa000, 0xc215450000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20eb45c80, 0x7f5d6b8a1528, 0xc20af4fb10, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4fb10, 0xc21545cb40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 70576 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc218650500, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2618122d0, 0x7f5d6ba4b5a8, 0xc2109374a0, 0xc25fa2ad80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2618122d0, 0x7f5d6ba4b5a8, 0xc2109374a0, 0xc25fa2ad80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2109374a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131082 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21538e7a8, 0xc214f815c0, 0xc21c04d620, 0xc21c04d680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 50859 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc213547800, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc213547800, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2135477a0, 0xc22dd4e000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc253927d18) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2153672c0, 0xc22dd4e000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2153672c0, 0xc22dd4e000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2153672c0, 0xc22dd4e000, 0xfa000, 0xfa000, 0xc213cb0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2135d9710, 0x7f5d6b8a1528, 0xc2153672c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2153672c0, 0xc213cbc9c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131000 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622efd8, 0xc2144402a0, 0xc23aa51e00, 0xc23aa51e60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32214 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2328ecb40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc256203a40, 0x7f5d6ba4b5a8, 0xc215366a80, 0xc20cfed440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc256203a40, 0x7f5d6ba4b5a8, 0xc215366a80, 0xc20cfed440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215366a80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 70208 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241065b80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc268483a90, 0x7f5d6ba4b5a8, 0xc210e641a8, 0xc21aab1440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc268483a90, 0x7f5d6ba4b5a8, 0xc210e641a8, 0xc21aab1440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210e641a8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9086314 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b765c8, 0xc2149a04e0, 0xc2608468a0, 0xc260846900) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 15306 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23b198c80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc264376820, 0x7f5d6ba4b5a8, 0xc215a2b330, 0xc258bce960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc264376820, 0x7f5d6ba4b5a8, 0xc215a2b330, 0xc258bce960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2b330) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32287 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23b362000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25b471ef0, 0x7f5d6ba4b5a8, 0xc21463a720, 0xc23b9d2840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25b471ef0, 0x7f5d6ba4b5a8, 0xc21463a720, 0xc23b9d2840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21463a720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131067 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff38aa0, 0xc216449e00, 0xc2150a6de0, 0xc2150a6e40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32219 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21510e610, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21510e610, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21510e5b0, 0xc228ef0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21fede320) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214de80a8, 0xc228ef0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214de80a8, 0xc228ef0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214de80a8, 0xc228ef0000, 0xfa000, 0xfa000, 0xc214c90000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214438390, 0x7f5d6b8a1528, 0xc214de80a8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214de80a8, 0xc214c9c180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131378 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21538ea28, 0xc213fa1200, 0xc2150a75c0, 0xc2150a7620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 30307 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc211822b40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25596a910, 0x7f5d6ba4b5a8, 0xc20af4e9e8, 0xc238d1b4a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25596a910, 0x7f5d6ba4b5a8, 0xc20af4e9e8, 0xc238d1b4a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4e9e8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 23593 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f415400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc259030b40, 0x7f5d6ba4b5a8, 0xc215188b88, 0xc26945e1e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc259030b40, 0x7f5d6ba4b5a8, 0xc215188b88, 0xc26945e1e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215188b88) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131142 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2153672c0, 0xc213cbc9c0, 0xc23cc70480, 0xc23cc704e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 70810 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d25cf00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2551e73b0, 0x7f5d6ba4b5a8, 0xc2110561d8, 0xc240bba180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2551e73b0, 0x7f5d6ba4b5a8, 0xc2110561d8, 0xc240bba180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2110561d8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 26905 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc25de36140, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265d6d180, 0x7f5d6ba4b5a8, 0xc214f1e158, 0xc2611e69c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265d6d180, 0x7f5d6ba4b5a8, 0xc214f1e158, 0xc2611e69c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1e158) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130542 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21437f5c0, 0xc20e7bd500, 0xc25fa2a000, 0xc25fa2a060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 51112 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20af1a610, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af1a610, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af1a5b0, 0xc2302f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc253927e98) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2142efc58, 0xc2302f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2142efc58, 0xc2302f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2142efc58, 0xc2302f0000, 0xfa000, 0xfa000, 0xc20baa0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc229fef1d0, 0x7f5d6b8a1528, 0xc2142efc58, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2142efc58, 0xc20baaeae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 11150 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241a6f7c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24e2d2140, 0x7f5d6ba4b5a8, 0xc20802f398, 0xc263f37620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24e2d2140, 0x7f5d6ba4b5a8, 0xc20802f398, 0xc263f37620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20802f398) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 73776 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f0352c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc258644500, 0x7f5d6ba4b5a8, 0xc210df0220, 0xc264625500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc258644500, 0x7f5d6ba4b5a8, 0xc210df0220, 0xc264625500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210df0220) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 54931 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bfe1e20, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bfe1e20, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bfe1dc0, 0xc230af0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2557b3d10) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2136fd060, 0xc230af0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2136fd060, 0xc230af0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2136fd060, 0xc230af0000, 0xfa000, 0xfa000, 0xc21b510000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ccbf800, 0x7f5d6b8a1528, 0xc2136fd060, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2136fd060, 0xc21b5132c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 49198 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23d417b80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2153c29b0, 0x7f5d6ba4b5a8, 0xc21632b4b0, 0xc21c04d500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2153c29b0, 0x7f5d6ba4b5a8, 0xc21632b4b0, 0xc21c04d500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21632b4b0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50962 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b9c3d40, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b9c3d40, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b9c3ce0, 0xc22d57e000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24d673558) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e32748, 0xc22d57e000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e32748, 0xc22d57e000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e32748, 0xc22d57e000, 0xfa000, 0xfa000, 0xc214ab0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214e3b860, 0x7f5d6b8a1528, 0xc215e32748, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e32748, 0xc214ab3ec0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 50949 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23b198dc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc264376910, 0x7f5d6ba4b5a8, 0xc2154183a8, 0xc258bcea80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc264376910, 0x7f5d6ba4b5a8, 0xc2154183a8, 0xc258bcea80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2154183a8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 28586 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc208057fe0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc208057fe0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc208057f80, 0xc221ef0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc212a6ad48) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214f1e158, 0xc221ef0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214f1e158, 0xc221ef0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214f1e158, 0xc221ef0000, 0xfa000, 0xfa000, 0xc213280000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc211bebbf0, 0x7f5d6b8a1528, 0xc214f1e158, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1e158, 0xc213288a80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9130896 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213efa080, 0xc20b040000, 0xc2150a6180, 0xc2150a61e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129887 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20802f398, 0xc215fead20, 0xc263f37620, 0xc263f37680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 16719 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a0444c0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a0444c0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a044460, 0xc224a1c000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25561b488) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21538e510, 0xc224a1c000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21538e510, 0xc224a1c000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21538e510, 0xc224a1c000, 0xfa000, 0xfa000, 0xc216390000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21da22030, 0x7f5d6b8a1528, 0xc21538e510, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21538e510, 0xc21639a360) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 13659 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc211c7a0d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc211c7a0d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc211c7a070, 0xc224c10000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211da00d8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215b42c20, 0xc224c10000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215b42c20, 0xc224c10000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215b42c20, 0xc224c10000, 0xfa000, 0xfa000, 0xc215eb0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2144e54d0, 0x7f5d6b8a1528, 0xc215b42c20, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b42c20, 0xc215eb1bc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 88126 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc216f2c500, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc26480c410, 0x7f5d6ba4b5a8, 0xc213ed9f38, 0xc2612504e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc26480c410, 0x7f5d6ba4b5a8, 0xc213ed9f38, 0xc2612504e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ed9f38) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131193 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215630510, 0xc21e55d260, 0xc240bba840, 0xc240bba8a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32087 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bb1e0d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bb1e0d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bb1e070, 0xc2282f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc213d1c2f8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215fff788, 0xc2282f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215fff788, 0xc2282f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215fff788, 0xc2282f0000, 0xfa000, 0xfa000, 0xc215fb0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214a713e0, 0x7f5d6b8a1528, 0xc215fff788, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215fff788, 0xc215fb2420) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 65928 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc220889e00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc226b4dc20, 0x7f5d6ba4b5a8, 0xc21367e4f0, 0xc260f8fda0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc226b4dc20, 0x7f5d6ba4b5a8, 0xc21367e4f0, 0xc260f8fda0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21367e4f0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9087976 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2132eb6d0, 0xc21469f3e0, 0xc2187fd5c0, 0xc2187fd620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 78087 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210ed3560, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210ed3560, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210ed3500, 0xc23692a000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc253927bb8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2080c6b88, 0xc23692a000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2080c6b88, 0xc23692a000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2080c6b88, 0xc23692a000, 0xfa000, 0xfa000, 0xc21a590000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2134b41b0, 0x7f5d6b8a1528, 0xc2080c6b88, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c6b88, 0xc21a5955c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 11152 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20af44450, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af44450, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af443f0, 0xc224efe000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc252036578) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215fffa48, 0xc224efe000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215fffa48, 0xc224efe000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215fffa48, 0xc224efe000, 0xfa000, 0xfa000, 0xc2162a0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214d21110, 0x7f5d6b8a1528, 0xc215fffa48, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215fffa48, 0xc2162aaae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 13714 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21d06b680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24566a870, 0x7f5d6ba4b5a8, 0xc215b42c10, 0xc25fa2a120) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24566a870, 0x7f5d6ba4b5a8, 0xc215b42c10, 0xc25fa2a120) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b42c10) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 30526 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc24f818280, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc211cfe280, 0x7f5d6ba4b5a8, 0xc2160fe1a8, 0xc23d84bc80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc211cfe280, 0x7f5d6ba4b5a8, 0xc2160fe1a8, 0xc23d84bc80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160fe1a8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 52711 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a0452c0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a0452c0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a045260, 0xc2303f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24a562f88) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e85b60, 0xc2303f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e85b60, 0xc2303f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e85b60, 0xc2303f0000, 0xfa000, 0xfa000, 0xc20b530000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20d7b9f50, 0x7f5d6b8a1528, 0xc215e85b60, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e85b60, 0xc20b535d40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 20210 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2080115d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2080115d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc208011570, 0xc222d2a000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25294c628) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e84900, 0xc222d2a000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e84900, 0xc222d2a000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e84900, 0xc222d2a000, 0xfa000, 0xfa000, 0xc214790000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214cfd230, 0x7f5d6b8a1528, 0xc215e84900, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e84900, 0xc214796d20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72576 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc215a6ca00, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc215a6ca00, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc215a6c9a0, 0xc2354a8000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2167a93d8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e84248, 0xc2354a8000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e84248, 0xc2354a8000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e84248, 0xc2354a8000, 0xfa000, 0xfa000, 0xc218c50000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21a81d470, 0x7f5d6b8a1528, 0xc215e84248, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e84248, 0xc218c52ea0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 13699 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23aabde00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc262d5c550, 0x7f5d6ba4b5a8, 0xc21632bfd0, 0xc23cc707e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc262d5c550, 0x7f5d6ba4b5a8, 0xc21632bfd0, 0xc23cc707e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21632bfd0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 13701 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc215658300, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc215658300, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2156582a0, 0xc2259ec000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211d64a48) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20b045888, 0xc2259ec000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20b045888, 0xc2259ec000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20b045888, 0xc2259ec000, 0xfa000, 0xfa000, 0xc215e90000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214f7f3b0, 0x7f5d6b8a1528, 0xc20b045888, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b045888, 0xc215e93a40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 13645 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc245500140, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc268483d60, 0x7f5d6ba4b5a8, 0xc215c5d178, 0xc21aab17a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc268483d60, 0x7f5d6ba4b5a8, 0xc215c5d178, 0xc21aab17a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215c5d178) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 13646 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21966c990, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21966c990, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21966c930, 0xc225ae6000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21606f408) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215c5d178, 0xc225ae6000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215c5d178, 0xc225ae6000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215c5d178, 0xc225ae6000, 0xfa000, 0xfa000, 0xc215eb0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214f20b70, 0x7f5d6b8a1528, 0xc215c5d178, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215c5d178, 0xc215eb3800) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 12746 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc208010a00, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc208010a00, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2080109a0, 0xc2239f6000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24f472ed8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215a2b330, 0xc2239f6000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215a2b330, 0xc2239f6000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215a2b330, 0xc2239f6000, 0xfa000, 0xfa000, 0xc215fc0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214180690, 0x7f5d6b8a1528, 0xc215a2b330, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2b330, 0xc215fc2300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 37430 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f4152c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25596a4b0, 0x7f5d6ba4b5a8, 0xc214271930, 0xc26945e0c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25596a4b0, 0x7f5d6ba4b5a8, 0xc214271930, 0xc26945e0c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214271930) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32174 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc218849b80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc218a3b9f0, 0x7f5d6ba4b5a8, 0xc215a05e78, 0xc20b27f6e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc218a3b9f0, 0x7f5d6ba4b5a8, 0xc215a05e78, 0xc20b27f6e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a05e78) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32330 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc213547410, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc213547410, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2135473b0, 0xc22a3f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24d673700) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2145f4d70, 0xc22a3f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2145f4d70, 0xc22a3f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2145f4d70, 0xc22a3f0000, 0xfa000, 0xfa000, 0xc213470000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21320c9c0, 0x7f5d6b8a1528, 0xc2145f4d70, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2145f4d70, 0xc21347a000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 70200 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21de34290, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21de34290, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21de34230, 0xc2335f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211da0d08) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21317f2f8, 0xc2335f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21317f2f8, 0xc2335f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21317f2f8, 0xc2335f0000, 0xfa000, 0xfa000, 0xc216010000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20fe6b920, 0x7f5d6b8a1528, 0xc21317f2f8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21317f2f8, 0xc2160104e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 25188 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f035cc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc259030d20, 0x7f5d6ba4b5a8, 0xc215e73b10, 0xc264625e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc259030d20, 0x7f5d6ba4b5a8, 0xc215e73b10, 0xc264625e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e73b10) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131203 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21437eb68, 0xc213421860, 0xc25fa2a5a0, 0xc25fa2a600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 49254 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21bdab400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc258644780, 0x7f5d6ba4b5a8, 0xc215eda5e0, 0xc2425818c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc258644780, 0x7f5d6ba4b5a8, 0xc215eda5e0, 0xc2425818c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215eda5e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 64188 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2630e6280, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc218a3be00, 0x7f5d6ba4b5a8, 0xc21361c148, 0xc238f478c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc218a3be00, 0x7f5d6ba4b5a8, 0xc21361c148, 0xc238f478c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21361c148) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 18584 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20af44610, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af44610, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af445b0, 0xc220a6e000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23faf8078) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215b42588, 0xc220a6e000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215b42588, 0xc220a6e000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215b42588, 0xc220a6e000, 0xfa000, 0xfa000, 0xc216180000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2131e9350, 0x7f5d6b8a1528, 0xc215b42588, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b42588, 0xc216181e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9086125 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bfa2198, 0xc20db21080, 0xc20d0dd440, 0xc20d0dd4a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131208 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215edbf68, 0xc214f99020, 0xc25fa2ab40, 0xc25fa2aba0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 42055 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20ce6a0d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20ce6a0d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20ce6a070, 0xc22c3ea000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211d64388) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21fac0578, 0xc22c3ea000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21fac0578, 0xc22c3ea000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21fac0578, 0xc22c3ea000, 0xfa000, 0xfa000, 0xc210570000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20c02c7b0, 0x7f5d6b8a1528, 0xc21fac0578, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21fac0578, 0xc210570000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72584 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23b198a00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc264376640, 0x7f5d6ba4b5a8, 0xc215fff568, 0xc258bce720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc264376640, 0x7f5d6ba4b5a8, 0xc215fff568, 0xc258bce720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215fff568) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131283 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a05568, 0xc2281ea960, 0xc238d1b5c0, 0xc238d1b620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129884 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21437e2e8, 0xc20fdb3320, 0xc263f372c0, 0xc263f37320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 64267 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23b362140, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25e36a000, 0x7f5d6ba4b5a8, 0xc21369fd40, 0xc23b9d2960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25e36a000, 0x7f5d6ba4b5a8, 0xc21369fd40, 0xc23b9d2960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21369fd40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32253 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc244c488c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2121534f0, 0x7f5d6ba4b5a8, 0xc215edb418, 0xc21758b8c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2121534f0, 0x7f5d6ba4b5a8, 0xc215edb418, 0xc21758b8c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215edb418) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 20001 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a044610, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a044610, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a0445b0, 0xc222b36000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bcc7828) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21538e430, 0xc222b36000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21538e430, 0xc222b36000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21538e430, 0xc222b36000, 0xfa000, 0xfa000, 0xc215dd0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214bec780, 0x7f5d6b8a1528, 0xc21538e430, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21538e430, 0xc215ddef00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131095 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215edaa68, 0xc215e77ce0, 0xc24759e060, 0xc24759e0c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 47246 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f0357c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc213e7b220, 0x7f5d6ba4b5a8, 0xc213ffabd0, 0xc264625980) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc213e7b220, 0x7f5d6ba4b5a8, 0xc213ffabd0, 0xc264625980) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffabd0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9088087 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212f9db60, 0xc21f7692c0, 0xc25e3c1da0, 0xc25e3c1e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 15392 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23dccb400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2560a8460, 0x7f5d6ba4b5a8, 0xc215419c18, 0xc265d91bc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2560a8460, 0x7f5d6ba4b5a8, 0xc215419c18, 0xc265d91bc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215419c18) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 72570 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23d417e00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2153c2b90, 0x7f5d6ba4b5a8, 0xc21622fc00, 0xc21c04d740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2153c2b90, 0x7f5d6ba4b5a8, 0xc21622fc00, 0xc21c04d740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622fc00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 31164 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d25cc80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2551e71d0, 0x7f5d6ba4b5a8, 0xc215e417e8, 0xc23b94df20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2551e71d0, 0x7f5d6ba4b5a8, 0xc215e417e8, 0xc23b94df20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e417e8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130822 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216266138, 0xc20c796240, 0xc20b27f260, 0xc20b27f2c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 49286 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20af456b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af456b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af45650, 0xc22cea8000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21e7cdb98) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215eda5e0, 0xc22cea8000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215eda5e0, 0xc22cea8000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215eda5e0, 0xc22cea8000, 0xfa000, 0xfa000, 0xc214c60000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2162ed380, 0x7f5d6b8a1528, 0xc215eda5e0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215eda5e0, 0xc214c63b00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72582 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21bdab180, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24a3a8e10, 0x7f5d6ba4b5a8, 0xc21ff617f0, 0xc242581680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24a3a8e10, 0x7f5d6ba4b5a8, 0xc21ff617f0, 0xc242581680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff617f0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 70199 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21816eed0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21816eed0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21816ee70, 0xc2334f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23d886588) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2130a2f68, 0xc2334f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2130a2f68, 0xc2334f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2130a2f68, 0xc2334f0000, 0xfa000, 0xfa000, 0xc216010000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc213b7b380, 0x7f5d6b8a1528, 0xc2130a2f68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2130a2f68, 0xc216010480) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131284 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139ce550, 0xc213ff1a40, 0xc238d1b6e0, 0xc238d1b740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 74685 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21bdab680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24a3a9130, 0x7f5d6ba4b5a8, 0xc214a08128, 0xc242581b00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24a3a9130, 0x7f5d6ba4b5a8, 0xc214a08128, 0xc242581b00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a08128) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 47248 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bf9ad80, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bf9ad80, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bf9ad20, 0xc22ccb4000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc212a6bee8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213ffb1f0, 0xc22ccb4000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213ffb1f0, 0xc22ccb4000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213ffb1f0, 0xc22ccb4000, 0xfa000, 0xfa000, 0xc216210000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20c196ed0, 0x7f5d6b8a1528, 0xc213ffb1f0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffb1f0, 0xc2162145a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 59535 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21f69abc0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21f69abc0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21f69ab60, 0xc2311f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21fede8c8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213657010, 0xc2311f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213657010, 0xc2311f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213657010, 0xc2311f0000, 0xfa000, 0xfa000, 0xc20def0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214c84cc0, 0x7f5d6b8a1528, 0xc213657010, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213657010, 0xc20def0720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131225 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214de9480, 0xc216142a20, 0xc2611e6c00, 0xc2611e6c60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 51109 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc22b3ead80, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc22b3ead80, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc22b3ead20, 0xc22fdf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc249b70118) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2139a5a40, 0xc22fdf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2139a5a40, 0xc22fdf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2139a5a40, 0xc22fdf0000, 0xfa000, 0xfa000, 0xc20e9e0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20b100420, 0x7f5d6b8a1528, 0xc2139a5a40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139a5a40, 0xc20e9ed560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131113 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21632b828, 0xc215498000, 0xc258bce600, 0xc258bce660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32215 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bb1e610, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bb1e610, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bb1e5b0, 0xc2288f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc244456078) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214f363d8, 0xc2288f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214f363d8, 0xc2288f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214f363d8, 0xc2288f0000, 0xfa000, 0xfa000, 0xc215ec0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2146591d0, 0x7f5d6b8a1528, 0xc214f363d8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f363d8, 0xc215ec1f80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9130256 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4eab8, 0xc213a871a0, 0xc21c04cae0, 0xc21c04cb40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 23426 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a045250, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a045250, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a0451f0, 0xc220c62000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21e7cdde0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215ffee78, 0xc220c62000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215ffee78, 0xc220c62000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215ffee78, 0xc220c62000, 0xfa000, 0xfa000, 0xc214350000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e5ba570, 0x7f5d6b8a1528, 0xc215ffee78, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215ffee78, 0xc214350840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 40680 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b26e4c0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b26e4c0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b26e460, 0xc22b8f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23d5d4138) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213aac9a8, 0xc22b8f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213aac9a8, 0xc22b8f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213aac9a8, 0xc22b8f0000, 0xfa000, 0xfa000, 0xc2118e0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc215e422a0, 0x7f5d6b8a1528, 0xc213aac9a8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213aac9a8, 0xc2118e8720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 51065 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20bbaba40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2586442d0, 0x7f5d6ba4b5a8, 0xc2139cf2a8, 0xc261252060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2586442d0, 0x7f5d6ba4b5a8, 0xc2139cf2a8, 0xc261252060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139cf2a8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 79731 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210ed3720, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210ed3720, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210ed36c0, 0xc236b1e000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211d64b78) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2080c7708, 0xc236b1e000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2080c7708, 0xc236b1e000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2080c7708, 0xc236b1e000, 0xfa000, 0xfa000, 0xc20bcd0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2136314a0, 0x7f5d6b8a1528, 0xc2080c7708, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c7708, 0xc20bcded20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131161 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21406b1a8, 0xc213292000, 0xc264aea960, 0xc264aea9c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 15390 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c29ed10, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c29ed10, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c29ecb0, 0xc223bea000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23d8870e8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21538e7a8, 0xc223bea000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21538e7a8, 0xc223bea000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21538e7a8, 0xc223bea000, 0xfa000, 0xfa000, 0xc214f80000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2142ad6e0, 0x7f5d6b8a1528, 0xc21538e7a8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21538e7a8, 0xc214f815c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 49336 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219b0f540, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc266117b30, 0x7f5d6ba4b5a8, 0xc21538ea28, 0xc2150a75c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc266117b30, 0x7f5d6ba4b5a8, 0xc21538ea28, 0xc2150a75c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21538ea28) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 43970 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210f01480, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210f01480, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210f01420, 0xc22bff0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ef50a38) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2139ce550, 0xc22bff0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2139ce550, 0xc22bff0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2139ce550, 0xc22bff0000, 0xfa000, 0xfa000, 0xc213ff0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20f12a990, 0x7f5d6b8a1528, 0xc2139ce550, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139ce550, 0xc213ff1a40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9130961 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff39828, 0xc21d3fbb00, 0xc238d1b380, 0xc238d1b3e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131202 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2a658, 0xc2118e8600, 0xc25fa2a480, 0xc25fa2a4e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129264 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214c773b0, 0xc2131d5740, 0xc2425817a0, 0xc242581800) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 50955 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d25d540, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2551e7b30, 0x7f5d6ba4b5a8, 0xc214c762d0, 0xc240bbaa80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2551e7b30, 0x7f5d6ba4b5a8, 0xc214c762d0, 0xc240bbaa80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214c762d0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9129009 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21621d058, 0xc20f9dbf20, 0xc20cfed680, 0xc20cfed6e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32308 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23dccb180, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2560a8280, 0x7f5d6ba4b5a8, 0xc214a17048, 0xc265d91980) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2560a8280, 0x7f5d6ba4b5a8, 0xc214a17048, 0xc265d91980) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a17048) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 51030 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21966c530, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21966c530, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21966c4d0, 0xc22eafa000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2444562c0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2149dee48, 0xc22eafa000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2149dee48, 0xc22eafa000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2149dee48, 0xc22eafa000, 0xfa000, 0xfa000, 0xc212470000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20f134990, 0x7f5d6b8a1528, 0xc2149dee48, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2149dee48, 0xc212478fc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 64235 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b26fb10, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b26fb10, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b26fab0, 0xc231bf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ac8be28) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2138c1f68, 0xc231bf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2138c1f68, 0xc231bf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2138c1f68, 0xc231bf0000, 0xfa000, 0xfa000, 0xc212970000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21fb1a4e0, 0x7f5d6b8a1528, 0xc2138c1f68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2138c1f68, 0xc21297c540) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 67446 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21816e840, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21816e840, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21816e7e0, 0xc2325f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc218699388) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2130a3620, 0xc2325f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2130a3620, 0xc2325f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2130a3620, 0xc2325f0000, 0xfa000, 0xfa000, 0xc21b440000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21431bbf0, 0x7f5d6b8a1528, 0xc2130a3620, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2130a3620, 0xc21b447620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 88120 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b9c3170, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b9c3170, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b9c3110, 0xc237af0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21e7cda98) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2141defc0, 0xc237af0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2141defc0, 0xc237af0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2141defc0, 0xc237af0000, 0xfa000, 0xfa000, 0xc213190000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ff41680, 0x7f5d6b8a1528, 0xc2141defc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2141defc0, 0xc213191f20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131286 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21324a1e0, 0xc21297c660, 0xc238d1b920, 0xc238d1b980) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130544 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215c5ce70, 0xc213fd80c0, 0xc25fa2a240, 0xc25fa2a2a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 42057 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241a6f680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24e2d2050, 0x7f5d6ba4b5a8, 0xc215c5ccb8, 0xc263f37500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24e2d2050, 0x7f5d6ba4b5a8, 0xc215c5ccb8, 0xc263f37500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215c5ccb8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 30306 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b9c26f0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b9c26f0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b9c2690, 0xc2229ae000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24d673cb8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214c77e00, 0xc2229ae000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214c77e00, 0xc2229ae000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214c77e00, 0xc2229ae000, 0xfa000, 0xfa000, 0xc214320000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc212fd7140, 0x7f5d6b8a1528, 0xc214c77e00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214c77e00, 0xc214323380) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 70739 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20bbab400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23fa332c0, 0x7f5d6ba4b5a8, 0xc211f10b90, 0xc25fa6b620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23fa332c0, 0x7f5d6ba4b5a8, 0xc211f10b90, 0xc25fa6b620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211f10b90) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 55990 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23aabdcc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc262d5c460, 0x7f5d6ba4b5a8, 0xc2080c6cd8, 0xc23cc706c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc262d5c460, 0x7f5d6ba4b5a8, 0xc2080c6cd8, 0xc23cc706c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c6cd8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 16865 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bd06ca0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bd06ca0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bd06c40, 0xc222c30000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc213d1c548) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af4f7b8, 0xc222c30000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20af4f7b8, 0xc222c30000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20af4f7b8, 0xc222c30000, 0xfa000, 0xfa000, 0xc214790000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214bec8d0, 0x7f5d6b8a1528, 0xc20af4f7b8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4f7b8, 0xc214796ae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 50875 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a0fb330, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a0fb330, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a0fb2d0, 0xc22dc54000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc244457458) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21538ea28, 0xc22dc54000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21538ea28, 0xc22dc54000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21538ea28, 0xc22dc54000, 0xfa000, 0xfa000, 0xc213fa0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20f67d0e0, 0x7f5d6b8a1528, 0xc21538ea28, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21538ea28, 0xc213fa1200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 51029 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc215bcbcc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265d6ceb0, 0x7f5d6ba4b5a8, 0xc214a16db0, 0xc2611e6660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265d6ceb0, 0x7f5d6ba4b5a8, 0xc214a16db0, 0xc2611e6660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a16db0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 52677 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2421d92c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2168f8f50, 0x7f5d6ba4b5a8, 0xc20af1c758, 0xc25e3c1c80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2168f8f50, 0x7f5d6ba4b5a8, 0xc20af1c758, 0xc25e3c1c80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af1c758) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 52679 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc24f818140, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc213214c80, 0x7f5d6ba4b5a8, 0xc20af1d460, 0xc23d84bb60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc213214c80, 0x7f5d6ba4b5a8, 0xc20af1d460, 0xc23d84bb60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af1d460) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 72530 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21de34920, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21de34920, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21de348c0, 0xc2343f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2186986d8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc210b720d0, 0xc2343f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc210b720d0, 0xc2343f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc210b720d0, 0xc2343f0000, 0xfa000, 0xfa000, 0xc216010000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc213b7bb30, 0x7f5d6b8a1528, 0xc210b720d0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210b720d0, 0xc216010780) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 16623 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b1380d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b1380d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b138070, 0xc223ed8000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2167a8d70) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2080c6000, 0xc223ed8000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2080c6000, 0xc223ed8000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2080c6000, 0xc223ed8000, 0xfa000, 0xfa000, 0xc214a40000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2136d12c0, 0x7f5d6b8a1528, 0xc2080c6000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c6000, 0xc214a49c80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 67305 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21c4b5180, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d5f2500, 0x7f5d6ba4b5a8, 0xc213574b38, 0xc238d1cf60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d5f2500, 0x7f5d6ba4b5a8, 0xc213574b38, 0xc238d1cf60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213574b38) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 40740 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23dcca8c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2105c7c70, 0x7f5d6ba4b5a8, 0xc213aac030, 0xc2459e85a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2105c7c70, 0x7f5d6ba4b5a8, 0xc213aac030, 0xc2459e85a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213aac030) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50984 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23aabda40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc262d5c280, 0x7f5d6ba4b5a8, 0xc2153672c0, 0xc23cc70480) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc262d5c280, 0x7f5d6ba4b5a8, 0xc2153672c0, 0xc23cc70480) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2153672c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 43959 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc215bcb2c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265d6c730, 0x7f5d6ba4b5a8, 0xc214321c68, 0xc20b281d40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265d6c730, 0x7f5d6ba4b5a8, 0xc214321c68, 0xc20b281d40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214321c68) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131123 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215d0c980, 0xc2141dbf20, 0xc24ec8bf80, 0xc26945e000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32221 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23d417680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2562036d0, 0x7f5d6ba4b5a8, 0xc21462e898, 0xc21c04d080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2562036d0, 0x7f5d6ba4b5a8, 0xc21462e898, 0xc21c04d080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21462e898) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 67384 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc215bcb900, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25596a1e0, 0x7f5d6ba4b5a8, 0xc21ff39cb0, 0xc2611e6300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25596a1e0, 0x7f5d6ba4b5a8, 0xc21ff39cb0, 0xc2611e6300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff39cb0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 33585 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2170d2df0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2170d2df0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2170d2d90, 0xc22a9f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ef502c8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e331a8, 0xc22a9f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e331a8, 0xc22a9f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e331a8, 0xc22a9f0000, 0xfa000, 0xfa000, 0xc210f00000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20dcec9c0, 0x7f5d6b8a1528, 0xc215e331a8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e331a8, 0xc210f08480) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 60506 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc216f2cdc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc26480cb40, 0x7f5d6ba4b5a8, 0xc21621d790, 0xc261250cc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc26480cb40, 0x7f5d6ba4b5a8, 0xc21621d790, 0xc261250cc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21621d790) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 27071 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b93c450, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b93c450, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b93c3f0, 0xc2271f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23d886338) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af4eab8, 0xc2271f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20af4eab8, 0xc2271f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20af4eab8, 0xc2271f0000, 0xfa000, 0xfa000, 0xc213a80000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc215fd63f0, 0x7f5d6b8a1528, 0xc20af4eab8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4eab8, 0xc213a871a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 40665 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20af44060, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af44060, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af44000, 0xc22c2f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23b9f9c78) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213efa080, 0xc22c2f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213efa080, 0xc22c2f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213efa080, 0xc22c2f0000, 0xfa000, 0xfa000, 0xc20b040000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20b008420, 0x7f5d6b8a1528, 0xc213efa080, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213efa080, 0xc20b040000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 70722 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2421d9400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2168f9040, 0x7f5d6ba4b5a8, 0xc212f9db60, 0xc25e3c1da0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2168f9040, 0x7f5d6ba4b5a8, 0xc212f9db60, 0xc25e3c1da0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212f9db60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131139 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff396c8, 0xc2152e0f00, 0xc23cc70120, 0xc23cc70180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130827 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21fac0578, 0xc210570000, 0xc20b27f800, 0xc20b27f860) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130941 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139ce670, 0xc20ed69c20, 0xc264c39bc0, 0xc264c39c20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 23290 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc208057640, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc208057640, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2080575e0, 0xc221338000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21fede528) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e73b10, 0xc221338000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e73b10, 0xc221338000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e73b10, 0xc221338000, 0xfa000, 0xfa000, 0xc214360000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20f1353e0, 0x7f5d6b8a1528, 0xc215e73b10, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e73b10, 0xc214362f00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 67387 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc245d8e3c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d40d9a0, 0x7f5d6ba4b5a8, 0xc2130a3620, 0xc26945e8a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d40d9a0, 0x7f5d6ba4b5a8, 0xc2130a3620, 0xc26945e8a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2130a3620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 86404 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc218650140, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc261812000, 0x7f5d6ba4b5a8, 0xc215c5d060, 0xc25fa2aa20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc261812000, 0x7f5d6ba4b5a8, 0xc215c5d060, 0xc25fa2aa20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215c5d060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130796 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a75738, 0xc215e92c00, 0xc264625860, 0xc2646258c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129872 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a08ab0, 0xc21483fd40, 0xc265d8fda0, 0xc265d8fe00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130794 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4f028, 0xc21b4c6ea0, 0xc264625620, 0xc264625680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131155 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd34b8, 0xc21402e540, 0xc264aea2a0, 0xc264aea300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 70738 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc24b9dd2c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc242ae7cc0, 0x7f5d6ba4b5a8, 0xc211f10378, 0xc2187fd4a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc242ae7cc0, 0x7f5d6ba4b5a8, 0xc211f10378, 0xc2187fd4a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211f10378) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 40223 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210dfd720, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210dfd720, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210dfd6c0, 0xc22b4f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25294c268) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21621d7a8, 0xc22b4f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21621d7a8, 0xc22b4f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21621d7a8, 0xc22b4f0000, 0xfa000, 0xfa000, 0xc2120b0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20c196360, 0x7f5d6b8a1528, 0xc21621d7a8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21621d7a8, 0xc2120bd4a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 26771 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21e708000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d40d0e0, 0x7f5d6ba4b5a8, 0xc20b045f58, 0xc2612524e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d40d0e0, 0x7f5d6ba4b5a8, 0xc20b045f58, 0xc2612524e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b045f58) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32328 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20eb43bf0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20eb43bf0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20eb43b90, 0xc22a2f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21fedecc8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2144377a8, 0xc22a2f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2144377a8, 0xc22a2f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2144377a8, 0xc22a2f0000, 0xfa000, 0xfa000, 0xc2134c0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2133a9ad0, 0x7f5d6b8a1528, 0xc2144377a8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2144377a8, 0xc2134ca240) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72467 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21816f2c0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21816f2c0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21816f260, 0xc233cf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bcc7118) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc211a49bb0, 0xc233cf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc211a49bb0, 0xc233cf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc211a49bb0, 0xc233cf0000, 0xfa000, 0xfa000, 0xc21f760000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20fe6bc80, 0x7f5d6b8a1528, 0xc211a49bb0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211a49bb0, 0xc21f7666c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 18645 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20ba20ca0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20ba20ca0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20ba20c40, 0xc2244b4000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc212a6ae48) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21622f740, 0xc2244b4000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21622f740, 0xc2244b4000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21622f740, 0xc2244b4000, 0xfa000, 0xfa000, 0xc216140000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21d7258c0, 0x7f5d6b8a1528, 0xc21622f740, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622f740, 0xc216143080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32217 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc217385f00, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc217385f00, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc217385ea0, 0xc228bf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bcc7ac8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215edbf68, 0xc228bf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215edbf68, 0xc228bf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215edbf68, 0xc228bf0000, 0xfa000, 0xfa000, 0xc214f90000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21460b3b0, 0x7f5d6b8a1528, 0xc215edbf68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215edbf68, 0xc214f99020) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 18521 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc208011170, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc208011170, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc208011110, 0xc2241c6000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ac8b048) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20802fc88, 0xc2241c6000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20802fc88, 0xc2241c6000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20802fc88, 0xc2241c6000, 0xfa000, 0xfa000, 0xc215fe0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2153b7cb0, 0x7f5d6b8a1528, 0xc20802fc88, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20802fc88, 0xc215fea7e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131213 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a17c00, 0xc215e07620, 0xc25fa2b0e0, 0xc25fa2b140) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131081 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21632b4b0, 0xc216215500, 0xc21c04d500, 0xc21c04d560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 65939 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc211c7b6b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc211c7b6b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc211c7b650, 0xc231cf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24d673ef8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21406b1a8, 0xc231cf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21406b1a8, 0xc231cf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21406b1a8, 0xc231cf0000, 0xfa000, 0xfa000, 0xc213290000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21a7de330, 0x7f5d6b8a1528, 0xc21406b1a8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21406b1a8, 0xc213292000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 35427 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2170d2d10, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2170d2d10, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2170d2cb0, 0xc22a7f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211da0938) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e32460, 0xc22a7f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e32460, 0xc22a7f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e32460, 0xc22a7f0000, 0xfa000, 0xfa000, 0xc20f9d0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20c6b6840, 0x7f5d6b8a1528, 0xc215e32460, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e32460, 0xc20f9dbbc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9088088 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b72868, 0xc21483e480, 0xc25e3c1ec0, 0xc25e3c1f20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130943 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ed91b0, 0xc213191f80, 0xc264c39e00, 0xc264c39e60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32317 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241803b80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23f2ce910, 0x7f5d6ba4b5a8, 0xc2145f4d70, 0xc264aea180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23f2ce910, 0x7f5d6ba4b5a8, 0xc2145f4d70, 0xc264aea180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2145f4d70) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 52683 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20af1a5a0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af1a5a0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af1a540, 0xc2300f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc212a6a0b8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2142ee640, 0xc2300f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2142ee640, 0xc2300f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2142ee640, 0xc2300f0000, 0xfa000, 0xfa000, 0xc20b900000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e14bad0, 0x7f5d6b8a1528, 0xc2142ee640, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2142ee640, 0xc20b906c60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72547 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc219c074f0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc219c074f0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc219c07490, 0xc2348f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2422e8a48) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc210937968, 0xc2348f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc210937968, 0xc2348f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc210937968, 0xc2348f0000, 0xfa000, 0xfa000, 0xc20d600000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc216efe060, 0x7f5d6b8a1528, 0xc210937968, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210937968, 0xc20d603440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32259 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f035a40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc259030be0, 0x7f5d6ba4b5a8, 0xc214de80a8, 0xc264625bc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc259030be0, 0x7f5d6ba4b5a8, 0xc214de80a8, 0xc264625bc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214de80a8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9088086 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af1c758, 0xc214042960, 0xc25e3c1c80, 0xc25e3c1ce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 12623 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b9c2300, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b9c2300, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b9c22a0, 0xc223fd2000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25294ce58) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20bfa2400, 0xc223fd2000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20bfa2400, 0xc223fd2000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20bfa2400, 0xc223fd2000, 0xfa000, 0xfa000, 0xc214ad0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214255320, 0x7f5d6b8a1528, 0xc20bfa2400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bfa2400, 0xc214ad5ce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 17033 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23b1988c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc264376550, 0x7f5d6ba4b5a8, 0xc21632b828, 0xc258bce600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc264376550, 0x7f5d6ba4b5a8, 0xc21632b828, 0xc258bce600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21632b828) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130849 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2145f5cc0, 0xc213445080, 0xc238d1d080, 0xc238d1d0e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131022 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2145e18a8, 0xc20fead740, 0xc261252ba0, 0xc261252c00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 49302 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20eb43100, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20eb43100, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20eb430a0, 0xc22d38a000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23d886f58) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21632b4b0, 0xc22d38a000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21632b4b0, 0xc22d38a000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21632b4b0, 0xc22d38a000, 0xfa000, 0xfa000, 0xc216210000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214eaf170, 0x7f5d6b8a1528, 0xc21632b4b0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21632b4b0, 0xc216215500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 15341 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bfe0450, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bfe0450, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bfe03f0, 0xc2251ec000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc256ce47e8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc216171a90, 0xc2251ec000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc216171a90, 0xc2251ec000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc216171a90, 0xc2251ec000, 0xfa000, 0xfa000, 0xc214f80000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21480ea20, 0x7f5d6b8a1528, 0xc216171a90, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216171a90, 0xc214f803c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 20403 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23b198280, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23f2ce870, 0x7f5d6ba4b5a8, 0xc2080c7660, 0xc258bce180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23f2ce870, 0x7f5d6ba4b5a8, 0xc2080c7660, 0xc258bce180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c7660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32183 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc218849540, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24566aa00, 0x7f5d6ba4b5a8, 0xc215b42d58, 0xc20b27f4a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24566aa00, 0x7f5d6ba4b5a8, 0xc215b42d58, 0xc20b27f4a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b42d58) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 51046 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc238e03cc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2643760a0, 0x7f5d6ba4b5a8, 0xc214835df0, 0xc24e0f1e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2643760a0, 0x7f5d6ba4b5a8, 0xc214835df0, 0xc24e0f1e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214835df0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9088007 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4ffd0, 0xc21006a360, 0xc218a334a0, 0xc218a33500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 89660 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21f69a300, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21f69a300, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21f69a2a0, 0xc237ef0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211d1a908) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2142709c8, 0xc237ef0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2142709c8, 0xc237ef0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2142709c8, 0xc237ef0000, 0xfa000, 0xfa000, 0xc20dbd0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc237fea000, 0x7f5d6b8a1528, 0xc2142709c8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2142709c8, 0xc20dbdc960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32184 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b4f6ae0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b4f6ae0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b4f6a80, 0xc227ef0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211d64090) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215b42d58, 0xc227ef0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215b42d58, 0xc227ef0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215b42d58, 0xc227ef0000, 0xfa000, 0xfa000, 0xc216130000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214bed680, 0x7f5d6b8a1528, 0xc215b42d58, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b42d58, 0xc21613dec0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 49299 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23d417900, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2153c27d0, 0x7f5d6ba4b5a8, 0xc215e72508, 0xc21c04d2c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2153c27d0, 0x7f5d6ba4b5a8, 0xc215e72508, 0xc21c04d2c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e72508) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 51077 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc212a151e0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc212a151e0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc212a15180, 0xc22f5f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25561b588) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214437e48, 0xc22f5f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214437e48, 0xc22f5f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214437e48, 0xc22f5f0000, 0xfa000, 0xfa000, 0xc20f930000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20c8eb500, 0x7f5d6b8a1528, 0xc214437e48, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214437e48, 0xc20f9392c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32203 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f415a40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc266116f50, 0x7f5d6ba4b5a8, 0xc214f363d8, 0xc24ec8bc20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc266116f50, 0x7f5d6ba4b5a8, 0xc214f363d8, 0xc24ec8bc20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f363d8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131173 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214c76a30, 0xc2132fe360, 0xc242581d40, 0xc242581da0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 92982 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21410b170, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21410b170, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21410b110, 0xc2386f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240b0ad58) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2138c10d0, 0xc2386f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2138c10d0, 0xc2386f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2138c10d0, 0xc2386f0000, 0xfa000, 0xfa000, 0xc213b60000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20cb8ae10, 0x7f5d6b8a1528, 0xc2138c10d0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2138c10d0, 0xc213b63980) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32186 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23d4177c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2153c26e0, 0x7f5d6ba4b5a8, 0xc215b43560, 0xc21c04d1a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2153c26e0, 0x7f5d6ba4b5a8, 0xc215b43560, 0xc21c04d1a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b43560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32095 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d666640, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21c03f360, 0x7f5d6ba4b5a8, 0xc214b765c8, 0xc2608468a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21c03f360, 0x7f5d6ba4b5a8, 0xc214b765c8, 0xc2608468a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b765c8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 18613 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2421d9540, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24b217ea0, 0x7f5d6ba4b5a8, 0xc20bfa3918, 0xc2150a62a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24b217ea0, 0x7f5d6ba4b5a8, 0xc20bfa3918, 0xc2150a62a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bfa3918) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 74787 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219b0f680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23f2cf1d0, 0x7f5d6ba4b5a8, 0xc215e41980, 0xc2150a76e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23f2cf1d0, 0x7f5d6ba4b5a8, 0xc215e41980, 0xc2150a76e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e41980) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 91706 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc216f2c8c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc26480c780, 0x7f5d6ba4b5a8, 0xc213ffbee0, 0xc261250840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc26480c780, 0x7f5d6ba4b5a8, 0xc213ffbee0, 0xc261250840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffbee0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 21923 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f035540, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc259030780, 0x7f5d6ba4b5a8, 0xc21fac0268, 0xc264625740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc259030780, 0x7f5d6ba4b5a8, 0xc21fac0268, 0xc264625740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21fac0268) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32304 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21d06bb80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d40d450, 0x7f5d6ba4b5a8, 0xc21437eb68, 0xc25fa2a5a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d40d450, 0x7f5d6ba4b5a8, 0xc21437eb68, 0xc25fa2a5a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21437eb68) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 51080 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c3b6d10, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c3b6d10, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c3b6cb0, 0xc22faf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24c517cf0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20bfa2198, 0xc22faf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20bfa2198, 0xc22faf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20bfa2198, 0xc22faf0000, 0xfa000, 0xfa000, 0xc20db20000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20cdb52c0, 0x7f5d6b8a1528, 0xc20bfa2198, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bfa2198, 0xc20db21080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 84698 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bfe0610, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bfe0610, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bfe05b0, 0xc2371f4000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2551c0e90) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af4ffd0, 0xc2371f4000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20af4ffd0, 0xc2371f4000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20af4ffd0, 0xc2371f4000, 0xfa000, 0xfa000, 0xc210060000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20baf49c0, 0x7f5d6b8a1528, 0xc20af4ffd0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4ffd0, 0xc21006a360) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 51064 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241803540, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2590305a0, 0x7f5d6ba4b5a8, 0xc2139ce670, 0xc264c39bc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2590305a0, 0x7f5d6ba4b5a8, 0xc2139ce670, 0xc264c39bc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139ce670) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32322 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21510e840, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21510e840, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21510e7e0, 0xc2297f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24a563400) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214a08ab0, 0xc2297f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214a08ab0, 0xc2297f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214a08ab0, 0xc2297f0000, 0xfa000, 0xfa000, 0xc214830000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2141ddbc0, 0x7f5d6b8a1528, 0xc214a08ab0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a08ab0, 0xc21483fd40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131116 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2b330, 0xc215fc2300, 0xc258bce960, 0xc258bce9c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 21908 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc215658d80, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc215658d80, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc215658d20, 0xc2234fa000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211da0f28) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215a2b7d0, 0xc2234fa000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215a2b7d0, 0xc2234fa000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215a2b7d0, 0xc2234fa000, 0xfa000, 0xfa000, 0xc214870000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2137dc660, 0x7f5d6b8a1528, 0xc215a2b7d0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2b7d0, 0xc21487ef60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 50863 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d667a40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc262d5c0a0, 0x7f5d6ba4b5a8, 0xc2147a1318, 0xc23e1f1ce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc262d5c0a0, 0x7f5d6ba4b5a8, 0xc2147a1318, 0xc23e1f1ce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2147a1318) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 51060 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc213163a40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc264ccfe50, 0x7f5d6ba4b5a8, 0xc21437e2e8, 0xc263f372c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc264ccfe50, 0x7f5d6ba4b5a8, 0xc21437e2e8, 0xc263f372c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21437e2e8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130865 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff39cb0, 0xc21b4471a0, 0xc2611e6300, 0xc2611e6360) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 72347 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f035180, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25b470b90, 0x7f5d6ba4b5a8, 0xc216267008, 0xc2646253e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25b470b90, 0x7f5d6ba4b5a8, 0xc216267008, 0xc2646253e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216267008) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50854 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2497352c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25b4711d0, 0x7f5d6ba4b5a8, 0xc215e32f20, 0xc23b9d23c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25b4711d0, 0x7f5d6ba4b5a8, 0xc215e32f20, 0xc23b9d23c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e32f20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 18606 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20ba205a0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20ba205a0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20ba20540, 0xc216544000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211da1660) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21622e658, 0xc216544000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21622e658, 0xc216544000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21622e658, 0xc216544000, 0xfa000, 0xfa000, 0xc215fd0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20b339470, 0x7f5d6b8a1528, 0xc21622e658, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622e658, 0xc215fd2360) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32268 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21e708140, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc258644820, 0x7f5d6ba4b5a8, 0xc2146172b8, 0xc261252600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc258644820, 0x7f5d6ba4b5a8, 0xc2146172b8, 0xc261252600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2146172b8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 72481 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc211c7a220, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc211c7a220, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc211c7a1c0, 0xc234cd8000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211dac448) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213656668, 0xc234cd8000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213656668, 0xc234cd8000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213656668, 0xc234cd8000, 0xfa000, 0xfa000, 0xc21dea0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ca132f0, 0x7f5d6b8a1528, 0xc213656668, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213656668, 0xc21dea4ea0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 26559 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23938be00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc26480c050, 0x7f5d6ba4b5a8, 0xc20b0456c8, 0xc261250060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc26480c050, 0x7f5d6ba4b5a8, 0xc20b0456c8, 0xc261250060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b0456c8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50993 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d6677c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc268483720, 0x7f5d6ba4b5a8, 0xc214c3e5a8, 0xc23e1f1aa0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc268483720, 0x7f5d6ba4b5a8, 0xc214c3e5a8, 0xc23e1f1aa0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214c3e5a8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 26751 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bf9b6b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bf9b6b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bf9b650, 0xc221b08000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc213d1c7a8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af1dde0, 0xc221b08000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20af1dde0, 0xc221b08000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20af1dde0, 0xc221b08000, 0xfa000, 0xfa000, 0xc213660000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2132e37d0, 0x7f5d6b8a1528, 0xc20af1dde0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af1dde0, 0xc213661620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 31160 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20ce6b480, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20ce6b480, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20ce6b420, 0xc2276f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc244456a18) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21ff38aa0, 0xc2276f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21ff38aa0, 0xc2276f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21ff38aa0, 0xc2276f0000, 0xfa000, 0xfa000, 0xc216440000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214c78fc0, 0x7f5d6b8a1528, 0xc21ff38aa0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff38aa0, 0xc216449e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 35740 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20ec06df0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20ec06df0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20ec06d90, 0xc22aef0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ef500c8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214271930, 0xc22aef0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214271930, 0xc22aef0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214271930, 0xc22aef0000, 0xfa000, 0xfa000, 0xc210330000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20b15fe30, 0x7f5d6b8a1528, 0xc214271930, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214271930, 0xc21033bbc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 39039 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21966c450, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21966c450, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21966c3f0, 0xc22b0f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2167a8b58) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21419a1d8, 0xc22b0f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21419a1d8, 0xc22b0f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21419a1d8, 0xc22b0f0000, 0xfa000, 0xfa000, 0xc20b040000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20dc71b00, 0x7f5d6b8a1528, 0xc21419a1d8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21419a1d8, 0xc20b041260) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32288 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc25d6823c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2590311d0, 0x7f5d6ba4b5a8, 0xc2144377a8, 0xc2633763c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2590311d0, 0x7f5d6ba4b5a8, 0xc2144377a8, 0xc2633763c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2144377a8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131111 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd2928, 0xc213685b60, 0xc258bce3c0, 0xc258bce420) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131254 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214437e48, 0xc20f9392c0, 0xc238d1d620, 0xc238d1d680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 26911 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20bbabb80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265d6c690, 0x7f5d6ba4b5a8, 0xc214f1e8c0, 0xc261252180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265d6c690, 0x7f5d6ba4b5a8, 0xc214f1e8c0, 0xc261252180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1e8c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 26907 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc262d84140, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2628a2550, 0x7f5d6ba4b5a8, 0xc214f1e6a0, 0xc24759e2a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2628a2550, 0x7f5d6ba4b5a8, 0xc214f1e6a0, 0xc24759e2a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1e6a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130830 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216027480, 0xc215fcc7e0, 0xc20b27fb60, 0xc20b27fbc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 30878 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21ff7c000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23f2ceb40, 0x7f5d6ba4b5a8, 0xc21ff390c0, 0xc264aea4e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23f2ceb40, 0x7f5d6ba4b5a8, 0xc21ff390c0, 0xc264aea4e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff390c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 72536 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d667400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc213e7b0e0, 0x7f5d6ba4b5a8, 0xc213656668, 0xc23e1f1740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc213e7b0e0, 0x7f5d6ba4b5a8, 0xc213656668, 0xc23e1f1740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213656668) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50879 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc213547f70, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc213547f70, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc213547f10, 0xc22e51e000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21606ead8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214c3e5a8, 0xc22e51e000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214c3e5a8, 0xc22e51e000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214c3e5a8, 0xc22e51e000, 0xfa000, 0xfa000, 0xc213530000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e5bb8f0, 0x7f5d6b8a1528, 0xc214c3e5a8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214c3e5a8, 0xc21353d6e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9088130 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160fe1a8, 0xc214c604e0, 0xc23d84bc80, 0xc23d84bce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129881 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160c4048, 0xc21634ccc0, 0xc263f36f60, 0xc263f36fc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 72459 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21bdab540, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc26480c550, 0x7f5d6ba4b5a8, 0xc21149f588, 0xc2425819e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc26480c550, 0x7f5d6ba4b5a8, 0xc21149f588, 0xc2425819e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21149f588) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 38986 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20ba216b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20ba216b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20ba21650, 0xc22b6f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23be9e238) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213ed9080, 0xc22b6f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213ed9080, 0xc22b6f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213ed9080, 0xc22b6f0000, 0xfa000, 0xfa000, 0xc20c050000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2110ceae0, 0x7f5d6b8a1528, 0xc213ed9080, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ed9080, 0xc20c0522a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 67350 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc238e037c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25a6a3680, 0x7f5d6ba4b5a8, 0xc215b42f68, 0xc217977bc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25a6a3680, 0x7f5d6ba4b5a8, 0xc215b42f68, 0xc217977bc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b42f68) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 30480 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c29f100, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c29f100, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c29f0a0, 0xc227bf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24a5636a8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e417e8, 0xc227bf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e417e8, 0xc227bf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e417e8, 0xc227bf0000, 0xfa000, 0xfa000, 0xc21d3f0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214f44b10, 0x7f5d6b8a1528, 0xc215e417e8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e417e8, 0xc21d3fbd40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131589 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211e88110, 0xc21f7665a0, 0xc24759f8c0, 0xc24759f920) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32169 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20ce6b790, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20ce6b790, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20ce6b730, 0xc2277f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24d673aa8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21ff390c0, 0xc2277f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21ff390c0, 0xc2277f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21ff390c0, 0xc2277f0000, 0xfa000, 0xfa000, 0xc216180000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc214c791a0, 0x7f5d6b8a1528, 0xc21ff390c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff390c0, 0xc2161842a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131146 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2162676c8, 0xc214116b40, 0xc23cc70900, 0xc23cc70960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131003 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160c4050, 0xc215fd01e0, 0xc23b9d2180, 0xc23b9d21e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 27061 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21510e6f0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21510e6f0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21510e690, 0xc2222d8000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc20f0b5ed8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214f1e8c0, 0xc2222d8000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214f1e8c0, 0xc2222d8000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214f1e8c0, 0xc2222d8000, 0xfa000, 0xfa000, 0xc213620000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20bdffef0, 0x7f5d6b8a1528, 0xc214f1e8c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1e8c0, 0xc21362ee40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 65995 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2328ed040, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc256203e00, 0x7f5d6ba4b5a8, 0xc21324b3b0, 0xc20cfed8c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc256203e00, 0x7f5d6ba4b5a8, 0xc21324b3b0, 0xc20cfed8c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21324b3b0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131186 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4eb30, 0xc216274480, 0xc240bba060, 0xc240bba0c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131019 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2145e06d8, 0xc2111f14a0, 0xc261252840, 0xc2612528a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131077 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21462e898, 0xc213860660, 0xc21c04d080, 0xc21c04d0e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130999 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215630408, 0xc20df67c20, 0xc23aa51ce0, 0xc23aa51d40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 50961 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210f00680, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210f00680, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210f00620, 0xc22ecf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc212a6a8a8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2147a0900, 0xc22ecf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2147a0900, 0xc22ecf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2147a0900, 0xc22ecf0000, 0xfa000, 0xfa000, 0xc2119c0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e5556b0, 0x7f5d6b8a1528, 0xc2147a0900, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2147a0900, 0xc2119ccc00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 42296 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b26e450, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b26e450, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b26e3f0, 0xc22b7f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21f042b58) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213aac030, 0xc22b7f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213aac030, 0xc22b7f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213aac030, 0xc22b7f0000, 0xfa000, 0xfa000, 0xc20f2c0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc219bde180, 0x7f5d6b8a1528, 0xc213aac030, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213aac030, 0xc20f2c5560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72485 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc25f0f6000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc262d5c820, 0x7f5d6ba4b5a8, 0xc210937968, 0xc23cc70b40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc262d5c820, 0x7f5d6ba4b5a8, 0xc210937968, 0xc23cc70b40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210937968) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 70701 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2080697c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc218a3b630, 0x7f5d6ba4b5a8, 0xc21ff38a68, 0xc238f470e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc218a3b630, 0x7f5d6ba4b5a8, 0xc21ff38a68, 0xc238f470e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff38a68) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131063 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2141defc0, 0xc213191f20, 0xc2150a6960, 0xc2150a69c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 89056 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc25de36640, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2618125a0, 0x7f5d6ba4b5a8, 0xc2142709c8, 0xc2611e6e40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2618125a0, 0x7f5d6ba4b5a8, 0xc2142709c8, 0xc2611e6e40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2142709c8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 51051 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219f31900, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d5f34a0, 0x7f5d6ba4b5a8, 0xc214437e48, 0xc238d1d620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d5f34a0, 0x7f5d6ba4b5a8, 0xc214437e48, 0xc238d1d620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214437e48) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130942 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21519f698, 0xc215eea240, 0xc264c39ce0, 0xc264c39d40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131006 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2149df558, 0xc211e9cde0, 0xc23b9d24e0, 0xc23b9d2540) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129760 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160ffa38, 0xc21619bc80, 0xc23b94de00, 0xc23b94de60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 49196 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bf9bdb0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bf9bdb0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bf9bd50, 0xc22d09c000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21606f2a8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e72de8, 0xc22d09c000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e72de8, 0xc22d09c000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e72de8, 0xc22d09c000, 0xfa000, 0xfa000, 0xc214c10000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21533f620, 0x7f5d6b8a1528, 0xc215e72de8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e72de8, 0xc214c19500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9087883 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213aad650, 0xc20f2c57a0, 0xc25a17a9c0, 0xc25a17aa20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131236 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214349d20, 0xc21db7f620, 0xc21aab1560, 0xc21aab15c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131048 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139a4f40, 0xc212caa900, 0xc261250720, 0xc261250780) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 89655 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc211822f00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25596aa00, 0x7f5d6ba4b5a8, 0xc215a05568, 0xc238d1b5c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25596aa00, 0x7f5d6ba4b5a8, 0xc215a05568, 0xc238d1b5c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a05568) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 37415 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a0fb720, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a0fb720, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a0fb6c0, 0xc22abf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc244456b88) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21fac0610, 0xc22abf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21fac0610, 0xc22abf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21fac0610, 0xc22abf0000, 0xfa000, 0xfa000, 0xc20ed30000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ec13aa0, 0x7f5d6b8a1528, 0xc21fac0610, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21fac0610, 0xc20ed386c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 55866 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b93d6b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b93d6b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b93d650, 0xc2308f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24a563d88) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21622eb98, 0xc2308f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21622eb98, 0xc2308f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21622eb98, 0xc2308f0000, 0xfa000, 0xfa000, 0xc214440000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ccbf6b0, 0x7f5d6b8a1528, 0xc21622eb98, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622eb98, 0xc214440240) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9859891 [semacquire]: sync.(*WaitGroup).Wait(0xc244297120) /usr/local/go/src/sync/waitgroup.go:132 +0x169 github.com/couchbase/indexing/secondary/indexer.(*storageMgr).createSnapshotWorker(0xc209f78900, 0xc20b240001, 0xc20bb21e70, 0x8, 0xc20ec06070, 0xc243a28c00, 0x400, 0xc243a28c30, 0xc243a28c60, 0xc20b248a50) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:386 +0x451 created by github.com/couchbase/indexing/secondary/indexer.(*storageMgr).handleCreateSnapshot /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:231 +0x712 goroutine 9131100 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e40af0, 0xc21d3fb8c0, 0xc24759e600, 0xc24759e660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131084 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2145f59f0, 0xc21c80d680, 0xc21c04d860, 0xc21c04d8c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129949 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210e65bf0, 0xc216010540, 0xc265d91aa0, 0xc265d91b00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 47569 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2170d3090, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2170d3090, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2170d3030, 0xc22fbf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc216423d40) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af1c758, 0xc22fbf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20af1c758, 0xc22fbf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20af1c758, 0xc22fbf0000, 0xfa000, 0xfa000, 0xc214040000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ce11b90, 0x7f5d6b8a1528, 0xc20af1c758, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af1c758, 0xc214042960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 64252 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc262d84000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2628a2460, 0x7f5d6ba4b5a8, 0xc21340e4b8, 0xc24759e180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2628a2460, 0x7f5d6ba4b5a8, 0xc21340e4b8, 0xc24759e180) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21340e4b8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 89699 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c3b6ca0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c3b6ca0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c3b6c40, 0xc2379f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ef508c8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215a05568, 0xc2379f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215a05568, 0xc2379f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215a05568, 0xc2379f0000, 0xfa000, 0xfa000, 0xc2281e0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc210e92660, 0x7f5d6b8a1528, 0xc215a05568, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a05568, 0xc2281ea960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 32633 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc211822000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25596a640, 0x7f5d6ba4b5a8, 0xc215e331a8, 0xc238d1b140) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25596a640, 0x7f5d6ba4b5a8, 0xc215e331a8, 0xc238d1b140) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e331a8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 86406 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc218848b40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24566b130, 0x7f5d6ba4b5a8, 0xc215c5ded8, 0xc20b27fec0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24566b130, 0x7f5d6ba4b5a8, 0xc215c5ded8, 0xc20b27fec0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215c5ded8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131551 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21533c010, 0xc213006000, 0xc25ecba720, 0xc25ecba780) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 51062 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21d06b540, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc259030a50, 0x7f5d6ba4b5a8, 0xc21437f5c0, 0xc25fa2a000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc259030a50, 0x7f5d6ba4b5a8, 0xc21437f5c0, 0xc25fa2a000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21437f5c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130156 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bfa2400, 0xc214ad5ce0, 0xc238f479e0, 0xc238f47a40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 32632 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23d417400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2153c2410, 0x7f5d6ba4b5a8, 0xc215e32c70, 0xc21c04ce40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2153c2410, 0x7f5d6ba4b5a8, 0xc215e32c70, 0xc21c04ce40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e32c70) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 74727 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d667b80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2628a2190, 0x7f5d6ba4b5a8, 0xc214f1ed18, 0xc23e1f1e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2628a2190, 0x7f5d6ba4b5a8, 0xc214f1ed18, 0xc23e1f1e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1ed18) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 65938 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2145a2000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24a3a9720, 0x7f5d6ba4b5a8, 0xc2138c1f68, 0xc26243e300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24a3a9720, 0x7f5d6ba4b5a8, 0xc2138c1f68, 0xc26243e300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2138c1f68) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 32631 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23dccac80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d5f2000, 0x7f5d6ba4b5a8, 0xc215e32460, 0xc265d91500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d5f2000, 0x7f5d6ba4b5a8, 0xc215e32460, 0xc265d91500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e32460) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 70571 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21e56e840, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21e56e840, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21e56e7e0, 0xc232cf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24efea428) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21367e9c8, 0xc232cf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21367e9c8, 0xc232cf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21367e9c8, 0xc232cf0000, 0xfa000, 0xfa000, 0xc216020000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21615e780, 0x7f5d6b8a1528, 0xc21367e9c8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21367e9c8, 0xc2160210e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 86402 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bd06d10, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bd06d10, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bd06cb0, 0xc2372f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25102c8f8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215d0d6c8, 0xc2372f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215d0d6c8, 0xc2372f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215d0d6c8, 0xc2372f0000, 0xfa000, 0xfa000, 0xc215ee0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc219ff43c0, 0x7f5d6b8a1528, 0xc215d0d6c8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215d0d6c8, 0xc215eea000) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 74726 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2328ecc80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc256203b30, 0x7f5d6ba4b5a8, 0xc214f1e6d0, 0xc20cfed560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc256203b30, 0x7f5d6ba4b5a8, 0xc214f1e6d0, 0xc20cfed560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1e6d0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50989 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23b199900, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc264376be0, 0x7f5d6ba4b5a8, 0xc215188738, 0xc258bcede0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc264376be0, 0x7f5d6ba4b5a8, 0xc215188738, 0xc258bcede0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215188738) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 40725 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20af45db0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20af45db0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20af45d50, 0xc22b5f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23e4c1d28) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215a2a660, 0xc22b5f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215a2a660, 0xc22b5f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215a2a660, 0xc22b5f0000, 0xfa000, 0xfa000, 0xc2119b0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2158cbe60, 0x7f5d6b8a1528, 0xc215a2a660, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2a660, 0xc2119b2e40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131241 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd3de8, 0xc213685e00, 0xc21aab1b00, 0xc21aab1b60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 74812 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b138450, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b138450, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b1383f0, 0xc236060000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ac8b648) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214a08128, 0xc236060000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214a08128, 0xc236060000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214a08128, 0xc236060000, 0xfa000, 0xfa000, 0xc215fb0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2160e4000, 0x7f5d6b8a1528, 0xc214a08128, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a08128, 0xc215fb3800) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131120 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215188738, 0xc2136e0480, 0xc258bcede0, 0xc258bcee40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131159 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214c77e00, 0xc214323380, 0xc264aea720, 0xc264aea780) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 38816 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d667cc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2628a2280, 0x7f5d6ba4b5a8, 0xc214349d28, 0xc23e1f1f20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2628a2280, 0x7f5d6ba4b5a8, 0xc214349d28, 0xc23e1f1f20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214349d28) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 74803 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc213249250, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc213249250, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2132491f0, 0xc235b7e000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2557b3908) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21533dc58, 0xc235b7e000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21533dc58, 0xc235b7e000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21533dc58, 0xc235b7e000, 0xfa000, 0xfa000, 0xc216220000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc216202630, 0x7f5d6b8a1528, 0xc21533dc58, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21533dc58, 0xc2162251a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72515 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241803900, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21d40d040, 0x7f5d6ba4b5a8, 0xc210b720d0, 0xc264c39f20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21d40d040, 0x7f5d6ba4b5a8, 0xc210b720d0, 0xc264c39f20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210b720d0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131080 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21419b0b8, 0xc20b8fc120, 0xc21c04d3e0, 0xc21c04d440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 79628 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21bdab7c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc26480c6e0, 0x7f5d6ba4b5a8, 0xc215b42a50, 0xc242581c20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc26480c6e0, 0x7f5d6ba4b5a8, 0xc215b42a50, 0xc242581c20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b42a50) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9129953 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4f7b8, 0xc214796ae0, 0xc265d91f20, 0xc265d91f80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 57951 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d25d2c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2551e7950, 0x7f5d6ba4b5a8, 0xc215630510, 0xc240bba840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2551e7950, 0x7f5d6ba4b5a8, 0xc215630510, 0xc240bba840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215630510) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 51037 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210f01330, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210f01330, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210f012d0, 0xc22f8f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21e7cd6d8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215630840, 0xc22f8f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215630840, 0xc22f8f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215630840, 0xc22f8f0000, 0xfa000, 0xfa000, 0xc20f000000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20b4cf320, 0x7f5d6b8a1528, 0xc215630840, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215630840, 0xc20f00baa0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 55861 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20ba21aa0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20ba21aa0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20ba21a40, 0xc2307f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc253927ff8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2080c6cd8, 0xc2307f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2080c6cd8, 0xc2307f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2080c6cd8, 0xc2307f0000, 0xfa000, 0xfa000, 0xc21b510000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21c21e630, 0x7f5d6b8a1528, 0xc2080c6cd8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c6cd8, 0xc21b517ec0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 62747 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20e598b50, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20e598b50, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20e598af0, 0xc2317f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ef50438) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21369fe18, 0xc2317f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21369fe18, 0xc2317f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21369fe18, 0xc2317f0000, 0xfa000, 0xfa000, 0xc219e00000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2170639e0, 0x7f5d6b8a1528, 0xc21369fe18, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21369fe18, 0xc219e0cde0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9129489 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2162662c8, 0xc213fa1b60, 0xc21aab1200, 0xc21aab1260) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131160 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd3028, 0xc21402e3c0, 0xc264aea840, 0xc264aea8a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 72154 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241064dc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25b470fa0, 0x7f5d6ba4b5a8, 0xc211a48d60, 0xc21aab0ea0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25b470fa0, 0x7f5d6ba4b5a8, 0xc211a48d60, 0xc21aab0ea0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211a48d60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131060 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e324f8, 0xc214a9ae40, 0xc2150a6600, 0xc2150a6660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 45642 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc24f819b80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc264376000, 0x7f5d6ba4b5a8, 0xc215ffe0d8, 0xc238d1ade0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc264376000, 0x7f5d6ba4b5a8, 0xc215ffe0d8, 0xc238d1ade0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215ffe0d8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 51058 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210f00760, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210f00760, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210f00700, 0xc22edf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc253927a28) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2147a1318, 0xc22edf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2147a1318, 0xc22edf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2147a1318, 0xc22edf0000, 0xfa000, 0xfa000, 0xc211740000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ee41b60, 0x7f5d6b8a1528, 0xc2147a1318, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2147a1318, 0xc211744960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 40743 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc238e03680, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25a6a3590, 0x7f5d6ba4b5a8, 0xc213aadb70, 0xc217977aa0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25a6a3590, 0x7f5d6ba4b5a8, 0xc213aadb70, 0xc217977aa0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213aadb70) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 74810 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b1383e0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b1383e0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b138380, 0xc235f66000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25294c3e8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214b77558, 0xc235f66000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214b77558, 0xc235f66000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214b77558, 0xc235f66000, 0xfa000, 0xfa000, 0xc216150000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2161906f0, 0x7f5d6b8a1528, 0xc214b77558, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b77558, 0xc2161595c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131122 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21519f7e8, 0xc21b4c69c0, 0xc24ec8be60, 0xc24ec8bec0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 40741 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23fbc3900, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc21c03f400, 0x7f5d6ba4b5a8, 0xc213aac9a8, 0xc23aca5020) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc21c03f400, 0x7f5d6ba4b5a8, 0xc213aac9a8, 0xc23aca5020) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213aac9a8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 40723 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21d06ba40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc251853bd0, 0x7f5d6ba4b5a8, 0xc215a2a658, 0xc25fa2a480) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc251853bd0, 0x7f5d6ba4b5a8, 0xc215a2a658, 0xc25fa2a480) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2a658) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 70697 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2170d2450, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2170d2450, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2170d23f0, 0xc234bde000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25294cbe8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc212bbfb30, 0xc234bde000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc212bbfb30, 0xc234bde000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc212bbfb30, 0xc234bde000, 0xfa000, 0xfa000, 0xc21a140000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ca12570, 0x7f5d6b8a1528, 0xc212bbfb30, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212bbfb30, 0xc21a14c2a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131253 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21538e510, 0xc21639a360, 0xc238d1d500, 0xc238d1d560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 43895 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc211c7a530, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc211c7a530, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc211c7a4d0, 0xc22bdf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc20f0b5f80) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214321c68, 0xc22bdf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214321c68, 0xc22bdf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214321c68, 0xc22bdf0000, 0xfa000, 0xfa000, 0xc213920000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20f6135f0, 0x7f5d6b8a1528, 0xc214321c68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214321c68, 0xc21392e840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 50878 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23f035e00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc259030e10, 0x7f5d6ba4b5a8, 0xc214f37198, 0xc264625f20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc259030e10, 0x7f5d6ba4b5a8, 0xc214f37198, 0xc264625f20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f37198) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 43962 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc211c7a5a0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc211c7a5a0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc211c7a540, 0xc22bef0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240d6a888) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214320a48, 0xc22bef0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214320a48, 0xc22bef0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214320a48, 0xc22bef0000, 0xfa000, 0xfa000, 0xc212ca0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20f19ed20, 0x7f5d6b8a1528, 0xc214320a48, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214320a48, 0xc212caa8a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72352 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2170d27d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2170d27d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2170d2770, 0xc2353ae000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240d6a9f8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2160269c8, 0xc2353ae000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2160269c8, 0xc2353ae000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2160269c8, 0xc2353ae000, 0xfa000, 0xfa000, 0xc21f760000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2158ef860, 0x7f5d6b8a1528, 0xc2160269c8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160269c8, 0xc21f769ec0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131007 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff38558, 0xc216007020, 0xc23b9d2600, 0xc23b9d2660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 50996 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc20d25c780, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2551e7770, 0x7f5d6ba4b5a8, 0xc214b73168, 0xc240bba600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2551e7770, 0x7f5d6ba4b5a8, 0xc214b73168, 0xc240bba600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b73168) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50880 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21510fe20, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21510fe20, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21510fdc0, 0xc22e712000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24a563f08) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214b73168, 0xc22e712000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214b73168, 0xc22e712000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214b73168, 0xc22e712000, 0xfa000, 0xfa000, 0xc2133e0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ec1e0c0, 0x7f5d6b8a1528, 0xc214b73168, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b73168, 0xc2133e2f00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 59927 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a83a760, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a83a760, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a83a700, 0xc2316f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240d6b988) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21621d790, 0xc2316f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21621d790, 0xc2316f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21621d790, 0xc2316f0000, 0xfa000, 0xfa000, 0xc20ca70000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc217063920, 0x7f5d6b8a1528, 0xc21621d790, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21621d790, 0xc20ca78420) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 74639 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc24f819400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc249c03ea0, 0x7f5d6ba4b5a8, 0xc21533dc58, 0xc238d1aa80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc249c03ea0, 0x7f5d6ba4b5a8, 0xc21533dc58, 0xc238d1aa80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21533dc58) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131090 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214c3e5a8, 0xc21353d6e0, 0xc23e1f1aa0, 0xc23e1f1b00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 43899 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc216f2c780, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc26480c5f0, 0x7f5d6ba4b5a8, 0xc2139a4f40, 0xc261250720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc26480c5f0, 0x7f5d6ba4b5a8, 0xc2139a4f40, 0xc261250720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139a4f40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 42060 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c29e300, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c29e300, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c29e2a0, 0xc22c6d8000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc210321e48) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc20af4eee0, 0xc22c6d8000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc20af4eee0, 0xc22c6d8000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc20af4eee0, 0xc22c6d8000, 0xfa000, 0xfa000, 0xc210570000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20afe3680, 0x7f5d6b8a1528, 0xc20af4eee0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4eee0, 0xc210571bc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131044 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214320a48, 0xc212caa8a0, 0xc2612502a0, 0xc261250300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 45135 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219b0e000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2560a81e0, 0x7f5d6ba4b5a8, 0xc20af4eee0, 0xc2150a6840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2560a81e0, 0x7f5d6ba4b5a8, 0xc20af4eee0, 0xc2150a6840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4eee0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 45136 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2135461b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2135461b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc213546150, 0xc22c7d2000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21e7cdfc8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2162669c0, 0xc22c7d2000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2162669c0, 0xc22c7d2000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2162669c0, 0xc22c7d2000, 0xfa000, 0xfa000, 0xc210b70000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc215e4d110, 0x7f5d6b8a1528, 0xc2162669c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2162669c0, 0xc210b7aea0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 45134 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc213546140, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc213546140, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2135460e0, 0xc22c5de000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24a563020) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc216266138, 0xc22c5de000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc216266138, 0xc22c5de000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc216266138, 0xc22c5de000, 0xfa000, 0xfa000, 0xc20c790000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20f08be30, 0x7f5d6b8a1528, 0xc216266138, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216266138, 0xc20c796240) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9129889 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21149e030, 0xc20d6028a0, 0xc263f37860, 0xc263f378c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 72514 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc2200ea0d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc2200ea0d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc2200ea070, 0xc2342f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21606efa8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc210e641a8, 0xc2342f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc210e641a8, 0xc2342f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc210e641a8, 0xc2342f0000, 0xfa000, 0xfa000, 0xc20bb90000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21a45fa70, 0x7f5d6b8a1528, 0xc210e641a8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210e641a8, 0xc20bb91440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72531 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21418e7d0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21418e7d0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21418e770, 0xc2344f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24a563068) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc210ad43b0, 0xc2344f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc210ad43b0, 0xc2344f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc210ad43b0, 0xc2344f0000, 0xfa000, 0xfa000, 0xc216010000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc217f123c0, 0x7f5d6b8a1528, 0xc210ad43b0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210ad43b0, 0xc2160107e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 88043 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c3b6c30, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c3b6c30, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c3b6bd0, 0xc2378f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21606f6e8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215a050e0, 0xc2378f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215a050e0, 0xc2378f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215a050e0, 0xc2378f0000, 0xfa000, 0xfa000, 0xc20e6f0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20b22bda0, 0x7f5d6b8a1528, 0xc215a050e0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a050e0, 0xc20e6fdf80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131083 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622fc00, 0xc20bb873e0, 0xc21c04d740, 0xc21c04d7a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 47378 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241a6fcc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24e2d2500, 0x7f5d6ba4b5a8, 0xc213ffb668, 0xc263f37aa0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24e2d2500, 0x7f5d6ba4b5a8, 0xc213ffb668, 0xc263f37aa0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffb668) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131005 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e32f20, 0xc20b041980, 0xc23b9d23c0, 0xc23b9d2420) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 52707 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241803400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc241a74370, 0x7f5d6ba4b5a8, 0xc20bb598a8, 0xc216b25620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc241a74370, 0x7f5d6ba4b5a8, 0xc20bb598a8, 0xc216b25620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bb598a8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 59830 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc215bcb7c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25b470f00, 0x7f5d6ba4b5a8, 0xc213718c28, 0xc2611e61e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25b470f00, 0x7f5d6ba4b5a8, 0xc213718c28, 0xc2611e61e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213718c28) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 60508 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc245500280, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc268483e50, 0x7f5d6ba4b5a8, 0xc21621dc18, 0xc21aab18c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc268483e50, 0x7f5d6ba4b5a8, 0xc21621dc18, 0xc21aab18c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21621dc18) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 72477 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21de34b50, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21de34b50, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21de34af0, 0xc2347f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc253927750) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc210b73e50, 0xc2347f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc210b73e50, 0xc2347f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc210b73e50, 0xc2347f0000, 0xfa000, 0xfa000, 0xc21f760000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc217f12570, 0x7f5d6b8a1528, 0xc210b73e50, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210b73e50, 0xc21f766cc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131194 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e859a8, 0xc21d3bca20, 0xc240bba960, 0xc240bba9c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131314 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2a588, 0xc20e516180, 0xc263f37980, 0xc263f379e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130959 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e331a8, 0xc210f08480, 0xc238d1b140, 0xc238d1b1a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 52629 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23dcca780, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc244813270, 0x7f5d6ba4b5a8, 0xc2139a5a40, 0xc2459e8480) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc244813270, 0x7f5d6ba4b5a8, 0xc2139a5a40, 0xc2459e8480) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139a5a40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50991 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a0fb480, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a0fb480, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a0fb420, 0xc22e230000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc218698e58) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214fef858, 0xc22e230000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214fef858, 0xc22e230000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214fef858, 0xc22e230000, 0xfa000, 0xfa000, 0xc2136b0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20f2dd950, 0x7f5d6b8a1528, 0xc214fef858, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214fef858, 0xc2136b71a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 51011 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc212a14ae0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc212a14ae0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc212a14a80, 0xc22e906000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24a563578) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214c773b0, 0xc22e906000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214c773b0, 0xc22e906000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214c773b0, 0xc22e906000, 0xfa000, 0xfa000, 0xc2131d0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ec17b00, 0x7f5d6b8a1528, 0xc214c773b0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214c773b0, 0xc2131d5740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 51014 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc249735400, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25b471c20, 0x7f5d6ba4b5a8, 0xc2149df558, 0xc23b9d24e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25b471c20, 0x7f5d6ba4b5a8, 0xc2149df558, 0xc23b9d24e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2149df558) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 50852 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241803a40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23f2ce780, 0x7f5d6ba4b5a8, 0xc215e32748, 0xc264aea060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23f2ce780, 0x7f5d6ba4b5a8, 0xc215e32748, 0xc264aea060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e32748) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130861 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2142ee640, 0xc20b906c60, 0xc20b281e60, 0xc20b281ec0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 93000 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241065cc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc268483b80, 0x7f5d6ba4b5a8, 0xc214349d20, 0xc21aab1560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc268483b80, 0x7f5d6ba4b5a8, 0xc214349d20, 0xc21aab1560) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214349d20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9084353 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af1d460, 0xc20b906900, 0xc23d84bb60, 0xc23d84bbc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129936 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b43f70, 0xc2162ab260, 0xc23e1f1860, 0xc23e1f18c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131131 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2130a3620, 0xc21b447620, 0xc26945e8a0, 0xc26945e900) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131064 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bb59a40, 0xc214350a80, 0xc2150a6a80, 0xc2150a6ae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 49240 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bf9bcd0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bf9bcd0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bf9bc70, 0xc22cfa2000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23d886c98) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e72508, 0xc22cfa2000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e72508, 0xc22cfa2000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e72508, 0xc22cfa2000, 0xfa000, 0xfa000, 0xc214c60000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc215e55980, 0x7f5d6b8a1528, 0xc215e72508, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e72508, 0xc214c61e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 65927 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b139950, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b139950, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b1398f0, 0xc2327f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21ad5a6e8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215b42f68, 0xc2327f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215b42f68, 0xc2327f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215b42f68, 0xc2327f0000, 0xfa000, 0xfa000, 0xc21e910000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ea54240, 0x7f5d6b8a1528, 0xc215b42f68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b42f68, 0xc21e91c660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 45636 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2188483c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24566ab90, 0x7f5d6ba4b5a8, 0xc21fac0578, 0xc20b27f800) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24566ab90, 0x7f5d6ba4b5a8, 0xc21fac0578, 0xc20b27f800) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21fac0578) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131008 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216170598, 0xc216184240, 0xc23b9d2720, 0xc23b9d2780) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 59672 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc211823180, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25596abe0, 0x7f5d6ba4b5a8, 0xc21ff391f0, 0xc238d1b800) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25596abe0, 0x7f5d6ba4b5a8, 0xc21ff391f0, 0xc238d1b800) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff391f0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 45608 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210f016b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210f016b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210f01650, 0xc22c1f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc25294c948) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2139cec00, 0xc22c1f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2139cec00, 0xc22c1f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2139cec00, 0xc22c1f0000, 0xfa000, 0xfa000, 0xc219e90000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20b5640f0, 0x7f5d6b8a1528, 0xc2139cec00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139cec00, 0xc219e91e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9129871 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2136fd060, 0xc21b5132c0, 0xc265d8fc80, 0xc265d8fce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 62709 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc244c48000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc264ccfd60, 0x7f5d6ba4b5a8, 0xc21369e0d8, 0xc21758b440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc264ccfd60, 0x7f5d6ba4b5a8, 0xc21369e0d8, 0xc21758b440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21369e0d8) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130956 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215ffe0d8, 0xc21619a3c0, 0xc238d1ade0, 0xc238d1ae40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 52631 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21c4b4280, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23ffede50, 0x7f5d6ba4b5a8, 0xc2139a5030, 0xc2458d34a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23ffede50, 0x7f5d6ba4b5a8, 0xc2139a5030, 0xc2458d34a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2139a5030) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 72151 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23dccaf00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2153c2320, 0x7f5d6ba4b5a8, 0xc2130a2f68, 0xc265d91740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2153c2320, 0x7f5d6ba4b5a8, 0xc2130a2f68, 0xc265d91740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2130a2f68) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9129948 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a17048, 0xc21466f740, 0xc265d91980, 0xc265d919e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131192 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b42588, 0xc216181e00, 0xc240bba720, 0xc240bba780) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131209 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160c4020, 0xc215fd0180, 0xc25fa2ac60, 0xc25fa2acc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131224 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622f740, 0xc216143080, 0xc2611e6ae0, 0xc2611e6b40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 59673 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21816e060, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21816e060, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21816e000, 0xc230ff0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24ef50b88) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21ff391f0, 0xc230ff0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21ff391f0, 0xc230ff0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21ff391f0, 0xc230ff0000, 0xfa000, 0xfa000, 0xc20def0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21bb94390, 0x7f5d6b8a1528, 0xc21ff391f0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff391f0, 0xc20def6840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131062 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af4eee0, 0xc210571bc0, 0xc2150a6840, 0xc2150a68a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 70205 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20e599e90, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20e599e90, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20e599e30, 0xc233ef0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc213d1c058) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc210e65bf0, 0xc233ef0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc210e65bf0, 0xc233ef0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc210e65bf0, 0xc233ef0000, 0xfa000, 0xfa000, 0xc216010000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2159267b0, 0x7f5d6b8a1528, 0xc210e65bf0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210e65bf0, 0xc216010540) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131074 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160269c8, 0xc21f769ec0, 0xc21c04cd20, 0xc21c04cd80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130791 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160c5c10, 0xc21613a4e0, 0xc2646252c0, 0xc264625320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129934 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e739b8, 0xc213b205a0, 0xc23e1f1620, 0xc23e1f1680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131379 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e41980, 0xc21a5a9ec0, 0xc2150a76e0, 0xc2150a7740) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 74835 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bb1e220, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bb1e220, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bb1e1c0, 0xc236542000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211d64fa8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2145e0078, 0xc236542000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2145e0078, 0xc236542000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2145e0078, 0xc236542000, 0xfa000, 0xfa000, 0xc214f80000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc215f94db0, 0x7f5d6b8a1528, 0xc2145e0078, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2145e0078, 0xc214f81e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131178 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2138c1f68, 0xc21297c540, 0xc26243e300, 0xc26243e360) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131158 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215fff7f0, 0xc2143cb380, 0xc264aea600, 0xc264aea660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130540 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214321070, 0xc212cb0360, 0xc262295da0, 0xc262295e00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9088101 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214427700, 0xc20e02f1a0, 0xc25fa6b740, 0xc25fa6b7a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129943 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff39aa0, 0xc216449f80, 0xc265d913e0, 0xc265d91440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 74725 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc211c7bf70, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc211c7bf70, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc211c7bf10, 0xc235890000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21ddd6328) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215e41980, 0xc235890000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215e41980, 0xc235890000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215e41980, 0xc235890000, 0xfa000, 0xfa000, 0xc21a5a0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc230eeecc0, 0x7f5d6b8a1528, 0xc215e41980, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e41980, 0xc21a5a9ec0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131141 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c6b88, 0xc21a5955c0, 0xc23cc70360, 0xc23cc703c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130151 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21306a2c0, 0xc21afe90e0, 0xc238f47440, 0xc238f474a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130960 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21369fe18, 0xc219e0cde0, 0xc238d1b260, 0xc238d1b2c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131332 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216267ed8, 0xc2162bde00, 0xc25a4502a0, 0xc25a450300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 74809 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc25de363c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc265d6d360, 0x7f5d6ba4b5a8, 0xc214de9480, 0xc2611e6c00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc265d6d360, 0x7f5d6ba4b5a8, 0xc214de9480, 0xc2611e6c00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214de9480) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 92979 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20c3b7480, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20c3b7480, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20c3b7420, 0xc2384f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc252036868) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2136fc418, 0xc2384f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2136fc418, 0xc2384f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2136fc418, 0xc2384f0000, 0xfa000, 0xfa000, 0xc213b60000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2109c35f0, 0x7f5d6b8a1528, 0xc2136fc418, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2136fc418, 0xc213b63920) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131227 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2142709c8, 0xc20dbdc960, 0xc2611e6e40, 0xc2611e6ea0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 74689 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bb1e140, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bb1e140, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bb1e0e0, 0xc23634e000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21bcc75b8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21462e850, 0xc23634e000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21462e850, 0xc23634e000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21462e850, 0xc23634e000, 0xfa000, 0xfa000, 0xc215490000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc215fb0030, 0x7f5d6b8a1528, 0xc21462e850, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21462e850, 0xc215499500) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72158 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23b198140, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc2518538b0, 0x7f5d6ba4b5a8, 0xc211a49bb0, 0xc258bce060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc2518538b0, 0x7f5d6ba4b5a8, 0xc211a49bb0, 0xc258bce060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211a49bb0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 67442 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21816e5a0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21816e5a0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21816e540, 0xc2321f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc2557b3c98) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21ff39cb0, 0xc2321f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21ff39cb0, 0xc2321f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21ff39cb0, 0xc2321f0000, 0xfa000, 0xfa000, 0xc21b440000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21baaeed0, 0x7f5d6b8a1528, 0xc21ff39cb0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff39cb0, 0xc21b4471a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131239 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21621dc18, 0xc20f56ea20, 0xc21aab18c0, 0xc21aab1920) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 72452 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241a6fa40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24e2d2320, 0x7f5d6ba4b5a8, 0xc21149e030, 0xc263f37860) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24e2d2320, 0x7f5d6ba4b5a8, 0xc21149e030, 0xc263f37860) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21149e030) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 54341 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc249734640, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25b470c30, 0x7f5d6ba4b5a8, 0xc215630408, 0xc23aa51ce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25b470c30, 0x7f5d6ba4b5a8, 0xc215630408, 0xc23aa51ce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215630408) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 54342 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210f019c0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210f019c0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210f01960, 0xc2306f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24458a220) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215630408, 0xc2306f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215630408, 0xc2306f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215630408, 0xc2306f0000, 0xfa000, 0xfa000, 0xc20df60000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20e438f30, 0x7f5d6b8a1528, 0xc215630408, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215630408, 0xc20df67c20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9129296 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f363d8, 0xc215ec1f80, 0xc24ec8bc20, 0xc24ec8bc80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131132 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215ffe078, 0xc215480de0, 0xc26945e9c0, 0xc26945ea20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131226 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160fe938, 0xc215fc30e0, 0xc2611e6d20, 0xc2611e6d80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 70816 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc21ff7cf00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23f2cf0e0, 0x7f5d6ba4b5a8, 0xc211056b20, 0xc264aeaba0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23f2cf0e0, 0x7f5d6ba4b5a8, 0xc211056b20, 0xc264aeaba0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211056b20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9087594 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b76bd0, 0xc214797680, 0xc255c3e8a0, 0xc255c3e900) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 89657 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b4f73a0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b4f73a0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b4f7340, 0xc237cf0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc248607a20) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213ed9a08, 0xc237cf0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213ed9a08, 0xc237cf0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213ed9a08, 0xc237cf0000, 0xfa000, 0xfa000, 0xc20dbd0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20ff41860, 0x7f5d6b8a1528, 0xc213ed9a08, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ed9a08, 0xc20dbdc780) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 72567 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc218848140, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc25596a0a0, 0x7f5d6ba4b5a8, 0xc21622e9b0, 0xc20b27f020) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc25596a0a0, 0x7f5d6ba4b5a8, 0xc21622e9b0, 0xc20b27f020) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622e9b0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130257 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2136fcfa0, 0xc21b513320, 0xc21c04cc00, 0xc21c04cc60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 69032 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc208069cc0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc264ccfcc0, 0x7f5d6ba4b5a8, 0xc21306a2c0, 0xc238f47440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc264ccfcc0, 0x7f5d6ba4b5a8, 0xc21306a2c0, 0xc238f47440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21306a2c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130842 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215419d88, 0xc21630c2a0, 0xc238d1c8a0, 0xc238d1c900) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131298 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214b73c98, 0xc2131e4660, 0xc20cfed7a0, 0xc20cfed800) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131023 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21519e5d0, 0xc2139a1260, 0xc261252cc0, 0xc261252d20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131076 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2131d0390, 0xc21297c6c0, 0xc21c04cf60, 0xc21c04cfc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131032 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216170148, 0xc215d41080, 0xc21758b7a0, 0xc21758b800) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 83111 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21a045e90, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21a045e90, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21a045e30, 0xc237000000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc21e6e88f8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215a2bb80, 0xc237000000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215a2bb80, 0xc237000000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215a2bb80, 0xc237000000, 0xfa000, 0xfa000, 0xc2159a0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc217f127e0, 0x7f5d6b8a1528, 0xc215a2bb80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2bb80, 0xc2159a6300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9130828 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b045888, 0xc215e93a40, 0xc20b27f920, 0xc20b27f980) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9087790 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213aadb70, 0xc2118e8780, 0xc217977aa0, 0xc217977b00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131128 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21361c738, 0xc21db7f740, 0xc26945e540, 0xc26945e5a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129263 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff617f0, 0xc2109226c0, 0xc242581680, 0xc2425816e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9088089 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ed9080, 0xc20c0522a0, 0xc248010000, 0xc248010060) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131037 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21437ed40, 0xc20f9db080, 0xc21758bd40, 0xc21758bda0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131009 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21463a720, 0xc21368c240, 0xc23b9d2840, 0xc23b9d28a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131079 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e72508, 0xc214c61e00, 0xc21c04d2c0, 0xc21c04d320) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131237 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e72de8, 0xc214c19500, 0xc21aab1680, 0xc21aab16e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131221 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bb58bd8, 0xc21d492000, 0xc2611e6780, 0xc2611e67e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 89661 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc21410aed0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc21410aed0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc21410ae70, 0xc2382f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24d6726a8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2138c0670, 0xc2382f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2138c0670, 0xc2382f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2138c0670, 0xc2382f0000, 0xfa000, 0xfa000, 0xc20dbd0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc20cf65fb0, 0x7f5d6b8a1528, 0xc2138c0670, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2138c0670, 0xc20dbdcae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131207 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215c5d060, 0xc2100cd8c0, 0xc25fa2aa20, 0xc25fa2aa80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 86405 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc217385fe0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc217385fe0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc217385f80, 0xc2375f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc24d6723a8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc21519f698, 0xc2375f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc21519f698, 0xc2375f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc21519f698, 0xc2375f0000, 0xfa000, 0xfa000, 0xc215ee0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21056fe00, 0x7f5d6b8a1528, 0xc21519f698, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21519f698, 0xc215eea240) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131070 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2147a1cc8, 0xc21147b800, 0xc2150a7140, 0xc2150a71a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131147 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215fff0e0, 0xc21621ecc0, 0xc23cc70a20, 0xc23cc70a80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 89730 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc219b0e140, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc258644640, 0x7f5d6ba4b5a8, 0xc2141defc0, 0xc2150a6960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc258644640, 0x7f5d6ba4b5a8, 0xc2141defc0, 0xc2150a6960) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2141defc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 78121 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241a6fb80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24e2d2410, 0x7f5d6ba4b5a8, 0xc215a2a588, 0xc263f37980) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24e2d2410, 0x7f5d6ba4b5a8, 0xc215a2a588, 0xc263f37980) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a2a588) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131144 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c6cd8, 0xc21b517ec0, 0xc23cc706c0, 0xc23cc70720) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129265 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215eda5e0, 0xc214c63b00, 0xc2425818c0, 0xc242581920) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9083803 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21367e4f0, 0xc21afe9320, 0xc260f8fda0, 0xc260f8fe00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131024 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215fffa48, 0xc2162aaae0, 0xc261252de0, 0xc261252e40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131068 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21fac0610, 0xc20ed386c0, 0xc2150a6f00, 0xc2150a6f60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130843 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210ad43b0, 0xc2160107e0, 0xc238d1c9c0, 0xc238d1ca20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131174 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e413f8, 0xc216184300, 0xc242581e60, 0xc242581ec0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131097 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1e6a0, 0xc213288ae0, 0xc24759e2a0, 0xc24759e300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 79343 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23aabd7c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc262d5c190, 0x7f5d6ba4b5a8, 0xc2080c6b88, 0xc23cc70360) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc262d5c190, 0x7f5d6ba4b5a8, 0xc2080c6b88, 0xc23cc70360) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c6b88) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9129883 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210df00c8, 0xc20d603200, 0xc263f371a0, 0xc263f37200) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131163 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211056b20, 0xc21f766960, 0xc264aeaba0, 0xc264aeac00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 79689 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc213249bf0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc213249bf0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc213249b90, 0xc236830000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc23d8873b8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc2145f59f0, 0xc236830000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc2145f59f0, 0xc236830000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc2145f59f0, 0xc236830000, 0xfa000, 0xfa000, 0xc21c800000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc2136a9110, 0x7f5d6b8a1528, 0xc2145f59f0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2145f59f0, 0xc21c80d680) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 79746 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc218848640, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc24566ad70, 0x7f5d6ba4b5a8, 0xc2080c7708, 0xc20b27fa40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc24566ad70, 0x7f5d6ba4b5a8, 0xc2080c7708, 0xc20b27fa40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c7708) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131299 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21324b3b0, 0xc21297d1a0, 0xc20cfed8c0, 0xc20cfed920) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130795 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21fac0268, 0xc215ec1bc0, 0xc264625740, 0xc2646257a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131154 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2145f4d70, 0xc21347a000, 0xc264aea180, 0xc264aea1e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131211 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211a48640, 0xc20d602ae0, 0xc25fa2aea0, 0xc25fa2af00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 82977 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20bcead80, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20bcead80, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20bcead20, 0xc2370fa000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc249b700c0) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc214427700, 0xc2370fa000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc214427700, 0xc2370fa000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc214427700, 0xc2370fa000, 0xfa000, 0xfa000, 0xc20e020000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21d2fd200, 0x7f5d6b8a1528, 0xc214427700, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214427700, 0xc20e02f1a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 83110 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc23aabcb40, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc23df6ac30, 0x7f5d6ba4b5a8, 0xc214426d00, 0xc253848900) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc23df6ac30, 0x7f5d6ba4b5a8, 0xc214426d00, 0xc253848900) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214426d00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 86407 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc210e95aa0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc210e95aa0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc210e95a40, 0xc2376f0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc211d65208) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc215c5ded8, 0xc2376f0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc215c5ded8, 0xc2376f0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc215c5ded8, 0xc2376f0000, 0xfa000, 0xfa000, 0xc215ee0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc219ff45d0, 0x7f5d6b8a1528, 0xc215c5ded8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215c5ded8, 0xc215eea2a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131145 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21632bfd0, 0xc215e76300, 0xc23cc707e0, 0xc23cc70840) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129004 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213656840, 0xc2193163c0, 0xc20cfed0e0, 0xc20cfed140) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129262 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2143207c8, 0xc214042f60, 0xc242581560, 0xc2425815c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130844 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c71a8, 0xc20bcdec00, 0xc238d1cae0, 0xc238d1cb40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131267 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffa248, 0xc2162142a0, 0xc263376060, 0xc2633760c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 89658 [IO wait, 1 minutes]: net.(*pollDesc).Wait(0xc20b4f7410, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(*pollDesc).WaitRead(0xc20b4f7410, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(*netFD).Read(0xc20b4f73b0, 0xc237df0000, 0x6, 0xfa000, 0x0, 0x7f5d6ba40e70, 0xc240d6abf8) /usr/local/go/src/net/fd_unix.go:242 +0x40f net.(*conn).Read(0xc213ed9f38, 0xc237df0000, 0x6, 0xfa000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:121 +0xdc github.com/couchbase/indexing/secondary/transport.fullRead(0x7f5d6b8a1528, 0xc213ed9f38, 0xc237df0000, 0x6, 0xfa000, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:188 +0xba github.com/couchbase/indexing/secondary/transport.Receive(0x7f5d6b8a1528, 0xc213ed9f38, 0xc237df0000, 0xfa000, 0xfa000, 0xc20dbd0000, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:45 +0xce github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0xc21bb0f110, 0x7f5d6b8a1528, 0xc213ed9f38, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:116 +0xb8 github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ed9f38, 0xc20dbdc900) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:217 +0x429 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:134 +0x1a1 goroutine 9131195 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214c762d0, 0xc2134b1e60, 0xc240bbaa80, 0xc240bbaae0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 89653 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc2455003c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc268483f40, 0x7f5d6ba4b5a8, 0xc215a050e0, 0xc21aab19e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc268483f40, 0x7f5d6ba4b5a8, 0xc215a050e0, 0xc21aab19e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a050e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9131220 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214a16db0, 0xc212f95680, 0xc2611e6660, 0xc2611e66c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131045 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215cd3e40, 0xc213b20660, 0xc2612503c0, 0xc261250420) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 77662 [select, 1 minutes]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).getRequestedIndexSnapshot(0xc2080a4100, 0xc241064f00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:543 +0x4db github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).serverCallback(0xc2080a4100, 0xd71d40, 0xc213e7bea0, 0x7f5d6ba4b5a8, 0xc2146170c0, 0xc21aab0fc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:662 +0x396 github.com/couchbase/indexing/secondary/indexer.*scanCoordinator.(github.com/couchbase/indexing/secondary/indexer.serverCallback)·fm(0xd71d40, 0xc213e7bea0, 0x7f5d6ba4b5a8, 0xc2146170c0, 0xc21aab0fc0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:204 +0x59 github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2146170c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:150 +0x4ec created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:108 +0xd7 goroutine 9130825 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2160c4058, 0xc215fd5500, 0xc20b27f5c0, 0xc20b27f620) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130897 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bfa3918, 0xc216274ae0, 0xc2150a62a0, 0xc2150a6300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131031 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21fac0950, 0xc20ed38960, 0xc21758b680, 0xc21758b6e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131189 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2134eec68, 0xc20f56e9c0, 0xc240bba3c0, 0xc240bba420) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131018 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215ffee78, 0xc214350840, 0xc261252720, 0xc261252780) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129951 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215fff788, 0xc215fb2420, 0xc265d91ce0, 0xc265d91d40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129882 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21317f2f8, 0xc2160104e0, 0xc263f37080, 0xc263f370e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131034 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215edacb0, 0xc216403080, 0xc21758b9e0, 0xc21758ba40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131316 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216171a90, 0xc214f803c0, 0xc263f37bc0, 0xc263f37c20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129007 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215366a80, 0xc215eb0060, 0xc20cfed440, 0xc20cfed4a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129487 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2146170c0, 0xc216103f20, 0xc21aab0fc0, 0xc21aab1020) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131066 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b43c20, 0xc213cbcea0, 0xc2150a6cc0, 0xc2150a6d20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131346 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2136fc418, 0xc213b63920, 0xc261253020, 0xc261253080) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130832 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2145e0078, 0xc214f81e00, 0xc20b27fda0, 0xc20b27fe00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129486 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211a48d60, 0xc20d602d20, 0xc21aab0ea0, 0xc21aab0f00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131330 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216266310, 0xc218a91920, 0xc25a450060, 0xc25a4500c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131271 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21533d350, 0xc213685ce0, 0xc2633764e0, 0xc263376540) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131300 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215d0d6c8, 0xc215eea000, 0xc20cfed9e0, 0xc20cfeda40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129006 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2140a6918, 0xc2100cd920, 0xc20cfed320, 0xc20cfed380) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130944 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210b720d0, 0xc216010780, 0xc264c39f20, 0xc264c39f80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9086126 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc212dec098, 0xc21ec91440, 0xc20d0dd560, 0xc20d0dd5c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131212 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216266c68, 0xc2162bda40, 0xc25fa2afc0, 0xc25fa2b020) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131143 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2142efc58, 0xc20baaeae0, 0xc23cc705a0, 0xc23cc70600) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131118 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215ffe528, 0xc21435b5c0, 0xc258bceba0, 0xc258bcec00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131112 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213f0f9c8, 0xc20af5a120, 0xc258bce4e0, 0xc258bce540) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130823 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20af1c830, 0xc20e7fd1a0, 0xc20b27f380, 0xc20b27f3e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131052 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20bfa2020, 0xc219a62000, 0xc261250ba0, 0xc261250c00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130845 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1f1b0, 0xc2151bd0e0, 0xc238d1cc00, 0xc238d1cc60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131177 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214617ed8, 0xc21355ed20, 0xc26243e1e0, 0xc26243e240) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131188 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21622f258, 0xc219a5e540, 0xc240bba2a0, 0xc240bba300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130840 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215b42c20, 0xc215eb1bc0, 0xc238d1c660, 0xc238d1c6c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129759 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c6000, 0xc214a49c80, 0xc23b94dce0, 0xc23b94dd40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131108 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc211a49bb0, 0xc21f7666c0, 0xc258bce060, 0xc258bce0c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131196 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc216266aa8, 0xc213a87140, 0xc240bbaba0, 0xc240bbac00) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129761 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e417e8, 0xc21d3fbd40, 0xc23b94df20, 0xc23b94df80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131025 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215631338, 0xc214363020, 0xc261252f00, 0xc261252f60) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131033 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215edb418, 0xc21544baa0, 0xc21758b8c0, 0xc21758b920) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129005 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214834b30, 0xc2144e1bc0, 0xc20cfed200, 0xc20cfed260) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131219 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2147a0900, 0xc2119ccc00, 0xc2611e6540, 0xc2611e65a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130797 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213ffabd0, 0xc216192180, 0xc264625980, 0xc2646259e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131156 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21ff38b08, 0xc20def6720, 0xc264aea3c0, 0xc264aea420) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129488 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e84900, 0xc214796d20, 0xc21aab10e0, 0xc21aab1140) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131017 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2146172b8, 0xc213982600, 0xc261252600, 0xc261252660) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131002 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc20b044c58, 0xc2162bd8c0, 0xc23b9d2060, 0xc23b9d20c0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131238 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215c5d178, 0xc215eb3800, 0xc21aab17a0, 0xc21aab1800) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130833 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215c5ded8, 0xc215eea2a0, 0xc20b27fec0, 0xc20b27ff20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131109 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc2080c7660, 0xc215ddf680, 0xc258bce180, 0xc258bce1e0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131124 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214271930, 0xc21033bbc0, 0xc26945e0c0, 0xc26945e120) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131004 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21533d600, 0xc213999ce0, 0xc23b9d22a0, 0xc23b9d2300) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131240 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215a050e0, 0xc20e6fdf80, 0xc21aab19e0, 0xc21aab1a40) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131094 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214349d28, 0xc20bd33020, 0xc23e1f1f20, 0xc23e1f1f80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131223 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc214f1e158, 0xc213288a80, 0xc2611e69c0, 0xc2611e6a20) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131027 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc21419a1d8, 0xc20b041260, 0xc21758b200, 0xc21758b260) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131252 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e73238, 0xc21431cfc0, 0xc238d1d3e0, 0xc238d1d440) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9131036 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc210df0838, 0xc21f766c60, 0xc21758bc20, 0xc21758bc80) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9130831 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc215e72b18, 0xc215fd2fc0, 0xc20b27fc80, 0xc20b27fce0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 goroutine 9129935 [select, 1 minutes]: github.com/couchbase/indexing/secondary/queryport.(*Server).monitorClient(0xc20af1a000, 0x7f5d6ba4b5a8, 0xc213656668, 0xc21dea4ea0, 0xc23e1f1740, 0xc23e1f17a0) /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:174 +0x680 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/sherlock-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:149 +0x4a2 rax 0x0 rbx 0x7f5d6becb000 rcx 0xffffffffffffffff rdx 0x6 rdi 0xdb82 rsi 0xdcb3 rbp 0x7f5d6bf0f888 rsp 0x7f4de7fdd788 r8 0xfefefefefefefeff r9 0xfeff092d63646b68 r10 0x8 r11 0x206 r12 0x7f5d6bf0f8d0 r13 0x7f5d6bf0fc20 r14 0x8 r15 0x0 rip 0x33d6e32925 rflags 0x206 cs 0x33 fs 0x0 gs 0x0 [goport] 2015/08/03 20:58:56 /opt/couchbase/bin/indexer terminated: exit status 2 2015-08-03T20:58:56.757Z-07:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -vbuckets=1024 -cluster=127.0.0.1:8091 -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 -storageDir=/data/@2i] 2015-08-03T20:58:56.79Z-07:00 [Info] Indexer::NewIndexer Status INIT 2015-08-03T20:58:56.794Z-07:00 [Info] Setting maxcpus = 24 2015-08-03T20:58:56.794Z-07:00 [Info] Setting buffer block size to 16384 bytes 2015-08-03T20:58:56.794Z-07:00 [Info] Setting log level to Info 2015-08-03T20:58:56.795Z-07:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2015-08-03T20:58:56.834Z-07:00 [Info] [Queryport ":9101"] started ... 2015-08-03T20:58:56.838Z-07:00 [Info] New settings received: {"indexer.settings.bufferPoolBlockSize":16384,"indexer.settings.compaction.check_period":30,"indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.compaction.min_frag":15,"indexer.settings.compaction.min_size":1048576,"indexer.settings.cpuProfFname":"","indexer.settings.cpuProfile":false,"indexer.settings.fast_flush_mode":true,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.largeSnapshotThreshold":200,"indexer.settings.log_level":"info","indexer.settings.maxVbQueueLength":10000,"indexer.settings.max_cpu_percent":2400,"indexer.settings.max_writer_lock_prob":20,"indexer.settings.memProfFname":"","indexer.settings.memProfile":false,"indexer.settings.memory_quota":104857600000,"indexer.settings.persisted_snapshot.interval":3000,"indexer.settings.persisted_snapshot_init_build.interval":5000,"indexer.settings.recovery.max_rollbacks":5,"indexer.settings.scan_timeout":0,"indexer.settings.send_buffer_size":1024,"indexer.settings.sliceBufSize":50000,"indexer.settings.smallSnapshotThreshold":30,"indexer.settings.statsLogDumpInterval":60,"indexer.settings.wal_size":40960,"projector.settings.log_level":"info","queryport.client.settings.poolOverflow":30,"queryport.client.settings.poolSize":1000} 2015-08-03T20:58:56.839Z-07:00 [Info] Setting maxcpus = 24 2015-08-03T20:58:56.839Z-07:00 [Info] Setting log level to Info 2015-08-03T20:59:00.211Z-07:00 [Info] Indexer Id e7:5e:71:29:9:be:f1:36 2015-08-03T20:59:00.252Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.70:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:59:00.253Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.70:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:59:00.253Z-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.100.70:9999 2015-08-03T20:59:00.253Z-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:59:00.253Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.71:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:59:00.254Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.71:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:59:00.254Z-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.100.71:9999 2015-08-03T20:59:00.254Z-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:59:00.254Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.72:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:59:00.255Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.72:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:59:00.255Z-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.100.72:9999 2015-08-03T20:59:00.255Z-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:59:00.256Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.73:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:59:00.256Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.73:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:59:00.256Z-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.100.73:9999 2015-08-03T20:59:00.257Z-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:59:00.28Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.70:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:59:00.281Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.70:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:59:00.281Z-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.100.70:9999 2015-08-03T20:59:00.281Z-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:59:00.281Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.71:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:59:00.282Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.71:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:59:00.282Z-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.100.71:9999 2015-08-03T20:59:00.282Z-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:59:00.282Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.72:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:59:00.282Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.72:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:59:00.282Z-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.100.72:9999 2015-08-03T20:59:00.282Z-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:59:00.282Z-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.100.73:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 2015-08-03T20:59:00.283Z-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.100.73:9999 Topic INIT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36. Err projector.topicMissing 2015-08-03T20:59:00.283Z-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.100.73:9999 2015-08-03T20:59:00.283Z-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2015-08-03T20:59:04.117Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37096" doReceive() ... 2015-08-03T20:59:04.137Z-07:00 [Info] SCAN##1 RESPONSE status:(error = Index not found) 2015-08-03T20:59:04.138Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37096" skip protobuf.EndStreamRequest 2015-08-03T20:59:04.139Z-07:00 [Info] SCAN##2 RESPONSE status:(error = Index not found) 2015-08-03T20:59:04.145Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37096" skip protobuf.EndStreamRequest 2015-08-03T20:59:04.948Z-07:00 [Info] SCAN##3 RESPONSE status:(error = Index not found) 2015-08-03T20:59:04.99Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37096" skip protobuf.EndStreamRequest 2015-08-03T20:59:05.032Z-07:00 [Info] SCAN##4 RESPONSE status:(error = Index not found) 2015-08-03T20:59:05.096Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37096" skip protobuf.EndStreamRequest 2015-08-03T20:59:05.098Z-07:00 [Info] SCAN##5 RESPONSE status:(error = Index not found) 2015-08-03T20:59:05.107Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37101" doReceive() ... 2015-08-03T20:59:05.127Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37102" doReceive() ... 2015-08-03T20:59:05.127Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37103" doReceive() ... 2015-08-03T20:59:05.153Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37104" doReceive() ... 2015-08-03T20:59:05.153Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37105" doReceive() ... 2015-08-03T20:59:05.153Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37106" doReceive() ... 2015-08-03T20:59:05.158Z-07:00 [Info] SCAN##6 RESPONSE status:(error = Index not found) 2015-08-03T20:59:05.16Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37096" skip protobuf.EndStreamRequest 2015-08-03T20:59:05.167Z-07:00 [Info] SCAN##7 RESPONSE status:(error = Index not found) 2015-08-03T20:59:05.168Z-07:00 [Info] SCAN##8 RESPONSE status:(error = Index not found) 2015-08-03T20:59:05.168Z-07:00 [Info] SCAN##9 RESPONSE status:(error = Index not found) 2015-08-03T20:59:05.177Z-07:00 [Info] SCAN##10 RESPONSE status:(error = Index not found) 2015-08-03T20:59:05.177Z-07:00 [Info] SCAN##11 RESPONSE status:(error = Index not found) 2015-08-03T20:59:05.177Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37101" skip protobuf.EndStreamRequest 2015-08-03T20:59:05.177Z-07:00 [Info] SCAN##12 RESPONSE status:(error = Index not found) 2015-08-03T20:59:05.178Z-07:00 [Info] SCAN##13 RESPONSE status:(error = Index not found) 2015-08-03T20:59:05.181Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37102" skip protobuf.EndStreamRequest 2015-08-03T20:59:05.181Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37103" skip protobuf.EndStreamRequest 2015-08-03T20:59:05.182Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37096" skip protobuf.EndStreamRequest 2015-08-03T20:59:05.187Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37101" skip protobuf.EndStreamRequest 2015-08-03T20:59:05.188Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37104" skip protobuf.EndStreamRequest 2015-08-03T20:59:05.188Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37106" skip protobuf.EndStreamRequest 2015-08-03T20:59:05.189Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37105" skip protobuf.EndStreamRequest 2015-08-03T20:59:05.19Z-07:00 [Info] SCAN##14 RESPONSE status:(error = Index not found) 2015-08-03T20:59:05.191Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37102" skip protobuf.EndStreamRequest 2015-08-03T20:59:05.191Z-07:00 [Info] SCAN##15 RESPONSE status:(error = Index not found) 2015-08-03T20:59:05.192Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37103" skip protobuf.EndStreamRequest 2015-08-03T20:59:05.452Z-07:00 [Info] SCAN##16 RESPONSE status:(error = Index not found) 2015-08-03T20:59:05.452Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37096" skip protobuf.EndStreamRequest 2015-08-03T20:59:05.543Z-07:00 [Info] SCAN##17 RESPONSE status:(error = Index not found) 2015-08-03T20:59:05.544Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37101" skip protobuf.EndStreamRequest 2015-08-03T20:59:05.723Z-07:00 [Info] SCAN##18 RESPONSE status:(error = Index not found) 2015-08-03T20:59:05.723Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37104" skip protobuf.EndStreamRequest 2015-08-03T20:59:05.858Z-07:00 [Info] SCAN##19 RESPONSE status:(error = Index not found) 2015-08-03T20:59:05.861Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37106" skip protobuf.EndStreamRequest 2015-08-03T20:59:05.912Z-07:00 [Info] SCAN##20 RESPONSE status:(error = Index not found) 2015-08-03T20:59:05.913Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37105" skip protobuf.EndStreamRequest 2015-08-03T20:59:05.972Z-07:00 [Info] SCAN##21 RESPONSE status:(error = Index not found) 2015-08-03T20:59:05.976Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37102" skip protobuf.EndStreamRequest 2015-08-03T20:59:06.289Z-07:00 [Info] SCAN##22 RESPONSE status:(error = Index not found) 2015-08-03T20:59:06.29Z-07:00 [Info] DCPT[secidx:getseqnos-36:f1:b:b0:8b:64:bf:8c] ##abba feed started ... 2015-08-03T20:59:06.299Z-07:00 [Info] DCPT[secidx:getseqnos-43:83:32:af:97:5c:af:5a] ##abba feed started ... 2015-08-03T20:59:06.303Z-07:00 [Info] DCPT[secidx:getseqnos-74:55:f4:19:2:1:ec:f6] ##abba feed started ... 2015-08-03T20:59:06.323Z-07:00 [Info] SCAN##23 RESPONSE status:(error = Index not found) 2015-08-03T20:59:06.496Z-07:00 [Info] SCAN##24 RESPONSE status:(error = Index not found) 2015-08-03T20:59:06.556Z-07:00 [Info] SCAN##25 RESPONSE status:(error = Index not found) 2015-08-03T20:59:06.56Z-07:00 [Info] DCPT[secidx:getseqnos-64:96:16:f:22:18:4a:62] ##abba feed started ... 2015-08-03T20:59:06.561Z-07:00 [Info] {bucket,feeds} "bucket-1" created for dcp_seqno cache... 2015-08-03T20:59:06.564Z-07:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 16558345882944865108 Partition: PartitionId: 0 Endpoints: [:9105] 2015-08-03T20:59:06.572Z-07:00 [Info] Indexer::initPartnInstance Initialized Slice: Index: 16558345882944865108 Slice: SliceId: 0 File: /data/@2i/bucket-1_by_email_16558345882944865108_0.index Index: 16558345882944865108 2015-08-03T20:59:06.595Z-07:00 [Info] StorageMgr::updateIndexSnapMap IndexInst:16558345882944865108 Attempting to open snapshot (SnapshotInfo: seqnos: 22574198, 11787099, 367 committed:true) 2015-08-03T20:59:06.596Z-07:00 [Info] MutationMgr::handleUpdateIndexInstMap Message: MsgUpdateInstMap InstanceId: 16558345882944865108 Name: by_email Bucket: bucket-1 State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM 2015-08-03T20:59:06.596Z-07:00 [Info] MutationMgr::handleUpdateIndexPartnMap Message: MsgUpdatePartnMap InstanceId: 16558345882944865108 PartitionId: 0 Endpoints: [:9105] 2015-08-03T20:59:06.62Z-07:00 [Info] DATP[->dataport ":9105"] started ... 2015-08-03T20:59:06.621Z-07:00 [Info] Indexer::NewIndexer Status ACTIVE 2015-08-03T20:59:06.621Z-07:00 [Info] CompactionManager: Refreshing settings 2015-08-03T20:59:06.659Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37103" skip protobuf.EndStreamRequest 2015-08-03T20:59:06.866Z-07:00 [Info] KVSender::sendMutationTopicRequest Projector 172.23.100.70:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 bucket-1 Instances [indexInstance: singlePartn: > ] 2015-08-03T20:59:06.872Z-07:00 [Info] DATP[->dataport ":9105"] new connection "172.23.100.70:45862" +1 2015-08-03T20:59:07.078Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1 2015-08-03T20:59:07.199Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37101" skip protobuf.EndStreamRequest 2015-08-03T20:59:07.2Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37096" skip protobuf.EndStreamRequest 2015-08-03T20:59:07.241Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 1 QueuedCount 127 2015-08-03T20:59:07.306Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37104" skip protobuf.EndStreamRequest 2015-08-03T20:59:07.353Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22578350, 11789175, 0 committed:false) SnapCreateDur 24.45µs SnapOpenDur 14.050207ms 2015-08-03T20:59:07.482Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10000 2015-08-03T20:59:07.498Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37166" doReceive() ... 2015-08-03T20:59:07.511Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37167" doReceive() ... 2015-08-03T20:59:07.55Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37168" doReceive() ... 2015-08-03T20:59:07.55Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37170" doReceive() ... 2015-08-03T20:59:07.55Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37172" doReceive() ... 2015-08-03T20:59:07.551Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37171" doReceive() ... 2015-08-03T20:59:07.551Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37173" doReceive() ... 2015-08-03T20:59:07.551Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37174" doReceive() ... 2015-08-03T20:59:07.551Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37175" doReceive() ... 2015-08-03T20:59:07.551Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37177" doReceive() ... 2015-08-03T20:59:07.551Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37176" doReceive() ... 2015-08-03T20:59:07.586Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37178" doReceive() ... 2015-08-03T20:59:07.586Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37179" doReceive() ... 2015-08-03T20:59:07.551Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37169" doReceive() ... 2015-08-03T20:59:07.586Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37180" doReceive() ... 2015-08-03T20:59:07.586Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37181" doReceive() ... 2015-08-03T20:59:07.586Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37183" doReceive() ... 2015-08-03T20:59:07.587Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37182" doReceive() ... 2015-08-03T20:59:07.63Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37184" doReceive() ... 2015-08-03T20:59:07.63Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37185" doReceive() ... 2015-08-03T20:59:07.63Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37186" doReceive() ... 2015-08-03T20:59:07.63Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37187" doReceive() ... 2015-08-03T20:59:07.63Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37188" doReceive() ... 2015-08-03T20:59:07.63Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37189" doReceive() ... 2015-08-03T20:59:07.63Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37190" doReceive() ... 2015-08-03T20:59:07.63Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37195" doReceive() ... 2015-08-03T20:59:07.63Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37197" doReceive() ... 2015-08-03T20:59:07.63Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37193" doReceive() ... 2015-08-03T20:59:07.631Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37194" doReceive() ... 2015-08-03T20:59:07.631Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37196" doReceive() ... 2015-08-03T20:59:07.631Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37191" doReceive() ... 2015-08-03T20:59:07.631Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37192" doReceive() ... 2015-08-03T20:59:07.638Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37198" doReceive() ... 2015-08-03T20:59:07.638Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37199" doReceive() ... 2015-08-03T20:59:07.638Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37200" doReceive() ... 2015-08-03T20:59:07.638Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37202" doReceive() ... 2015-08-03T20:59:07.638Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37203" doReceive() ... 2015-08-03T20:59:07.638Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37207" doReceive() ... 2015-08-03T20:59:07.638Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37204" doReceive() ... 2015-08-03T20:59:07.638Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37209" doReceive() ... 2015-08-03T20:59:07.638Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37206" doReceive() ... 2015-08-03T20:59:07.638Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37201" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37208" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37210" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37205" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37211" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37213" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37212" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37219" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37216" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37215" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37221" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37217" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37214" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37220" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37223" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37218" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37222" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37224" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37228" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37227" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37225" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37226" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37229" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37230" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37232" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37231" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37233" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37234" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37235" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37236" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37237" doReceive() ... 2015-08-03T20:59:07.639Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37240" doReceive() ... 2015-08-03T20:59:07.64Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37238" doReceive() ... 2015-08-03T20:59:07.64Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37239" doReceive() ... 2015-08-03T20:59:07.64Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37241" doReceive() ... 2015-08-03T20:59:07.64Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37244" doReceive() ... 2015-08-03T20:59:07.64Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37243" doReceive() ... 2015-08-03T20:59:07.64Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37246" doReceive() ... 2015-08-03T20:59:07.64Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37249" doReceive() ... 2015-08-03T20:59:07.64Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37250" doReceive() ... 2015-08-03T20:59:07.64Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37242" doReceive() ... 2015-08-03T20:59:07.64Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37247" doReceive() ... 2015-08-03T20:59:07.64Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37245" doReceive() ... 2015-08-03T20:59:07.64Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37253" doReceive() ... 2015-08-03T20:59:07.64Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37254" doReceive() ... 2015-08-03T20:59:07.64Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37255" doReceive() ... 2015-08-03T20:59:07.64Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37251" doReceive() ... 2015-08-03T20:59:07.64Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37252" doReceive() ... 2015-08-03T20:59:07.64Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37257" doReceive() ... 2015-08-03T20:59:07.659Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37256" doReceive() ... 2015-08-03T20:59:07.64Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37248" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37261" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37288" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37259" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37280" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37260" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37264" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37276" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37266" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37274" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37273" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37281" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37277" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37262" doReceive() ... 2015-08-03T20:59:07.662Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37268" doReceive() ... 2015-08-03T20:59:07.662Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37284" doReceive() ... 2015-08-03T20:59:07.662Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37275" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37267" doReceive() ... 2015-08-03T20:59:07.662Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37285" doReceive() ... 2015-08-03T20:59:07.662Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37289" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37265" doReceive() ... 2015-08-03T20:59:07.662Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37286" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37271" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37272" doReceive() ... 2015-08-03T20:59:07.662Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37287" doReceive() ... 2015-08-03T20:59:07.662Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37294" doReceive() ... 2015-08-03T20:59:07.662Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37269" doReceive() ... 2015-08-03T20:59:07.662Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37293" doReceive() ... 2015-08-03T20:59:07.662Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37291" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37263" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37258" doReceive() ... 2015-08-03T20:59:07.662Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37270" doReceive() ... 2015-08-03T20:59:07.662Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37279" doReceive() ... 2015-08-03T20:59:07.662Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37295" doReceive() ... 2015-08-03T20:59:07.662Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37290" doReceive() ... 2015-08-03T20:59:07.662Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37282" doReceive() ... 2015-08-03T20:59:07.661Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37278" doReceive() ... 2015-08-03T20:59:07.662Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37283" doReceive() ... 2015-08-03T20:59:07.663Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37292" doReceive() ... 2015-08-03T20:59:07.772Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37297" doReceive() ... 2015-08-03T20:59:07.771Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37296" doReceive() ... 2015-08-03T20:59:07.772Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37298" doReceive() ... 2015-08-03T20:59:07.772Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37299" doReceive() ... 2015-08-03T20:59:07.772Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37302" doReceive() ... 2015-08-03T20:59:07.772Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37301" doReceive() ... 2015-08-03T20:59:07.772Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37303" doReceive() ... 2015-08-03T20:59:07.772Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37300" doReceive() ... 2015-08-03T20:59:07.773Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37304" doReceive() ... 2015-08-03T20:59:07.773Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37305" doReceive() ... 2015-08-03T20:59:07.773Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37306" doReceive() ... 2015-08-03T20:59:07.774Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37307" doReceive() ... 2015-08-03T20:59:07.774Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37308" doReceive() ... 2015-08-03T20:59:07.774Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37309" doReceive() ... 2015-08-03T20:59:07.774Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37310" doReceive() ... 2015-08-03T20:59:07.775Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37311" doReceive() ... 2015-08-03T20:59:07.775Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37312" doReceive() ... 2015-08-03T20:59:07.776Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37313" doReceive() ... 2015-08-03T20:59:07.776Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37314" doReceive() ... 2015-08-03T20:59:07.778Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37315" doReceive() ... 2015-08-03T20:59:07.778Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37316" doReceive() ... 2015-08-03T20:59:07.778Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37317" doReceive() ... 2015-08-03T20:59:07.782Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37318" doReceive() ... 2015-08-03T20:59:07.782Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37319" doReceive() ... 2015-08-03T20:59:07.782Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37320" doReceive() ... 2015-08-03T20:59:07.782Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37321" doReceive() ... 2015-08-03T20:59:07.783Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37322" doReceive() ... 2015-08-03T20:59:07.783Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37323" doReceive() ... 2015-08-03T20:59:07.783Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37324" doReceive() ... 2015-08-03T20:59:07.783Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37328" doReceive() ... 2015-08-03T20:59:07.783Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37325" doReceive() ... 2015-08-03T20:59:07.783Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37326" doReceive() ... 2015-08-03T20:59:07.783Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37329" doReceive() ... 2015-08-03T20:59:07.783Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37327" doReceive() ... 2015-08-03T20:59:07.783Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37333" doReceive() ... 2015-08-03T20:59:07.783Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37330" doReceive() ... 2015-08-03T20:59:07.783Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37334" doReceive() ... 2015-08-03T20:59:07.783Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37332" doReceive() ... 2015-08-03T20:59:07.783Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37331" doReceive() ... 2015-08-03T20:59:07.784Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37335" doReceive() ... 2015-08-03T20:59:07.784Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37336" doReceive() ... 2015-08-03T20:59:07.784Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37338" doReceive() ... 2015-08-03T20:59:07.784Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37337" doReceive() ... 2015-08-03T20:59:07.784Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37339" doReceive() ... 2015-08-03T20:59:07.784Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37341" doReceive() ... 2015-08-03T20:59:07.784Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37340" doReceive() ... 2015-08-03T20:59:07.784Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37342" doReceive() ... 2015-08-03T20:59:07.784Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37344" doReceive() ... 2015-08-03T20:59:07.784Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37343" doReceive() ... 2015-08-03T20:59:07.784Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37345" doReceive() ... 2015-08-03T20:59:07.784Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37346" doReceive() ... 2015-08-03T20:59:07.784Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37347" doReceive() ... 2015-08-03T20:59:07.784Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37350" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37351" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37352" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37349" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37348" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37353" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37354" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37356" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37355" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37358" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37357" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37359" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37360" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37361" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37362" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37365" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37366" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37364" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37367" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37363" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37369" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37368" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37370" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37371" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37372" doReceive() ... 2015-08-03T20:59:07.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37373" doReceive() ... 2015-08-03T20:59:07.786Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37374" doReceive() ... 2015-08-03T20:59:07.786Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37376" doReceive() ... 2015-08-03T20:59:07.786Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37375" doReceive() ... 2015-08-03T20:59:07.786Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37377" doReceive() ... 2015-08-03T20:59:07.786Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37379" doReceive() ... 2015-08-03T20:59:07.786Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37378" doReceive() ... 2015-08-03T20:59:07.786Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37380" doReceive() ... 2015-08-03T20:59:07.786Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37382" doReceive() ... 2015-08-03T20:59:07.786Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37381" doReceive() ... 2015-08-03T20:59:07.786Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37383" doReceive() ... 2015-08-03T20:59:07.786Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37384" doReceive() ... 2015-08-03T20:59:07.786Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37385" doReceive() ... 2015-08-03T20:59:07.786Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37386" doReceive() ... 2015-08-03T20:59:07.788Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37387" doReceive() ... 2015-08-03T20:59:07.788Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37388" doReceive() ... 2015-08-03T20:59:07.788Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37389" doReceive() ... 2015-08-03T20:59:07.826Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37390" doReceive() ... 2015-08-03T20:59:07.826Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22588654, 11794327, 0 committed:false) SnapCreateDur 29.886µs SnapOpenDur 87.865816ms 2015-08-03T20:59:07.826Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37391" doReceive() ... 2015-08-03T20:59:07.83Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37463" doReceive() ... 2015-08-03T20:59:07.83Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37466" doReceive() ... 2015-08-03T20:59:07.83Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37480" doReceive() ... 2015-08-03T20:59:07.83Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37446" doReceive() ... 2015-08-03T20:59:07.83Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37412" doReceive() ... 2015-08-03T20:59:07.83Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37443" doReceive() ... 2015-08-03T20:59:07.83Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37503" doReceive() ... 2015-08-03T20:59:07.83Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37496" doReceive() ... 2015-08-03T20:59:07.83Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37454" doReceive() ... 2015-08-03T20:59:07.83Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37460" doReceive() ... 2015-08-03T20:59:07.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37444" doReceive() ... 2015-08-03T20:59:07.83Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37458" doReceive() ... 2015-08-03T20:59:07.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37462" doReceive() ... 2015-08-03T20:59:07.83Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37476" doReceive() ... 2015-08-03T20:59:07.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37442" doReceive() ... 2015-08-03T20:59:07.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37495" doReceive() ... 2015-08-03T20:59:07.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37419" doReceive() ... 2015-08-03T20:59:07.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37420" doReceive() ... 2015-08-03T20:59:07.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37445" doReceive() ... 2015-08-03T20:59:07.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37449" doReceive() ... 2015-08-03T20:59:07.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37439" doReceive() ... 2015-08-03T20:59:07.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37434" doReceive() ... 2015-08-03T20:59:07.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37406" doReceive() ... 2015-08-03T20:59:07.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37421" doReceive() ... 2015-08-03T20:59:07.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37407" doReceive() ... 2015-08-03T20:59:07.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37422" doReceive() ... 2015-08-03T20:59:07.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37438" doReceive() ... 2015-08-03T20:59:07.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37459" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37441" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37451" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37423" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37424" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37425" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37497" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37428" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37464" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37455" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37410" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37409" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37450" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37405" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37426" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37429" doReceive() ... 2015-08-03T20:59:07.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37408" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37513" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37468" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37461" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37507" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37452" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37447" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37470" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37453" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37465" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37514" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37411" doReceive() ... 2015-08-03T20:59:07.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37440" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37448" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37498" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37456" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37392" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37413" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37499" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37435" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37414" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37504" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37436" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37415" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37430" doReceive() ... 2015-08-03T20:59:07.834Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37437" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37469" doReceive() ... 2015-08-03T20:59:07.834Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37471" doReceive() ... 2015-08-03T20:59:07.834Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37481" doReceive() ... 2015-08-03T20:59:07.834Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37432" doReceive() ... 2015-08-03T20:59:07.834Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37433" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37505" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37427" doReceive() ... 2015-08-03T20:59:07.834Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37502" doReceive() ... 2015-08-03T20:59:07.834Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37508" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37500" doReceive() ... 2015-08-03T20:59:07.834Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37501" doReceive() ... 2015-08-03T20:59:07.834Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37509" doReceive() ... 2015-08-03T20:59:07.834Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37478" doReceive() ... 2015-08-03T20:59:07.834Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37482" doReceive() ... 2015-08-03T20:59:07.834Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37416" doReceive() ... 2015-08-03T20:59:07.835Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37417" doReceive() ... 2015-08-03T20:59:07.835Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37510" doReceive() ... 2015-08-03T20:59:07.833Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37477" doReceive() ... 2015-08-03T20:59:07.835Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37472" doReceive() ... 2015-08-03T20:59:07.835Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37506" doReceive() ... 2015-08-03T20:59:07.835Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37473" doReceive() ... 2015-08-03T20:59:07.836Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37483" doReceive() ... 2015-08-03T20:59:07.836Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37515" doReceive() ... 2015-08-03T20:59:07.836Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37516" doReceive() ... 2015-08-03T20:59:07.835Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37511" doReceive() ... 2015-08-03T20:59:07.836Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37517" doReceive() ... 2015-08-03T20:59:07.836Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37512" doReceive() ... 2015-08-03T20:59:07.832Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37467" doReceive() ... 2015-08-03T20:59:07.836Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37479" doReceive() ... 2015-08-03T20:59:07.836Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37393" doReceive() ... 2015-08-03T20:59:07.837Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37400" doReceive() ... 2015-08-03T20:59:07.837Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37484" doReceive() ... 2015-08-03T20:59:07.837Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37396" doReceive() ... 2015-08-03T20:59:07.835Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37457" doReceive() ... 2015-08-03T20:59:07.837Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37474" doReceive() ... 2015-08-03T20:59:07.838Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37492" doReceive() ... 2015-08-03T20:59:07.838Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37485" doReceive() ... 2015-08-03T20:59:07.838Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37401" doReceive() ... 2015-08-03T20:59:07.838Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37475" doReceive() ... 2015-08-03T20:59:07.834Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37431" doReceive() ... 2015-08-03T20:59:07.838Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37394" doReceive() ... 2015-08-03T20:59:07.838Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37494" doReceive() ... 2015-08-03T20:59:07.838Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37493" doReceive() ... 2015-08-03T20:59:07.839Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37398" doReceive() ... 2015-08-03T20:59:07.839Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37395" doReceive() ... 2015-08-03T20:59:07.839Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37399" doReceive() ... 2015-08-03T20:59:07.839Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37402" doReceive() ... 2015-08-03T20:59:07.839Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37397" doReceive() ... 2015-08-03T20:59:07.835Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37418" doReceive() ... 2015-08-03T20:59:07.839Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37486" doReceive() ... 2015-08-03T20:59:07.84Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37489" doReceive() ... 2015-08-03T20:59:07.84Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37487" doReceive() ... 2015-08-03T20:59:07.84Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37403" doReceive() ... 2015-08-03T20:59:07.84Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37404" doReceive() ... 2015-08-03T20:59:07.84Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37488" doReceive() ... 2015-08-03T20:59:07.84Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37490" doReceive() ... 2015-08-03T20:59:07.84Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37491" doReceive() ... 2015-08-03T20:59:07.846Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37518" doReceive() ... 2015-08-03T20:59:07.846Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37519" doReceive() ... 2015-08-03T20:59:07.846Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37520" doReceive() ... 2015-08-03T20:59:07.846Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37521" doReceive() ... 2015-08-03T20:59:07.846Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37522" doReceive() ... 2015-08-03T20:59:07.846Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37523" doReceive() ... 2015-08-03T20:59:07.851Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37528" doReceive() ... 2015-08-03T20:59:07.854Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20000 2015-08-03T20:59:07.854Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37529" doReceive() ... 2015-08-03T20:59:07.859Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37530" doReceive() ... 2015-08-03T20:59:07.859Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37531" doReceive() ... 2015-08-03T20:59:07.873Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37536" doReceive() ... 2015-08-03T20:59:07.875Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37537" doReceive() ... 2015-08-03T20:59:07.917Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37538" doReceive() ... 2015-08-03T20:59:07.92Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37539" doReceive() ... 2015-08-03T20:59:07.927Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 10000 QueuedCount 1967 2015-08-03T20:59:07.981Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37540" doReceive() ... 2015-08-03T20:59:07.984Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37541" doReceive() ... 2015-08-03T20:59:08.018Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37542" doReceive() ... 2015-08-03T20:59:08.04Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22597754, 11798877, 0 committed:false) SnapCreateDur 13.533µs SnapOpenDur 57.467176ms 2015-08-03T20:59:08.057Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37543" doReceive() ... 2015-08-03T20:59:08.088Z-07:00 [Info] KVSender::sendMutationTopicRequest Projector 172.23.100.71:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 bucket-1 Instances [indexInstance: singlePartn: > ] 2015-08-03T20:59:08.093Z-07:00 [Info] DATP[->dataport ":9105"] new connection "172.23.100.71:47570" +2 2015-08-03T20:59:08.117Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37544" doReceive() ... 2015-08-03T20:59:08.123Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37545" doReceive() ... 2015-08-03T20:59:08.137Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37546" doReceive() ... 2015-08-03T20:59:08.151Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37547" doReceive() ... 2015-08-03T20:59:08.163Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30000 2015-08-03T20:59:08.177Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37548" doReceive() ... 2015-08-03T20:59:08.202Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37549" doReceive() ... 2015-08-03T20:59:08.217Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37550" doReceive() ... 2015-08-03T20:59:08.424Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 40000 2015-08-03T20:59:08.63Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 50000 2015-08-03T20:59:08.785Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 60000 2015-08-03T20:59:08.921Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 70000 2015-08-03T20:59:09.062Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 80000 2015-08-03T20:59:09.218Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 90000 2015-08-03T20:59:09.265Z-07:00 [Info] KVSender::sendMutationTopicRequest Projector 172.23.100.72:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 bucket-1 Instances [indexInstance: singlePartn: > ] 2015-08-03T20:59:09.27Z-07:00 [Info] DATP[->dataport ":9105"] new connection "172.23.100.72:34134" +3 2015-08-03T20:59:09.398Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22611192, 11805596, 0 committed:false) SnapCreateDur 11.389µs SnapOpenDur 79.809861ms 2015-08-03T20:59:09.402Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 100000 2015-08-03T20:59:09.452Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 20000 QueuedCount 10188 2015-08-03T20:59:09.629Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 110000 2015-08-03T20:59:09.778Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 120000 2015-08-03T20:59:09.941Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 130000 2015-08-03T20:59:10.129Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 140000 2015-08-03T20:59:10.311Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 150000 2015-08-03T20:59:10.354Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 30000 QueuedCount 1238 2015-08-03T20:59:10.471Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 160000 2015-08-03T20:59:10.509Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22635660, 11817830, 0 committed:false) SnapCreateDur 18.424µs SnapOpenDur 124.195002ms 2015-08-03T20:59:10.546Z-07:00 [Info] KVSender::sendMutationTopicRequest Projector 172.23.100.73:9999 Topic MAINT_STREAM_TOPIC_e7:5e:71:29:9:be:f1:36 bucket-1 Instances [indexInstance: singlePartn: > ] 2015-08-03T20:59:10.55Z-07:00 [Info] DATP[->dataport ":9105"] new connection "172.23.100.73:33234" +4 2015-08-03T20:59:10.728Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 170000 2015-08-03T20:59:10.859Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 180000 2015-08-03T20:59:10.988Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 190000 2015-08-03T20:59:11.113Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 200000 2015-08-03T20:59:11.236Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 210000 2015-08-03T20:59:11.359Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 220000 2015-08-03T20:59:11.375Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.195µs CommitTime 630.738119ms TotalFlushTime 2.870281362s TotalCommitTime 630.738119ms 2015-08-03T20:59:11.375Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22649724, 11824862, 368 committed:true) SnapCreateDur 647.185767ms SnapOpenDur 29.995µs 2015-08-03T20:59:11.424Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 40000 QueuedCount 7730 2015-08-03T20:59:11.511Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 230000 2015-08-03T20:59:11.595Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 50000 QueuedCount 26859 2015-08-03T20:59:11.762Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 240000 2015-08-03T20:59:11.995Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 250000 2015-08-03T20:59:12.168Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 260000 2015-08-03T20:59:12.307Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 60000 QueuedCount 16859 2015-08-03T20:59:12.33Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 270000 2015-08-03T20:59:12.461Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 280000 2015-08-03T20:59:12.594Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 290000 2015-08-03T20:59:12.951Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 300000 2015-08-03T20:59:13.009Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 70000 QueuedCount 6859 2015-08-03T20:59:13.22Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22725396, 11862698, 0 committed:false) SnapCreateDur 11.22µs SnapOpenDur 93.1339ms 2015-08-03T20:59:13.313Z-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 310000 2015-08-03T20:59:13.373Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 80000 QueuedCount 29559 2015-08-03T20:59:14.528Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 90000 QueuedCount 19559 2015-08-03T20:59:15.343Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 100000 QueuedCount 9559 2015-08-03T20:59:16.142Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22789802, 11894901, 0 committed:false) SnapCreateDur 11.057µs SnapOpenDur 65.03662ms 2015-08-03T20:59:16.173Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 110000 QueuedCount 10507 2015-08-03T20:59:16.328Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 120000 QueuedCount 507 2015-08-03T20:59:17.571Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.89µs CommitTime 1.200151767s TotalFlushTime 7.551096792s TotalCommitTime 1.830889886s 2015-08-03T20:59:17.571Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22811310, 11905655, 369 committed:true) SnapCreateDur 1.228805593s SnapOpenDur 45.041µs 2015-08-03T20:59:17.804Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 130000 QueuedCount 28465 2015-08-03T20:59:18.655Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 140000 QueuedCount 18465 2015-08-03T20:59:19.35Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 150000 QueuedCount 8465 2015-08-03T20:59:19.568Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22886028, 11943014, 0 committed:false) SnapCreateDur 20.15µs SnapOpenDur 90.389013ms 2015-08-03T20:59:19.649Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 160000 QueuedCount 7162 2015-08-03T20:59:21.112Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.715µs CommitTime 351.334151ms TotalFlushTime 10.557748217s TotalCommitTime 2.182224037s 2015-08-03T20:59:21.112Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22903128, 11951564, 370 committed:true) SnapCreateDur 365.299013ms SnapOpenDur 34.256µs 2015-08-03T20:59:21.188Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 170000 QueuedCount 22440 2015-08-03T20:59:21.345Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 180000 QueuedCount 26315 2015-08-03T20:59:22.02Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 190000 QueuedCount 16315 2015-08-03T20:59:22.693Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 200000 QueuedCount 6315 2015-08-03T20:59:22.888Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 22980204, 11990102, 0 committed:false) SnapCreateDur 20.203µs SnapOpenDur 92.113114ms 2015-08-03T20:59:23.734Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37551" doReceive() ... 2015-08-03T20:59:23.737Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37552" doReceive() ... 2015-08-03T20:59:23.737Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37553" doReceive() ... 2015-08-03T20:59:23.737Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37554" doReceive() ... 2015-08-03T20:59:23.737Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37555" doReceive() ... 2015-08-03T20:59:23.737Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37556" doReceive() ... 2015-08-03T20:59:23.737Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37557" doReceive() ... 2015-08-03T20:59:23.74Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37558" doReceive() ... 2015-08-03T20:59:23.74Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37560" doReceive() ... 2015-08-03T20:59:23.74Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37561" doReceive() ... 2015-08-03T20:59:23.74Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37562" doReceive() ... 2015-08-03T20:59:23.74Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37559" doReceive() ... 2015-08-03T20:59:23.74Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37563" doReceive() ... 2015-08-03T20:59:23.74Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37567" doReceive() ... 2015-08-03T20:59:23.74Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37564" doReceive() ... 2015-08-03T20:59:23.74Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37565" doReceive() ... 2015-08-03T20:59:23.74Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37566" doReceive() ... 2015-08-03T20:59:23.743Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37568" doReceive() ... 2015-08-03T20:59:23.746Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37570" doReceive() ... 2015-08-03T20:59:23.746Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37569" doReceive() ... 2015-08-03T20:59:23.757Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37571" doReceive() ... 2015-08-03T20:59:23.759Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37573" doReceive() ... 2015-08-03T20:59:23.76Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37574" doReceive() ... 2015-08-03T20:59:23.759Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37572" doReceive() ... 2015-08-03T20:59:23.762Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37575" doReceive() ... 2015-08-03T20:59:23.762Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37576" doReceive() ... 2015-08-03T20:59:23.765Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37577" doReceive() ... 2015-08-03T20:59:23.768Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37578" doReceive() ... 2015-08-03T20:59:23.768Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37582" doReceive() ... 2015-08-03T20:59:23.768Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37579" doReceive() ... 2015-08-03T20:59:23.768Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37583" doReceive() ... 2015-08-03T20:59:23.768Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37581" doReceive() ... 2015-08-03T20:59:23.768Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37580" doReceive() ... 2015-08-03T20:59:23.772Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37584" doReceive() ... 2015-08-03T20:59:23.777Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37585" doReceive() ... 2015-08-03T20:59:23.78Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37586" doReceive() ... 2015-08-03T20:59:23.78Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37587" doReceive() ... 2015-08-03T20:59:23.78Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37590" doReceive() ... 2015-08-03T20:59:23.78Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37589" doReceive() ... 2015-08-03T20:59:23.78Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37588" doReceive() ... 2015-08-03T20:59:23.78Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37591" doReceive() ... 2015-08-03T20:59:23.782Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37592" doReceive() ... 2015-08-03T20:59:23.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37593" doReceive() ... 2015-08-03T20:59:23.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37594" doReceive() ... 2015-08-03T20:59:23.785Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37595" doReceive() ... 2015-08-03T20:59:23.788Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37596" doReceive() ... 2015-08-03T20:59:23.788Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37597" doReceive() ... 2015-08-03T20:59:23.791Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37598" doReceive() ... 2015-08-03T20:59:23.794Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37599" doReceive() ... 2015-08-03T20:59:23.794Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37600" doReceive() ... 2015-08-03T20:59:23.794Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37601" doReceive() ... 2015-08-03T20:59:23.794Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37602" doReceive() ... 2015-08-03T20:59:23.794Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37603" doReceive() ... 2015-08-03T20:59:23.797Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37604" doReceive() ... 2015-08-03T20:59:23.797Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37605" doReceive() ... 2015-08-03T20:59:23.8Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37606" doReceive() ... 2015-08-03T20:59:23.8Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37607" doReceive() ... 2015-08-03T20:59:23.808Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37608" doReceive() ... 2015-08-03T20:59:23.81Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37609" doReceive() ... 2015-08-03T20:59:23.81Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37611" doReceive() ... 2015-08-03T20:59:23.81Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37610" doReceive() ... 2015-08-03T20:59:23.811Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37612" doReceive() ... 2015-08-03T20:59:23.814Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37613" doReceive() ... 2015-08-03T20:59:23.817Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37614" doReceive() ... 2015-08-03T20:59:23.817Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37615" doReceive() ... 2015-08-03T20:59:23.817Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37616" doReceive() ... 2015-08-03T20:59:23.819Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37617" doReceive() ... 2015-08-03T20:59:23.823Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37618" doReceive() ... 2015-08-03T20:59:23.826Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37619" doReceive() ... 2015-08-03T20:59:23.826Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37620" doReceive() ... 2015-08-03T20:59:23.826Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37621" doReceive() ... 2015-08-03T20:59:23.828Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37622" doReceive() ... 2015-08-03T20:59:23.828Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37623" doReceive() ... 2015-08-03T20:59:23.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37625" doReceive() ... 2015-08-03T20:59:23.831Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37624" doReceive() ... 2015-08-03T20:59:23.834Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37626" doReceive() ... 2015-08-03T20:59:23.838Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37627" doReceive() ... 2015-08-03T20:59:23.841Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37628" doReceive() ... 2015-08-03T20:59:23.841Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37629" doReceive() ... 2015-08-03T20:59:23.841Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37630" doReceive() ... 2015-08-03T20:59:23.841Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37632" doReceive() ... 2015-08-03T20:59:23.841Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37634" doReceive() ... 2015-08-03T20:59:23.841Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37631" doReceive() ... 2015-08-03T20:59:23.841Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37636" doReceive() ... 2015-08-03T20:59:23.841Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37635" doReceive() ... 2015-08-03T20:59:23.841Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37633" doReceive() ... 2015-08-03T20:59:23.846Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37637" doReceive() ... 2015-08-03T20:59:23.849Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37639" doReceive() ... 2015-08-03T20:59:23.849Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37640" doReceive() ... 2015-08-03T20:59:23.849Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37638" doReceive() ... 2015-08-03T20:59:23.849Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37641" doReceive() ... 2015-08-03T20:59:23.849Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37642" doReceive() ... 2015-08-03T20:59:23.852Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37643" doReceive() ... 2015-08-03T20:59:23.852Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37644" doReceive() ... 2015-08-03T20:59:23.852Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37646" doReceive() ... 2015-08-03T20:59:23.852Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37645" doReceive() ... 2015-08-03T20:59:23.855Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37647" doReceive() ... 2015-08-03T20:59:23.857Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37648" doReceive() ... 2015-08-03T20:59:23.86Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37649" doReceive() ... 2015-08-03T20:59:23.86Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37650" doReceive() ... 2015-08-03T20:59:23.863Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37651" doReceive() ... 2015-08-03T20:59:23.868Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37652" doReceive() ... 2015-08-03T20:59:23.871Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37653" doReceive() ... 2015-08-03T20:59:23.871Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37654" doReceive() ... 2015-08-03T20:59:23.874Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37655" doReceive() ... 2015-08-03T20:59:23.876Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37656" doReceive() ... 2015-08-03T20:59:23.876Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37657" doReceive() ... 2015-08-03T20:59:23.885Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37658" doReceive() ... 2015-08-03T20:59:23.888Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37659" doReceive() ... 2015-08-03T20:59:23.888Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37660" doReceive() ... 2015-08-03T20:59:23.888Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37661" doReceive() ... 2015-08-03T20:59:23.888Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37662" doReceive() ... 2015-08-03T20:59:23.891Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37663" doReceive() ... 2015-08-03T20:59:23.893Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37664" doReceive() ... 2015-08-03T20:59:23.893Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37665" doReceive() ... 2015-08-03T20:59:23.893Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37667" doReceive() ... 2015-08-03T20:59:23.893Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37666" doReceive() ... 2015-08-03T20:59:23.896Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37668" doReceive() ... 2015-08-03T20:59:23.896Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37669" doReceive() ... 2015-08-03T20:59:23.896Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37670" doReceive() ... 2015-08-03T20:59:23.899Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37671" doReceive() ... 2015-08-03T20:59:23.899Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37672" doReceive() ... 2015-08-03T20:59:23.899Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37674" doReceive() ... 2015-08-03T20:59:23.899Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37673" doReceive() ... 2015-08-03T20:59:23.902Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37675" doReceive() ... 2015-08-03T20:59:23.904Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37677" doReceive() ... 2015-08-03T20:59:23.904Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37678" doReceive() ... 2015-08-03T20:59:23.904Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37676" doReceive() ... 2015-08-03T20:59:23.904Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37680" doReceive() ... 2015-08-03T20:59:23.904Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37682" doReceive() ... 2015-08-03T20:59:23.905Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37683" doReceive() ... 2015-08-03T20:59:23.905Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37679" doReceive() ... 2015-08-03T20:59:23.905Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37681" doReceive() ... 2015-08-03T20:59:23.907Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37685" doReceive() ... 2015-08-03T20:59:23.907Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37684" doReceive() ... 2015-08-03T20:59:23.907Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37688" doReceive() ... 2015-08-03T20:59:23.907Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37690" doReceive() ... 2015-08-03T20:59:23.907Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37687" doReceive() ... 2015-08-03T20:59:23.907Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37691" doReceive() ... 2015-08-03T20:59:23.907Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37689" doReceive() ... 2015-08-03T20:59:23.91Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37693" doReceive() ... 2015-08-03T20:59:23.91Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37692" doReceive() ... 2015-08-03T20:59:23.91Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37700" doReceive() ... 2015-08-03T20:59:23.91Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37697" doReceive() ... 2015-08-03T20:59:23.91Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37696" doReceive() ... 2015-08-03T20:59:23.91Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37699" doReceive() ... 2015-08-03T20:59:23.91Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37695" doReceive() ... 2015-08-03T20:59:23.91Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37698" doReceive() ... 2015-08-03T20:59:23.913Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37701" doReceive() ... 2015-08-03T20:59:23.915Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37702" doReceive() ... 2015-08-03T20:59:23.915Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37704" doReceive() ... 2015-08-03T20:59:23.916Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37703" doReceive() ... 2015-08-03T20:59:23.916Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37706" doReceive() ... 2015-08-03T20:59:23.916Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37705" doReceive() ... 2015-08-03T20:59:23.918Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37707" doReceive() ... 2015-08-03T20:59:23.921Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37710" doReceive() ... 2015-08-03T20:59:23.921Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37709" doReceive() ... 2015-08-03T20:59:23.921Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37711" doReceive() ... 2015-08-03T20:59:23.921Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37712" doReceive() ... 2015-08-03T20:59:23.921Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37708" doReceive() ... 2015-08-03T20:59:23.923Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37713" doReceive() ... 2015-08-03T20:59:23.926Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37714" doReceive() ... 2015-08-03T20:59:23.926Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37715" doReceive() ... 2015-08-03T20:59:23.929Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37716" doReceive() ... 2015-08-03T20:59:23.929Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37718" doReceive() ... 2015-08-03T20:59:23.929Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37717" doReceive() ... 2015-08-03T20:59:23.932Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37719" doReceive() ... 2015-08-03T20:59:23.932Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37720" doReceive() ... 2015-08-03T20:59:23.934Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37721" doReceive() ... 2015-08-03T20:59:23.937Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37722" doReceive() ... 2015-08-03T20:59:23.94Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37723" doReceive() ... 2015-08-03T20:59:23.94Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37724" doReceive() ... 2015-08-03T20:59:23.94Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37725" doReceive() ... 2015-08-03T20:59:23.944Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37726" doReceive() ... 2015-08-03T20:59:23.95Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37727" doReceive() ... 2015-08-03T20:59:23.953Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37728" doReceive() ... 2015-08-03T20:59:23.953Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37729" doReceive() ... 2015-08-03T20:59:23.956Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37730" doReceive() ... 2015-08-03T20:59:23.958Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37731" doReceive() ... 2015-08-03T20:59:23.961Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37733" doReceive() ... 2015-08-03T20:59:23.961Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37732" doReceive() ... 2015-08-03T20:59:23.965Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37734" doReceive() ... 2015-08-03T20:59:23.967Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37735" doReceive() ... 2015-08-03T20:59:23.967Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37738" doReceive() ... 2015-08-03T20:59:23.967Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37736" doReceive() ... 2015-08-03T20:59:23.967Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37737" doReceive() ... 2015-08-03T20:59:23.967Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37740" doReceive() ... 2015-08-03T20:59:23.967Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37739" doReceive() ... 2015-08-03T20:59:23.967Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37741" doReceive() ... 2015-08-03T20:59:23.97Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37742" doReceive() ... 2015-08-03T20:59:23.973Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37743" doReceive() ... 2015-08-03T20:59:23.977Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37744" doReceive() ... 2015-08-03T20:59:23.98Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37746" doReceive() ... 2015-08-03T20:59:23.98Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37745" doReceive() ... 2015-08-03T20:59:23.98Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37747" doReceive() ... 2015-08-03T20:59:23.98Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37748" doReceive() ... 2015-08-03T20:59:23.982Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37749" doReceive() ... 2015-08-03T20:59:23.985Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37750" doReceive() ... 2015-08-03T20:59:23.985Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37751" doReceive() ... 2015-08-03T20:59:23.985Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37752" doReceive() ... 2015-08-03T20:59:23.985Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37753" doReceive() ... 2015-08-03T20:59:23.988Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37754" doReceive() ... 2015-08-03T20:59:23.988Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37756" doReceive() ... 2015-08-03T20:59:23.988Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37755" doReceive() ... 2015-08-03T20:59:23.99Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37757" doReceive() ... 2015-08-03T20:59:23.997Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37758" doReceive() ... 2015-08-03T20:59:24Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37759" doReceive() ... 2015-08-03T20:59:24Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37760" doReceive() ... 2015-08-03T20:59:24Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37762" doReceive() ... 2015-08-03T20:59:24Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37763" doReceive() ... 2015-08-03T20:59:24Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37761" doReceive() ... 2015-08-03T20:59:24Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37764" doReceive() ... 2015-08-03T20:59:24Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37765" doReceive() ... 2015-08-03T20:59:24.003Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37767" doReceive() ... 2015-08-03T20:59:24.003Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37766" doReceive() ... 2015-08-03T20:59:24.006Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37768" doReceive() ... 2015-08-03T20:59:24.01Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37769" doReceive() ... 2015-08-03T20:59:24.013Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37770" doReceive() ... 2015-08-03T20:59:24.013Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37772" doReceive() ... 2015-08-03T20:59:24.013Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37773" doReceive() ... 2015-08-03T20:59:24.013Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37771" doReceive() ... 2015-08-03T20:59:24.013Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37774" doReceive() ... 2015-08-03T20:59:24.013Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37775" doReceive() ... 2015-08-03T20:59:24.013Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37777" doReceive() ... 2015-08-03T20:59:24.013Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37776" doReceive() ... 2015-08-03T20:59:24.016Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37778" doReceive() ... 2015-08-03T20:59:24.016Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37779" doReceive() ... 2015-08-03T20:59:24.018Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37780" doReceive() ... 2015-08-03T20:59:24.025Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37781" doReceive() ... 2015-08-03T20:59:24.028Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37782" doReceive() ... 2015-08-03T20:59:24.031Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37783" doReceive() ... 2015-08-03T20:59:24.031Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37784" doReceive() ... 2015-08-03T20:59:24.031Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37785" doReceive() ... 2015-08-03T20:59:24.034Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37786" doReceive() ... 2015-08-03T20:59:24.065Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37787" doReceive() ... 2015-08-03T20:59:24.068Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37792" doReceive() ... 2015-08-03T20:59:24.068Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37790" doReceive() ... 2015-08-03T20:59:24.068Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37791" doReceive() ... 2015-08-03T20:59:24.068Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37798" doReceive() ... 2015-08-03T20:59:24.068Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37806" doReceive() ... 2015-08-03T20:59:24.068Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37801" doReceive() ... 2015-08-03T20:59:24.069Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37789" doReceive() ... 2015-08-03T20:59:24.069Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37804" doReceive() ... 2015-08-03T20:59:24.069Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37805" doReceive() ... 2015-08-03T20:59:24.069Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37799" doReceive() ... 2015-08-03T20:59:24.069Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37796" doReceive() ... 2015-08-03T20:59:24.069Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37797" doReceive() ... 2015-08-03T20:59:24.069Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37803" doReceive() ... 2015-08-03T20:59:24.069Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37788" doReceive() ... 2015-08-03T20:59:24.069Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37807" doReceive() ... 2015-08-03T20:59:24.069Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37794" doReceive() ... 2015-08-03T20:59:24.069Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37795" doReceive() ... 2015-08-03T20:59:24.07Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37793" doReceive() ... 2015-08-03T20:59:24.07Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37800" doReceive() ... 2015-08-03T20:59:24.07Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37802" doReceive() ... 2015-08-03T20:59:24.071Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37810" doReceive() ... 2015-08-03T20:59:24.071Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37808" doReceive() ... 2015-08-03T20:59:24.072Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37809" doReceive() ... 2015-08-03T20:59:24.072Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37811" doReceive() ... 2015-08-03T20:59:24.072Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37812" doReceive() ... 2015-08-03T20:59:24.077Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37813" doReceive() ... 2015-08-03T20:59:24.079Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 210000 QueuedCount 12821 2015-08-03T20:59:24.08Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37814" doReceive() ... 2015-08-03T20:59:24.083Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37815" doReceive() ... 2015-08-03T20:59:24.086Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37816" doReceive() ... 2015-08-03T20:59:24.088Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37818" doReceive() ... 2015-08-03T20:59:24.088Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37817" doReceive() ... 2015-08-03T20:59:24.09Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37819" doReceive() ... 2015-08-03T20:59:24.095Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37820" doReceive() ... 2015-08-03T20:59:24.097Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37821" doReceive() ... 2015-08-03T20:59:24.1Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37822" doReceive() ... 2015-08-03T20:59:24.1Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37826" doReceive() ... 2015-08-03T20:59:24.1Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37823" doReceive() ... 2015-08-03T20:59:24.1Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37824" doReceive() ... 2015-08-03T20:59:24.1Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37825" doReceive() ... 2015-08-03T20:59:24.12Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37827" doReceive() ... 2015-08-03T20:59:24.123Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37828" doReceive() ... 2015-08-03T20:59:24.123Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37834" doReceive() ... 2015-08-03T20:59:24.123Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37833" doReceive() ... 2015-08-03T20:59:24.123Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37830" doReceive() ... 2015-08-03T20:59:24.123Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37835" doReceive() ... 2015-08-03T20:59:24.123Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37831" doReceive() ... 2015-08-03T20:59:24.124Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37829" doReceive() ... 2015-08-03T20:59:24.124Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37832" doReceive() ... 2015-08-03T20:59:24.142Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37836" doReceive() ... 2015-08-03T20:59:24.145Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37837" doReceive() ... 2015-08-03T20:59:24.147Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37838" doReceive() ... 2015-08-03T20:59:24.147Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37840" doReceive() ... 2015-08-03T20:59:24.147Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37839" doReceive() ... 2015-08-03T20:59:24.15Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37842" doReceive() ... 2015-08-03T20:59:24.15Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37841" doReceive() ... 2015-08-03T20:59:24.154Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37843" doReceive() ... 2015-08-03T20:59:24.167Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37844" doReceive() ... 2015-08-03T20:59:24.17Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37845" doReceive() ... 2015-08-03T20:59:24.171Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37846" doReceive() ... 2015-08-03T20:59:24.174Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37847" doReceive() ... 2015-08-03T20:59:24.176Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37848" doReceive() ... 2015-08-03T20:59:24.183Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37849" doReceive() ... 2015-08-03T20:59:24.193Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37850" doReceive() ... 2015-08-03T20:59:24.196Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37851" doReceive() ... 2015-08-03T20:59:24.196Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37853" doReceive() ... 2015-08-03T20:59:24.197Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37852" doReceive() ... 2015-08-03T20:59:24.204Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37854" doReceive() ... 2015-08-03T20:59:24.207Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37855" doReceive() ... 2015-08-03T20:59:24.207Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37856" doReceive() ... 2015-08-03T20:59:24.207Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37858" doReceive() ... 2015-08-03T20:59:24.208Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37857" doReceive() ... 2015-08-03T20:59:24.21Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37859" doReceive() ... 2015-08-03T20:59:24.213Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37861" doReceive() ... 2015-08-03T20:59:24.213Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37860" doReceive() ... 2015-08-03T20:59:24.216Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37864" doReceive() ... 2015-08-03T20:59:24.216Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37862" doReceive() ... 2015-08-03T20:59:24.216Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37865" doReceive() ... 2015-08-03T20:59:24.216Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37863" doReceive() ... 2015-08-03T20:59:24.218Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37866" doReceive() ... 2015-08-03T20:59:24.218Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37868" doReceive() ... 2015-08-03T20:59:24.219Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37870" doReceive() ... 2015-08-03T20:59:24.219Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37867" doReceive() ... 2015-08-03T20:59:24.376Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 220000 QueuedCount 2821 2015-08-03T20:59:25.28Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 2.154µs CommitTime 145.042031ms TotalFlushTime 14.390572665s TotalCommitTime 2.327266068s 2015-08-03T20:59:25.28Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 23012716, 12006358, 371 committed:true) SnapCreateDur 158.582861ms SnapOpenDur 37.157µs 2015-08-03T20:59:25.427Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 230000 QueuedCount 14333 2015-08-03T20:59:26.123Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 240000 QueuedCount 4333 2015-08-03T20:59:26.332Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 23055040, 12027520, 0 committed:false) SnapCreateDur 19.56µs SnapOpenDur 139.918474ms 2015-08-03T20:59:27.581Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 8.013µs CommitTime 1.032248849s TotalFlushTime 15.415113781s TotalCommitTime 3.359514917s 2015-08-03T20:59:27.581Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 23063688, 12031844, 372 committed:true) SnapCreateDur 1.060263881s SnapOpenDur 25.017µs 2015-08-03T20:59:27.634Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 250000 QueuedCount 11326 2015-08-03T20:59:27.854Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 260000 QueuedCount 15505 2015-08-03T20:59:28.728Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 270000 QueuedCount 5505 2015-08-03T20:59:28.952Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 23116390, 12058195, 0 committed:false) SnapCreateDur 11.075µs SnapOpenDur 111.603149ms 2015-08-03T20:59:30.071Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 280000 QueuedCount 15424 2015-08-03T20:59:30.291Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 290000 QueuedCount 5424 2015-08-03T20:59:31.107Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 23155584, 12077792, 0 committed:false) SnapCreateDur 12µs SnapOpenDur 83.929136ms 2015-08-03T20:59:31.219Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 300000 QueuedCount 382 2015-08-03T20:59:31.755Z-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 16558345882944865108 FlushTime 4.377µs CommitTime 510.673114ms TotalFlushTime 18.748531877s TotalCommitTime 3.870188031s 2015-08-03T20:59:31.755Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 23165338, 12082669, 373 committed:true) SnapCreateDur 521.461786ms SnapOpenDur 24.631µs 2015-08-03T20:59:31.903Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 23177118, 12088559, 0 committed:false) SnapCreateDur 9.606µs SnapOpenDur 18.773896ms 2015-08-03T20:59:32.058Z-07:00 [Info] logWriterStat:: 16558345882944865108 FlushedCount 310000 QueuedCount 3528 2015-08-03T20:59:32.302Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 23191158, 12095579, 0 committed:false) SnapCreateDur 31.187µs SnapOpenDur 108.338858ms 2015-08-03T20:59:33.638Z-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16558345882944865108 PartitionId: 0 SliceId: 0 Crc64: 10335495992493484539 (SnapshotInfo: seqnos: 23200464, 12100232, 0 committed:false) SnapCreateDur 37.074µs SnapOpenDur 118.296998ms 2015-08-03T20:59:33.643Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37175" skip protobuf.EndStreamRequest 2015-08-03T20:59:33.651Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37636" skip protobuf.EndStreamRequest 2015-08-03T20:59:33.652Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37643" skip protobuf.EndStreamRequest 2015-08-03T20:59:33.653Z-07:00 [Info] [Queryport ":9101"] connection "172.23.100.55:37260" skip protobuf.EndStreamRequest 2015-08-03T20:59:36.621Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:513273557, Disk:2640687104, ExtraSnapshotData:0, Fragmentation:80.56287864539063% 2015-08-03T20:59:36.621Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T20:59:36.644Z-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 22811310. Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T21:00:00.211Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":18555850903,"bucket-1:by_email:avg_scan_wait_latency":18550665237,"bucket-1:by_email:avg_ts_interval":939270548,"bucket-1:by_email:avg_ts_items_count":6454,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":513050216,"bucket-1:by_email:delete_bytes":13464719,"bucket-1:by_email:disk_size":2711916544,"bucket-1:by_email:flush_queue_size":0,"bucket-1:by_email:frag_percent":81,"bucket-1:by_email:get_bytes":13745337,"bucket-1:by_email:insert_bytes":32878965,"bucket-1:by_email:items_count":1000001,"bucket-1:by_email:num_commits":6,"bucket-1:by_email:num_compactions":1,"bucket-1:by_email:num_completed_requests":701,"bucket-1:by_email:num_docs_indexed":12771000,"bucket-1:by_email:num_docs_pending":0,"bucket-1:by_email:num_docs_queued":0,"bucket-1:by_email:num_flush_queued":318245,"bucket-1:by_email:num_last_snapshot_reply":701,"bucket-1:by_email:num_requests":701,"bucket-1:by_email:num_rows_returned":6,"bucket-1:by_email:num_snapshot_waiters":0,"bucket-1:by_email:num_snapshots":21,"bucket-1:by_email:scan_bytes_read":240,"bucket-1:by_email:scan_wait_duration":13004016331822,"bucket-1:by_email:since_last_snapshot":1339037908,"bucket-1:by_email:timings/storage_clone_handle":"701 131401852 405768668098654","bucket-1:by_email:timings/storage_commit":"6 3870188031 3308992430632658009","bucket-1:by_email:timings/storage_del":"313133 4986963982 2553609449133095514","bucket-1:by_email:timings/storage_get":"318245 3197443238 3554056360840632","bucket-1:by_email:timings/storage_handle_open":"7 229236 8269562712","bucket-1:by_email:timings/storage_iterator_next":"6 121019 2732041911","bucket-1:by_email:timings/storage_new_iterator":"701 174150716 252170699798138","bucket-1:by_email:timings/storage_set":"626266 11826478474 8074711926370599850","bucket-1:by_email:timings/storage_snapshot_close":"14 192765543 3039674777848235","bucket-1:by_email:timings/storage_snapshot_create":"15 1284801009 127715017824886775","bucket-1:by_email:total_scan_duration":13007651483237,"bucket-1:mutation_queue_size":0,"bucket-1:num_mutations_queued":318245,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":0,"memory_quota":104857600000,"memory_used":2782842880,"needs_restart":false,"num_connections":701,"timings/stats_response":"64 27427646 26410760964686"} 2015-08-03T21:00:04.429Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108 2015-08-03T21:00:34.43Z-07:00 [Info] CompactionDaemon: Checking fragmentation, IndexInstId: 16558345882944865108 Data:347265882, Disk:641916928, ExtraSnapshotData:0, Fragmentation:45.901741042728816% 2015-08-03T21:00:34.43Z-07:00 [Info] CompactionDaemon: Compacting index instance:16558345882944865108 2015-08-03T21:00:34.452Z-07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 22811310. Slice Id 0, IndexInstId 16558345882944865108, IndexDefnId 16558345882944865108 2015-08-03T21:01:00.211Z-07:00 [Info] PeriodicStats = {"bucket-1:by_email:avg_scan_latency":18555850903,"bucket-1:by_email:avg_scan_wait_latency":18550665237,"bucket-1:by_email:avg_ts_interval":939270548,"bucket-1:by_email:avg_ts_items_count":6454,"bucket-1:by_email:build_progress":100,"bucket-1:by_email:data_size":347265882,"bucket-1:by_email:delete_bytes":13464719,"bucket-1:by_email:disk_size":641925120,"bucket-1:by_email:flush_queue_size":0,"bucket-1:by_email:frag_percent":45,"bucket-1:by_email:get_bytes":13745337,"bucket-1:by_email:insert_bytes":32878965,"bucket-1:by_email:items_count":1000001,"bucket-1:by_email:num_commits":6,"bucket-1:by_email:num_compactions":2,"bucket-1:by_email:num_completed_requests":701,"bucket-1:by_email:num_docs_indexed":12771000,"bucket-1:by_email:num_docs_pending":0,"bucket-1:by_email:num_docs_queued":0,"bucket-1:by_email:num_flush_queued":318245,"bucket-1:by_email:num_last_snapshot_reply":701,"bucket-1:by_email:num_requests":701,"bucket-1:by_email:num_rows_returned":6,"bucket-1:by_email:num_snapshot_waiters":0,"bucket-1:by_email:num_snapshots":21,"bucket-1:by_email:scan_bytes_read":240,"bucket-1:by_email:scan_wait_duration":13004016331822,"bucket-1:by_email:since_last_snapshot":1339037908,"bucket-1:by_email:timings/storage_clone_handle":"701 131401852 405768668098654","bucket-1:by_email:timings/storage_commit":"6 3870188031 3308992430632658009","bucket-1:by_email:timings/storage_del":"313133 4986963982 2553609449133095514","bucket-1:by_email:timings/storage_get":"318245 3197443238 3554056360840632","bucket-1:by_email:timings/storage_handle_open":"7 229236 8269562712","bucket-1:by_email:timings/storage_iterator_next":"6 121019 2732041911","bucket-1:by_email:timings/storage_new_iterator":"701 174150716 252170699798138","bucket-1:by_email:timings/storage_set":"626266 11826478474 8074711926370599850","bucket-1:by_email:timings/storage_snapshot_close":"14 192765543 3039674777848235","bucket-1:by_email:timings/storage_snapshot_create":"15 1284801009 127715017824886775","bucket-1:by_email:total_scan_duration":13007651483237,"bucket-1:mutation_queue_size":0,"bucket-1:num_mutations_queued":318245,"bucket-1:num_nonalign_ts":0,"bucket-1:ts_queue_size":0,"memory_quota":104857600000,"memory_used":3534053376,"needs_restart":false,"num_connections":701,"timings/stats_response":"136 52165287 35280809325955"} 2015-08-03T21:01:01.789Z-07:00 [Info] CompactionDaemon: Finished compacting index instance:16558345882944865108