============================================================================== couchbase logs (indexer.log) cbbrowse_logs indexer.log ============================================================================== 2016-04-07T01:25:49.193-07:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -vbuckets=1024 -cluster=127.0.0.1:8091 -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 -storageDir=/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -nodeUUID=103aa9f3d2294e264367f01b0999fb0e -storageMode=] 2016-04-07T01:25:49.208-07:00 [Info] Indexer::NewIndexer Status Bootstrap 2016-04-07T01:25:49.211-07:00 [Info] Setting buffer block size to 16384 bytes 2016-04-07T01:25:49.211-07:00 [Info] Setting maxcpus = 48 2016-04-07T01:25:49.211-07:00 [Info] Setting log level to Info 2016-04-07T01:25:49.211-07:00 [Info] 127.0.0.1:8091 starting RESTful services 2016-04-07T01:25:49.213-07:00 [Info] New settings received: {"indexer.settings.compaction.days_of_week":"Sunday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.log_level":"info","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"","indexer.settings.recovery.max_rollbacks":5,"indexer.settings.memory_quota":31457280000,"indexer.settings.compaction.abort_exceed_interval":false} 2016-04-07T01:25:49.213-07:00 [Info] Setting maxcpus = 48 2016-04-07T01:25:49.213-07:00 [Info] Setting log level to Info 2016-04-07T01:25:49.221-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:25:49.221-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:25:49.222-07:00 [Error] WatcherServer.runOnce() error : dial tcp 172.23.99.247:9100: getsockopt: connection refused 2016-04-07T01:25:50.221-07:00 [Info] WatchMetadata(): unable to reach indexer at 172.23.99.247:9100. Retry in background. 2016-04-07T01:25:50.221-07:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2016-04-07T01:25:50.221-07:00 [Info] servicesChangeNotifier: Creating new notifier instance for http://%40index-cbauth:2d6c39e0d8f5be127d754756f7839391@127.0.0.1:8091, default 2016-04-07T01:25:50.222-07:00 [Error] WatcherServer.runOnce() error : dial tcp 172.23.99.247:9100: getsockopt: connection refused 2016-04-07T01:25:50.224-07:00 [Info] [Queryport ":9101"] started ... 2016-04-07T01:25:50.236-07:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2016-04-07T01:25:50.24-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:25:50.242-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:25:50.242-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:25:50.242-07:00 [Error] updateIndexerList(): Cannot locate cluster node. 2016-04-07T01:25:50.262-07:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2016-04-07T01:25:50.263-07:00 [Info] ClustMgr:handleSetLocalValue Key IndexerId Value 52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:25:50.263-07:00 [Info] Indexer Id 52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:25:50.268-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.241:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:25:50.27-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.241:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:25:50.27-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.99.241:9999 2016-04-07T01:25:50.27-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:25:50.27-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.245:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:25:50.271-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.245:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:25:50.271-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.99.245:9999 2016-04-07T01:25:50.271-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:25:50.271-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.246:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:25:50.272-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.246:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:25:50.272-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.99.246:9999 2016-04-07T01:25:50.272-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:25:50.272-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.251:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:25:50.273-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.251:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:25:50.273-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.99.251:9999 2016-04-07T01:25:50.273-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:25:50.278-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.241:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:25:50.278-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.241:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:25:50.278-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.99.241:9999 2016-04-07T01:25:50.278-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:25:50.278-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.245:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:25:50.279-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.245:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:25:50.279-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.99.245:9999 2016-04-07T01:25:50.279-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:25:50.279-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.246:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:25:50.279-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.246:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:25:50.279-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.99.246:9999 2016-04-07T01:25:50.279-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:25:50.279-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.251:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:25:50.28-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.251:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:25:50.28-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.99.251:9999 2016-04-07T01:25:50.28-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:25:50.28-07:00 [Info] ClustMgr:handleGetGlobalTopology &{map[]} 2016-04-07T01:25:50.28-07:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap 2016-04-07T01:25:50.28-07:00 [Info] MutationMgr::handleUpdateIndexInstMap Message: MsgUpdateInstMap 2016-04-07T01:25:50.28-07:00 [Info] MutationMgr::handleUpdateIndexPartnMap Message: MsgUpdatePartnMap 2016-04-07T01:25:50.28-07:00 [Info] ClustMgr:handleIndexerReady &{52 [] {false false false false} 0} 2016-04-07T01:25:50.28-07:00 [Info] ClustMgr:handleSetLocalValue Key IndexerNodeUUID Value 103aa9f3d2294e264367f01b0999fb0e 2016-04-07T01:25:50.288-07:00 [Info] Indexer::NewIndexer Status Active 2016-04-07T01:25:50.288-07:00 [Info] Indexer::monitorMemUsage started... 2016-04-07T01:25:50.288-07:00 [Info] memstats {"Alloc":0, "TotalAlloc":0, "Sys":0, "Lookups":0, "Mallocs":0,"Frees":0, "HeapAlloc":0, "HeapSys":0, "HeapIdle":0, "HeapInuse":0,"HeapReleased":0, "HeapObjects":0,"GCSys":0, "LastGC":0,"PauseTotalNs":0, "PauseNs":[], "NumGC":0} 2016-04-07T01:25:50.288-07:00 [Info] Indexer: Setting GC percent to 100 2016-04-07T01:25:50.298-07:00 [Info] CompactionManager: Refreshing settings 2016-04-07T01:25:50.897-07:00 [Info] Indexer::ReadMemstats Time Taken 125.93µs 2016-04-07T01:25:51.163-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:25:52.234-07:00 [Info] WatchMetadata(): Successfully connected to indexer at 172.23.99.247:9100 after retry. 2016-04-07T01:25:52.234-07:00 [Info] Acknowledged that new indexer is registered. Indexer = 172.23.99.247:9100, id = 52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:25:56.163-07:00 [Warn] servicesChangeNotifier: Consumer for http://%40index-cbauth:2d6c39e0d8f5be127d754756f7839391@127.0.0.1:8091 took too long to read notification, making the consumer invalid 2016-04-07T01:25:56.164-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:25:56.164-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:25:56.164-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:25:56.178-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:25:56.178-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:25:56.191-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:25:56.191-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:25:56.205-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:25:56.205-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:25:57.239-07:00 [Info] New settings received: {"indexer.settings.bufferPoolBlockSize":16384,"indexer.settings.compaction.abort_exceed_interval":false,"indexer.settings.compaction.check_period":30,"indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.days_of_week":"sunday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.compaction.min_frag":30,"indexer.settings.compaction.min_size":524288000,"indexer.settings.cpuProfFname":"","indexer.settings.cpuProfile":false,"indexer.settings.fast_flush_mode":true,"indexer.settings.gc_percent":100,"indexer.settings.inmemory_snapshot.fdb.interval":200,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.inmemory_snapshot.moi.interval":20,"indexer.settings.largeSnapshotThreshold":200,"indexer.settings.log_level":"info","indexer.settings.maxVbQueueLength":0,"indexer.settings.max_array_seckey_size":10240,"indexer.settings.max_cpu_percent":2400,"indexer.settings.max_writer_lock_prob":20,"indexer.settings.memProfFname":"","indexer.settings.memProfile":false,"indexer.settings.memory_quota":31457280000,"indexer.settings.moi.persistence_threads":96,"indexer.settings.moi.recovery_threads":48,"indexer.settings.persisted_snapshot.fdb.interval":5000,"indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.persisted_snapshot.moi.interval":600000,"indexer.settings.persisted_snapshot_init_build.fdb.interval":5000,"indexer.settings.persisted_snapshot_init_build.interval":5000,"indexer.settings.persisted_snapshot_init_build.moi.interval":600000,"indexer.settings.recovery.max_rollbacks":5,"indexer.settings.scan_getseqnos_retries":30,"indexer.settings.scan_timeout":0,"indexer.settings.send_buffer_size":1024,"indexer.settings.sliceBufSize":50000,"indexer.settings.smallSnapshotThreshold":30,"indexer.settings.statsLogDumpInterval":60,"indexer.settings.storage_mode":"","indexer.settings.wal_size":4096,"projector.settings.log_level":"info","queryport.client.settings.poolOverflow":30,"queryport.client.settings.poolSize":1000} 2016-04-07T01:25:57.239-07:00 [Info] Setting maxcpus = 24 2016-04-07T01:25:57.239-07:00 [Info] Setting log level to Info 2016-04-07T01:25:57.239-07:00 [Info] Indexer: Setting GC percent to 100 2016-04-07T01:25:57.25-07:00 [Info] CompactionManager: Refreshing settings 2016-04-07T01:25:57.276-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:25:57.289-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:25:57.289-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] [goport] 2016/04/07 01:26:07 got new line on a stdin; terminating. 2016-04-07T01:26:14.726-07:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -vbuckets=1024 -cluster=127.0.0.1:8091 -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 -storageDir=/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -nodeUUID=103aa9f3d2294e264367f01b0999fb0e -storageMode=] 2016-04-07T01:26:14.742-07:00 [Info] Indexer::NewIndexer Status Bootstrap 2016-04-07T01:26:14.75-07:00 [Info] Setting buffer block size to 16384 bytes 2016-04-07T01:26:14.75-07:00 [Info] Setting maxcpus = 24 2016-04-07T01:26:14.75-07:00 [Info] Setting log level to Info 2016-04-07T01:26:14.75-07:00 [Info] 127.0.0.1:8091 starting RESTful services 2016-04-07T01:26:14.752-07:00 [Info] New settings received: {"indexer.settings.bufferPoolBlockSize":16384,"indexer.settings.compaction.abort_exceed_interval":false,"indexer.settings.compaction.check_period":30,"indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.days_of_week":"sunday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.compaction.min_frag":30,"indexer.settings.compaction.min_size":524288000,"indexer.settings.cpuProfFname":"","indexer.settings.cpuProfile":false,"indexer.settings.fast_flush_mode":true,"indexer.settings.gc_percent":100,"indexer.settings.inmemory_snapshot.fdb.interval":200,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.inmemory_snapshot.moi.interval":20,"indexer.settings.largeSnapshotThreshold":200,"indexer.settings.log_level":"info","indexer.settings.maxVbQueueLength":0,"indexer.settings.max_array_seckey_size":10240,"indexer.settings.max_cpu_percent":2400,"indexer.settings.max_writer_lock_prob":20,"indexer.settings.memProfFname":"","indexer.settings.memProfile":false,"indexer.settings.memory_quota":31457280000,"indexer.settings.moi.persistence_threads":96,"indexer.settings.moi.recovery_threads":48,"indexer.settings.persisted_snapshot.fdb.interval":5000,"indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.persisted_snapshot.moi.interval":600000,"indexer.settings.persisted_snapshot_init_build.fdb.interval":5000,"indexer.settings.persisted_snapshot_init_build.interval":5000,"indexer.settings.persisted_snapshot_init_build.moi.interval":600000,"indexer.settings.recovery.max_rollbacks":5,"indexer.settings.scan_getseqnos_retries":30,"indexer.settings.scan_timeout":0,"indexer.settings.send_buffer_size":1024,"indexer.settings.sliceBufSize":50000,"indexer.settings.smallSnapshotThreshold":30,"indexer.settings.statsLogDumpInterval":60,"indexer.settings.storage_mode":"","indexer.settings.wal_size":4096,"projector.settings.log_level":"info","queryport.client.settings.poolOverflow":30,"queryport.client.settings.poolSize":1000} 2016-04-07T01:26:14.753-07:00 [Info] Setting maxcpus = 24 2016-04-07T01:26:14.753-07:00 [Info] Setting log level to Info 2016-04-07T01:26:14.798-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:26:14.798-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:26:14.798-07:00 [Error] WatcherServer.runOnce() error : dial tcp 172.23.99.247:9100: getsockopt: connection refused 2016-04-07T01:26:15.798-07:00 [Info] WatchMetadata(): unable to reach indexer at 172.23.99.247:9100. Retry in background. 2016-04-07T01:26:15.798-07:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2016-04-07T01:26:15.799-07:00 [Error] WatcherServer.runOnce() error : dial tcp 172.23.99.247:9100: getsockopt: connection refused 2016-04-07T01:26:15.798-07:00 [Info] servicesChangeNotifier: Creating new notifier instance for http://%40index-cbauth:2d6c39e0d8f5be127d754756f7839391@127.0.0.1:8091, default 2016-04-07T01:26:15.8-07:00 [Info] [Queryport ":9101"] started ... 2016-04-07T01:26:15.81-07:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2016-04-07T01:26:15.815-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:26:15.826-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:26:15.826-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:26:15.826-07:00 [Error] updateIndexerList(): Cannot locate cluster node. 2016-04-07T01:26:15.83-07:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2016-04-07T01:26:15.838-07:00 [Info] Indexer Id 52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:26:15.849-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.241:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:26:15.85-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.241:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err Post http://172.23.99.241:9999/adminport/shutdownTopicRequest: dial tcp 172.23.99.241:9999: getsockopt: connection refused 2016-04-07T01:26:15.85-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received Post http://172.23.99.241:9999/adminport/shutdownTopicRequest: dial tcp 172.23.99.241:9999: getsockopt: connection refused from 172.23.99.241:9999 2016-04-07T01:26:15.85-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.245:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:26:15.85-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.245:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err Post http://172.23.99.245:9999/adminport/shutdownTopicRequest: dial tcp 172.23.99.245:9999: getsockopt: connection refused 2016-04-07T01:26:15.85-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received Post http://172.23.99.245:9999/adminport/shutdownTopicRequest: dial tcp 172.23.99.245:9999: getsockopt: connection refused from 172.23.99.245:9999 2016-04-07T01:26:15.85-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.246:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:26:15.851-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.246:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err Post http://172.23.99.246:9999/adminport/shutdownTopicRequest: dial tcp 172.23.99.246:9999: getsockopt: connection refused 2016-04-07T01:26:15.851-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received Post http://172.23.99.246:9999/adminport/shutdownTopicRequest: dial tcp 172.23.99.246:9999: getsockopt: connection refused from 172.23.99.246:9999 2016-04-07T01:26:15.851-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.251:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:26:15.851-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.251:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err Post http://172.23.99.251:9999/adminport/shutdownTopicRequest: dial tcp 172.23.99.251:9999: getsockopt: connection refused 2016-04-07T01:26:15.851-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received Post http://172.23.99.251:9999/adminport/shutdownTopicRequest: dial tcp 172.23.99.251:9999: getsockopt: connection refused from 172.23.99.251:9999 2016-04-07T01:26:16.851-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received Post http://172.23.99.251:9999/adminport/shutdownTopicRequest: dial tcp 172.23.99.251:9999: getsockopt: connection refused 2016-04-07T01:26:16.851-07:00 [Error] Indexer::closeAllStreams Stream MAINT_STREAM Error from Projector Post http://172.23.99.251:9999/adminport/shutdownTopicRequest: dial tcp 172.23.99.251:9999: getsockopt: connection refused. Retrying. 2016-04-07T01:26:17.801-07:00 [Info] WatchMetadata(): Successfully connected to indexer at 172.23.99.247:9100 after retry. 2016-04-07T01:26:17.801-07:00 [Info] Acknowledged that new indexer is registered. Indexer = 172.23.99.247:9100, id = 52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:26:19.57-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:26:19.583-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:26:19.583-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:26:21.866-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.241:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:26:21.867-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.241:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:26:21.867-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.99.241:9999 2016-04-07T01:26:21.867-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:26:21.867-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.245:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:26:21.868-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.245:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:26:21.868-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.99.245:9999 2016-04-07T01:26:21.868-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:26:21.869-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.246:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:26:21.87-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.246:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:26:21.87-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.99.246:9999 2016-04-07T01:26:21.87-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:26:21.87-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.251:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:26:21.871-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.251:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:26:21.871-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.99.251:9999 2016-04-07T01:26:21.871-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:26:21.883-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.241:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:26:21.884-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.241:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:26:21.884-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.99.241:9999 2016-04-07T01:26:21.884-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:26:21.884-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.245:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:26:21.884-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.245:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:26:21.884-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.99.245:9999 2016-04-07T01:26:21.884-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:26:21.884-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.246:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:26:21.885-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.246:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:26:21.885-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.99.246:9999 2016-04-07T01:26:21.885-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:26:21.885-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.251:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:26:21.885-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.251:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:26:21.885-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.99.251:9999 2016-04-07T01:26:21.885-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:26:21.885-07:00 [Info] ClustMgr:handleGetGlobalTopology &{map[]} 2016-04-07T01:26:21.885-07:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap 2016-04-07T01:26:21.885-07:00 [Info] MutationMgr::handleUpdateIndexInstMap Message: MsgUpdateInstMap 2016-04-07T01:26:21.886-07:00 [Info] MutationMgr::handleUpdateIndexPartnMap Message: MsgUpdatePartnMap 2016-04-07T01:26:21.886-07:00 [Info] ClustMgr:handleIndexerReady &{52 [] {false false false false} 0} 2016-04-07T01:26:21.886-07:00 [Info] ClustMgr:handleSetLocalValue Key IndexerNodeUUID Value 103aa9f3d2294e264367f01b0999fb0e 2016-04-07T01:26:21.886-07:00 [Info] Indexer::NewIndexer Status Active 2016-04-07T01:26:21.886-07:00 [Info] Indexer::monitorMemUsage started... 2016-04-07T01:26:21.886-07:00 [Info] Indexer: Setting GC percent to 100 2016-04-07T01:26:21.886-07:00 [Info] memstats {"Alloc":0, "TotalAlloc":0, "Sys":0, "Lookups":0, "Mallocs":0,"Frees":0, "HeapAlloc":0, "HeapSys":0, "HeapIdle":0, "HeapInuse":0,"HeapReleased":0, "HeapObjects":0,"GCSys":0, "LastGC":0,"PauseTotalNs":0, "PauseNs":[], "NumGC":0} 2016-04-07T01:26:21.897-07:00 [Info] CompactionManager: Refreshing settings 2016-04-07T01:26:24.736-07:00 [Info] Indexer::ReadMemstats Time Taken 108.588µs 2016-04-07T01:26:33.031-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:26:33.044-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:26:33.044-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:27:15.8-07:00 [Info] connected with 1 indexers 2016-04-07T01:27:15.8-07:00 [Info] client load stats {} 2016-04-07T01:27:15.83-07:00 [Info] PeriodicStats = {"indexer_state":"Active","memory_quota":31457280000,"memory_used":54185984,"memory_used_storage":188416,"needs_restart":false,"num_connections":0,"storage_mode":"not_set","timings/stats_response":"59 2817817 171487174429","uptime":"1m1.104143754s"} ==== StorageStats ==== 2016-04-07T01:27:21.897-07:00 [Info] memstats {"Alloc":50427848, "TotalAlloc":54719240, "Sys":64084216, "Lookups":42, "Mallocs":107389,"Frees":88307, "HeapAlloc":50427848, "HeapSys":55214080, "HeapIdle":3194880, "HeapInuse":52019200,"HeapReleased":0, "HeapObjects":19082,"GCSys":1978368, "LastGC":1460017581897194824,"PauseTotalNs":15569481, "PauseNs":[3244470, 334556, 1344986, 10645469], "NumGC":4} 2016-04-07T01:27:24.972-07:00 [Info] Indexer::ReadMemstats Time Taken 229.742µs 2016-04-07T01:28:15.8-07:00 [Info] connected with 1 indexers 2016-04-07T01:28:15.8-07:00 [Info] client load stats {} 2016-04-07T01:28:15.83-07:00 [Info] PeriodicStats = {"indexer_state":"Active","memory_quota":31457280000,"memory_used":61132800,"memory_used_storage":188416,"needs_restart":false,"num_connections":0,"storage_mode":"not_set","timings/stats_response":"131 6416411 383165465695","uptime":"2m1.104417097s"} ==== StorageStats ==== 2016-04-07T01:28:21.897-07:00 [Info] memstats {"Alloc":57643808, "TotalAlloc":61935200, "Sys":68413688, "Lookups":90, "Mallocs":256690,"Frees":146444, "HeapAlloc":57643808, "HeapSys":59310080, "HeapIdle":475136, "HeapInuse":58834944,"HeapReleased":0, "HeapObjects":110246,"GCSys":2109440, "LastGC":1460017581897194824,"PauseTotalNs":15569481, "PauseNs":[], "NumGC":4} [goport] 2016/04/07 01:28:21 got new line on a stdin; terminating. 2016-04-07T01:28:31.276-07:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -vbuckets=1024 -cluster=127.0.0.1:8091 -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 -storageDir=/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -nodeUUID=103aa9f3d2294e264367f01b0999fb0e -storageMode=] 2016-04-07T01:28:31.31-07:00 [Info] Indexer::NewIndexer Status Bootstrap 2016-04-07T01:28:31.326-07:00 [Info] Setting buffer block size to 16384 bytes 2016-04-07T01:28:31.326-07:00 [Info] Setting maxcpus = 24 2016-04-07T01:28:31.326-07:00 [Info] Setting log level to Info 2016-04-07T01:28:31.326-07:00 [Info] 127.0.0.1:8091 starting RESTful services 2016-04-07T01:28:31.329-07:00 [Info] New settings received: {"indexer.settings.bufferPoolBlockSize":16384,"indexer.settings.compaction.abort_exceed_interval":false,"indexer.settings.compaction.check_period":30,"indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.days_of_week":"sunday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.compaction.min_frag":30,"indexer.settings.compaction.min_size":524288000,"indexer.settings.cpuProfFname":"","indexer.settings.cpuProfile":false,"indexer.settings.fast_flush_mode":true,"indexer.settings.gc_percent":100,"indexer.settings.inmemory_snapshot.fdb.interval":200,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.inmemory_snapshot.moi.interval":20,"indexer.settings.largeSnapshotThreshold":200,"indexer.settings.log_level":"info","indexer.settings.maxVbQueueLength":0,"indexer.settings.max_array_seckey_size":10240,"indexer.settings.max_cpu_percent":2400,"indexer.settings.max_writer_lock_prob":20,"indexer.settings.memProfFname":"","indexer.settings.memProfile":false,"indexer.settings.memory_quota":31457280000,"indexer.settings.moi.persistence_threads":96,"indexer.settings.moi.recovery_threads":48,"indexer.settings.persisted_snapshot.fdb.interval":5000,"indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.persisted_snapshot.moi.interval":600000,"indexer.settings.persisted_snapshot_init_build.fdb.interval":5000,"indexer.settings.persisted_snapshot_init_build.interval":5000,"indexer.settings.persisted_snapshot_init_build.moi.interval":600000,"indexer.settings.recovery.max_rollbacks":5,"indexer.settings.scan_getseqnos_retries":30,"indexer.settings.scan_timeout":0,"indexer.settings.send_buffer_size":1024,"indexer.settings.sliceBufSize":50000,"indexer.settings.smallSnapshotThreshold":30,"indexer.settings.statsLogDumpInterval":60,"indexer.settings.storage_mode":"","indexer.settings.wal_size":4096,"projector.settings.log_level":"info","queryport.client.settings.poolOverflow":30,"queryport.client.settings.poolSize":1000} 2016-04-07T01:28:31.329-07:00 [Info] Setting maxcpus = 24 2016-04-07T01:28:31.329-07:00 [Info] Setting log level to Info 2016-04-07T01:28:31.37-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:28:31.37-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:28:31.37-07:00 [Error] WatcherServer.runOnce() error : dial tcp 172.23.99.247:9100: getsockopt: connection refused 2016-04-07T01:28:32.37-07:00 [Info] WatchMetadata(): unable to reach indexer at 172.23.99.247:9100. Retry in background. 2016-04-07T01:28:32.371-07:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2016-04-07T01:28:32.372-07:00 [Info] servicesChangeNotifier: Creating new notifier instance for http://%40index-cbauth:2d6c39e0d8f5be127d754756f7839391@127.0.0.1:8091, default 2016-04-07T01:28:32.372-07:00 [Error] WatcherServer.runOnce() error : dial tcp 172.23.99.247:9100: getsockopt: connection refused 2016-04-07T01:28:32.372-07:00 [Info] [Queryport ":9101"] started ... 2016-04-07T01:28:32.383-07:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2016-04-07T01:28:32.388-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:28:32.4-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:28:32.4-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:28:32.4-07:00 [Error] updateIndexerList(): Cannot locate cluster node. 2016-04-07T01:28:32.411-07:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2016-04-07T01:28:32.419-07:00 [Info] Indexer Id 52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:28:32.432-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.241:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:28:32.434-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.241:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:28:32.434-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.99.241:9999 2016-04-07T01:28:32.434-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:28:32.434-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.245:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:28:32.435-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.245:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:28:32.435-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.99.245:9999 2016-04-07T01:28:32.435-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:28:32.435-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.246:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:28:32.436-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.246:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:28:32.436-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.99.246:9999 2016-04-07T01:28:32.436-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:28:32.436-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.251:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:28:32.437-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.251:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:28:32.437-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.99.251:9999 2016-04-07T01:28:32.437-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:28:32.449-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.241:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:28:32.45-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.241:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:28:32.45-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.99.241:9999 2016-04-07T01:28:32.45-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:28:32.45-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.245:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:28:32.45-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.245:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:28:32.45-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.99.245:9999 2016-04-07T01:28:32.45-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:28:32.45-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.246:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:28:32.451-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.246:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:28:32.451-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.99.246:9999 2016-04-07T01:28:32.451-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:28:32.451-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.251:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:28:32.451-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.251:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:28:32.451-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.99.251:9999 2016-04-07T01:28:32.451-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:28:32.451-07:00 [Info] ClustMgr:handleGetGlobalTopology &{map[]} 2016-04-07T01:28:32.451-07:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap 2016-04-07T01:28:32.451-07:00 [Info] MutationMgr::handleUpdateIndexInstMap Message: MsgUpdateInstMap 2016-04-07T01:28:32.451-07:00 [Info] MutationMgr::handleUpdateIndexPartnMap Message: MsgUpdatePartnMap 2016-04-07T01:28:32.452-07:00 [Info] ClustMgr:handleIndexerReady &{52 [] {false false false false} 0} 2016-04-07T01:28:32.452-07:00 [Info] ClustMgr:handleSetLocalValue Key IndexerNodeUUID Value 103aa9f3d2294e264367f01b0999fb0e 2016-04-07T01:28:32.452-07:00 [Info] Indexer::NewIndexer Status Active 2016-04-07T01:28:32.452-07:00 [Info] Indexer::monitorMemUsage started... 2016-04-07T01:28:32.452-07:00 [Info] Indexer: Setting GC percent to 100 2016-04-07T01:28:32.452-07:00 [Info] memstats {"Alloc":0, "TotalAlloc":0, "Sys":0, "Lookups":0, "Mallocs":0,"Frees":0, "HeapAlloc":0, "HeapSys":0, "HeapIdle":0, "HeapInuse":0,"HeapReleased":0, "HeapObjects":0,"GCSys":0, "LastGC":0,"PauseTotalNs":0, "PauseNs":[], "NumGC":0} 2016-04-07T01:28:32.462-07:00 [Info] CompactionManager: Refreshing settings 2016-04-07T01:28:32.817-07:00 [Info] Indexer::ReadMemstats Time Taken 76.546µs 2016-04-07T01:28:34.382-07:00 [Info] WatchMetadata(): Successfully connected to indexer at 172.23.99.247:9100 after retry. 2016-04-07T01:28:34.382-07:00 [Info] Acknowledged that new indexer is registered. Indexer = 172.23.99.247:9100, id = 52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:29:08.199-07:00 [Info] New settings received: {"indexer.settings.bufferPoolBlockSize":16384,"indexer.settings.compaction.abort_exceed_interval":false,"indexer.settings.compaction.check_period":30,"indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.days_of_week":"sunday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.compaction.min_frag":30,"indexer.settings.compaction.min_size":524288000,"indexer.settings.cpuProfFname":"","indexer.settings.cpuProfile":false,"indexer.settings.fast_flush_mode":true,"indexer.settings.gc_percent":200,"indexer.settings.inmemory_snapshot.fdb.interval":200,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.inmemory_snapshot.moi.interval":20,"indexer.settings.largeSnapshotThreshold":200,"indexer.settings.log_level":"info","indexer.settings.maxVbQueueLength":0,"indexer.settings.max_array_seckey_size":10240,"indexer.settings.max_cpu_percent":2400,"indexer.settings.max_writer_lock_prob":20,"indexer.settings.memProfFname":"","indexer.settings.memProfile":false,"indexer.settings.memory_quota":31457280000,"indexer.settings.moi.persistence_threads":96,"indexer.settings.moi.recovery_threads":48,"indexer.settings.persisted_snapshot.fdb.interval":5000,"indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.persisted_snapshot.moi.interval":600000,"indexer.settings.persisted_snapshot_init_build.fdb.interval":5000,"indexer.settings.persisted_snapshot_init_build.interval":5000,"indexer.settings.persisted_snapshot_init_build.moi.interval":600000,"indexer.settings.recovery.max_rollbacks":5,"indexer.settings.scan_getseqnos_retries":30,"indexer.settings.scan_timeout":0,"indexer.settings.send_buffer_size":1024,"indexer.settings.sliceBufSize":50000,"indexer.settings.smallSnapshotThreshold":30,"indexer.settings.statsLogDumpInterval":60,"indexer.settings.storage_mode":"","indexer.settings.wal_size":4096,"projector.settings.log_level":"info","queryport.client.settings.poolOverflow":30,"queryport.client.settings.poolSize":1000} 2016-04-07T01:29:08.199-07:00 [Info] Setting maxcpus = 24 2016-04-07T01:29:08.199-07:00 [Info] Setting log level to Info 2016-04-07T01:29:08.199-07:00 [Info] Indexer: Setting GC percent to 200 2016-04-07T01:29:08.211-07:00 [Info] CompactionManager: Refreshing settings 2016-04-07T01:29:08.217-07:00 [Info] New settings received: {"indexer.settings.bufferPoolBlockSize":16384,"indexer.settings.compaction.abort_exceed_interval":false,"indexer.settings.compaction.check_period":30,"indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.days_of_week":"sunday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.compaction.min_frag":30,"indexer.settings.compaction.min_size":524288000,"indexer.settings.cpuProfFname":"","indexer.settings.cpuProfile":false,"indexer.settings.fast_flush_mode":true,"indexer.settings.gc_percent":200,"indexer.settings.inmemory_snapshot.fdb.interval":200,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.inmemory_snapshot.moi.interval":20,"indexer.settings.largeSnapshotThreshold":200,"indexer.settings.log_level":"info","indexer.settings.maxVbQueueLength":0,"indexer.settings.max_array_seckey_size":10240,"indexer.settings.max_cpu_percent":2400,"indexer.settings.max_writer_lock_prob":20,"indexer.settings.memProfFname":"","indexer.settings.memProfile":false,"indexer.settings.memory_quota":31457280000,"indexer.settings.moi.persistence_threads":96,"indexer.settings.moi.recovery_threads":48,"indexer.settings.persisted_snapshot.fdb.interval":5000,"indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.persisted_snapshot.moi.interval":600000,"indexer.settings.persisted_snapshot_init_build.fdb.interval":5000,"indexer.settings.persisted_snapshot_init_build.interval":5000,"indexer.settings.persisted_snapshot_init_build.moi.interval":600000,"indexer.settings.recovery.max_rollbacks":5,"indexer.settings.scan_getseqnos_retries":30,"indexer.settings.scan_timeout":0,"indexer.settings.send_buffer_size":1024,"indexer.settings.sliceBufSize":50000,"indexer.settings.smallSnapshotThreshold":30,"indexer.settings.statsLogDumpInterval":60,"indexer.settings.storage_mode":"forestdb","indexer.settings.wal_size":4096,"projector.settings.log_level":"info","queryport.client.settings.poolOverflow":30,"queryport.client.settings.poolSize":1000} 2016-04-07T01:29:08.217-07:00 [Info] Setting maxcpus = 24 2016-04-07T01:29:08.217-07:00 [Info] Setting log level to Info 2016-04-07T01:29:08.217-07:00 [Info] Indexer::ConfigUpdate Storage Mode Set forestdb 2016-04-07T01:29:08.217-07:00 [Info] Indexer: Setting GC percent to 200 2016-04-07T01:29:08.227-07:00 [Info] CompactionManager: Refreshing settings [goport] 2016/04/07 01:29:08 got new line on a stdin; terminating. 2016-04-07T01:29:08.307-07:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -vbuckets=1024 -cluster=127.0.0.1:8091 -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 -storageDir=/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -nodeUUID=103aa9f3d2294e264367f01b0999fb0e -storageMode=forestdb] 2016-04-07T01:29:08.308-07:00 [Info] Indexer::Main Storage Mode Set forestdb 2016-04-07T01:29:08.321-07:00 [Info] Indexer::NewIndexer Status Bootstrap 2016-04-07T01:29:08.324-07:00 [Info] Setting buffer block size to 16384 bytes 2016-04-07T01:29:08.324-07:00 [Info] Setting maxcpus = 24 2016-04-07T01:29:08.324-07:00 [Info] Setting log level to Info 2016-04-07T01:29:08.324-07:00 [Info] 127.0.0.1:8091 starting RESTful services 2016-04-07T01:29:08.326-07:00 [Info] New settings received: {"indexer.settings.bufferPoolBlockSize":16384,"indexer.settings.compaction.abort_exceed_interval":false,"indexer.settings.compaction.check_period":30,"indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.days_of_week":"sunday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.compaction.min_frag":30,"indexer.settings.compaction.min_size":524288000,"indexer.settings.cpuProfFname":"","indexer.settings.cpuProfile":false,"indexer.settings.fast_flush_mode":true,"indexer.settings.gc_percent":200,"indexer.settings.inmemory_snapshot.fdb.interval":200,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.inmemory_snapshot.moi.interval":20,"indexer.settings.largeSnapshotThreshold":200,"indexer.settings.log_level":"info","indexer.settings.maxVbQueueLength":0,"indexer.settings.max_array_seckey_size":10240,"indexer.settings.max_cpu_percent":2400,"indexer.settings.max_writer_lock_prob":20,"indexer.settings.memProfFname":"","indexer.settings.memProfile":false,"indexer.settings.memory_quota":31457280000,"indexer.settings.moi.persistence_threads":96,"indexer.settings.moi.recovery_threads":48,"indexer.settings.persisted_snapshot.fdb.interval":5000,"indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.persisted_snapshot.moi.interval":600000,"indexer.settings.persisted_snapshot_init_build.fdb.interval":5000,"indexer.settings.persisted_snapshot_init_build.interval":5000,"indexer.settings.persisted_snapshot_init_build.moi.interval":600000,"indexer.settings.recovery.max_rollbacks":5,"indexer.settings.scan_getseqnos_retries":30,"indexer.settings.scan_timeout":0,"indexer.settings.send_buffer_size":1024,"indexer.settings.sliceBufSize":50000,"indexer.settings.smallSnapshotThreshold":30,"indexer.settings.statsLogDumpInterval":60,"indexer.settings.storage_mode":"forestdb","indexer.settings.wal_size":4096,"projector.settings.log_level":"info","queryport.client.settings.poolOverflow":30,"queryport.client.settings.poolSize":1000} 2016-04-07T01:29:08.326-07:00 [Info] Setting maxcpus = 24 2016-04-07T01:29:08.326-07:00 [Info] Setting log level to Info 2016-04-07T01:29:08.373-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:29:08.373-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:29:08.373-07:00 [Error] WatcherServer.runOnce() error : dial tcp 172.23.99.247:9100: getsockopt: connection refused 2016-04-07T01:29:09.373-07:00 [Info] WatchMetadata(): unable to reach indexer at 172.23.99.247:9100. Retry in background. 2016-04-07T01:29:09.373-07:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2016-04-07T01:29:09.373-07:00 [Info] servicesChangeNotifier: Creating new notifier instance for http://%40index-cbauth:2d6c39e0d8f5be127d754756f7839391@127.0.0.1:8091, default 2016-04-07T01:29:09.375-07:00 [Error] WatcherServer.runOnce() error : dial tcp 172.23.99.247:9100: getsockopt: connection refused 2016-04-07T01:29:09.384-07:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2016-04-07T01:29:09.385-07:00 [Info] [Queryport ":9101"] started ... 2016-04-07T01:29:09.39-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:29:09.401-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:29:09.401-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:29:09.401-07:00 [Error] updateIndexerList(): Cannot locate cluster node. 2016-04-07T01:29:09.626-07:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2016-04-07T01:29:09.627-07:00 [Info] Indexer Id 52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:29:09.641-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.241:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:29:09.642-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.241:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:29:09.642-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.99.241:9999 2016-04-07T01:29:09.642-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:29:09.643-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.245:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:29:09.644-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.245:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:29:09.644-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.99.245:9999 2016-04-07T01:29:09.644-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:29:09.644-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.246:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:29:09.645-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.246:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:29:09.645-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.99.246:9999 2016-04-07T01:29:09.645-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:29:09.645-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.251:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:29:09.646-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.251:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:29:09.646-07:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 172.23.99.251:9999 2016-04-07T01:29:09.646-07:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:29:09.66-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.241:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:29:09.66-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.241:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:29:09.66-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.99.241:9999 2016-04-07T01:29:09.66-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:29:09.66-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.245:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:29:09.661-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.245:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:29:09.661-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.99.245:9999 2016-04-07T01:29:09.661-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:29:09.661-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.246:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:29:09.661-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.246:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:29:09.661-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.99.246:9999 2016-04-07T01:29:09.661-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:29:09.661-07:00 [Info] KVSender::sendShutdownTopic Projector 172.23.99.251:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:29:09.662-07:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 172.23.99.251:9999 Topic INIT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0. Err projector.topicMissing 2016-04-07T01:29:09.662-07:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 172.23.99.251:9999 2016-04-07T01:29:09.662-07:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2016-04-07T01:29:09.662-07:00 [Info] ClustMgr:handleGetGlobalTopology &{map[]} 2016-04-07T01:29:09.662-07:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap 2016-04-07T01:29:09.662-07:00 [Info] MutationMgr::handleUpdateIndexInstMap Message: MsgUpdateInstMap 2016-04-07T01:29:09.662-07:00 [Info] MutationMgr::handleUpdateIndexPartnMap Message: MsgUpdatePartnMap 2016-04-07T01:29:09.662-07:00 [Info] ClustMgr:handleIndexerReady &{52 [] {false false false false} 0} 2016-04-07T01:29:09.662-07:00 [Info] ClustMgr:handleSetLocalValue Key IndexerNodeUUID Value 103aa9f3d2294e264367f01b0999fb0e 2016-04-07T01:29:09.663-07:00 [Info] Indexer::NewIndexer Status Active 2016-04-07T01:29:09.663-07:00 [Info] Indexer::monitorMemUsage started... 2016-04-07T01:29:09.663-07:00 [Info] Indexer: Setting GC percent to 200 2016-04-07T01:29:09.673-07:00 [Info] memstats {"Alloc":0, "TotalAlloc":0, "Sys":0, "Lookups":0, "Mallocs":0,"Frees":0, "HeapAlloc":0, "HeapSys":0, "HeapIdle":0, "HeapInuse":0,"HeapReleased":0, "HeapObjects":0,"GCSys":0, "LastGC":0,"PauseTotalNs":0, "PauseNs":[], "NumGC":0} 2016-04-07T01:29:09.673-07:00 [Info] CompactionManager: Refreshing settings 2016-04-07T01:29:09.817-07:00 [Info] Indexer::ReadMemstats Time Taken 106.921µs 2016-04-07T01:29:11.376-07:00 [Info] WatchMetadata(): Successfully connected to indexer at 172.23.99.247:9100 after retry. 2016-04-07T01:29:11.376-07:00 [Info] Acknowledged that new indexer is registered. Indexer = 172.23.99.247:9100, id = 52:a9:25:ef:79:3b:5c:c0 2016-04-07T01:30:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:30:09.374-07:00 [Info] client load stats {} 2016-04-07T01:30:09.626-07:00 [Info] PeriodicStats = {"indexer_state":"Active","memory_quota":31457280000,"memory_used":53657600,"memory_used_storage":262144,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"72 3506228 212701811788","uptime":"1m1.320169941s"} ==== StorageStats ==== 2016-04-07T01:30:09.673-07:00 [Info] memstats {"Alloc":49727280, "TotalAlloc":52558824, "Sys":62441720, "Lookups":28, "Mallocs":65088,"Frees":52586, "HeapAlloc":49727280, "HeapSys":53182464, "HeapIdle":1679360, "HeapInuse":51503104,"HeapReleased":0, "HeapObjects":12502,"GCSys":1892352, "LastGC":1460017749673439006,"PauseTotalNs":12505580, "PauseNs":[487340, 358691, 1338809, 10320740], "NumGC":4} 2016-04-07T01:30:11.631-07:00 [Info] Indexer::ReadMemstats Time Taken 272.787µs 2016-04-07T01:31:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:31:09.374-07:00 [Info] client load stats {} 2016-04-07T01:31:09.627-07:00 [Info] PeriodicStats = {"indexer_state":"Active","memory_quota":31457280000,"memory_used":60956672,"memory_used_storage":262144,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"144 7369673 477331836997","uptime":"2m1.320446807s"} ==== StorageStats ==== 2016-04-07T01:31:09.673-07:00 [Info] memstats {"Alloc":57384896, "TotalAlloc":60216440, "Sys":68937976, "Lookups":87, "Mallocs":217111,"Frees":111034, "HeapAlloc":57384896, "HeapSys":59277312, "HeapIdle":671744, "HeapInuse":58605568,"HeapReleased":0, "HeapObjects":106077,"GCSys":2088960, "LastGC":1460017749673439006,"PauseTotalNs":12505580, "PauseNs":[], "NumGC":4} 2016-04-07T01:31:11.867-07:00 [Info] Indexer::ReadMemstats Time Taken 409.51µs 2016-04-07T01:32:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:32:09.374-07:00 [Info] client load stats {} 2016-04-07T01:32:09.627-07:00 [Info] PeriodicStats = {"indexer_state":"Active","memory_quota":31457280000,"memory_used":55115776,"memory_used_storage":262144,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"216 10842811 677341702173","uptime":"3m1.320670461s"} ==== StorageStats ==== 2016-04-07T01:32:09.674-07:00 [Info] memstats {"Alloc":50482680, "TotalAlloc":66966888, "Sys":75430136, "Lookups":136, "Mallocs":355787,"Frees":336093, "HeapAlloc":50482680, "HeapSys":65503232, "HeapIdle":12935168, "HeapInuse":52568064,"HeapReleased":0, "HeapObjects":19694,"GCSys":2285568, "LastGC":1460017869699234385,"PauseTotalNs":13044311, "PauseNs":[538731], "NumGC":5} 2016-04-07T01:32:12.103-07:00 [Info] Indexer::ReadMemstats Time Taken 142.555µs 2016-04-07T01:33:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:33:09.374-07:00 [Info] client load stats {} 2016-04-07T01:33:09.627-07:00 [Info] PeriodicStats = {"indexer_state":"Active","memory_quota":31457280000,"memory_used":61063168,"memory_used_storage":262144,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"288 14206885 862333260931","uptime":"4m1.320934451s"} ==== StorageStats ==== 2016-04-07T01:33:09.674-07:00 [Info] memstats {"Alloc":57189120, "TotalAlloc":73673328, "Sys":75430136, "Lookups":184, "Mallocs":494088,"Frees":389412, "HeapAlloc":57189120, "HeapSys":65437696, "HeapIdle":6922240, "HeapInuse":58515456,"HeapReleased":0, "HeapObjects":104676,"GCSys":2285568, "LastGC":1460017869699234385,"PauseTotalNs":13044311, "PauseNs":[], "NumGC":5} 2016-04-07T01:33:12.327-07:00 [Info] Indexer::ReadMemstats Time Taken 158.083µs 2016-04-07T01:34:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:34:09.374-07:00 [Info] client load stats {} 2016-04-07T01:34:09.627-07:00 [Info] PeriodicStats = {"indexer_state":"Active","memory_quota":31457280000,"memory_used":55222272,"memory_used_storage":262144,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"360 18181498 1164932099240","uptime":"5m1.321131353s"} ==== StorageStats ==== 2016-04-07T01:34:09.674-07:00 [Info] memstats {"Alloc":50481104, "TotalAlloc":80409400, "Sys":75430136, "Lookups":233, "Mallocs":632567,"Frees":612910, "HeapAlloc":50481104, "HeapSys":65470464, "HeapIdle":12795904, "HeapInuse":52674560,"HeapReleased":0, "HeapObjects":19657,"GCSys":2285568, "LastGC":1460017989826693413,"PauseTotalNs":13455544, "PauseNs":[411233], "NumGC":6} 2016-04-07T01:34:12.554-07:00 [Info] Indexer::ReadMemstats Time Taken 229.325µs 2016-04-07T01:35:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:35:09.374-07:00 [Info] client load stats {} 2016-04-07T01:35:09.628-07:00 [Info] PeriodicStats = {"indexer_state":"Active","memory_quota":31457280000,"memory_used":61267968,"memory_used_storage":262144,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"432 21882852 1385676533770","uptime":"6m1.321363497s"} ==== StorageStats ==== 2016-04-07T01:35:09.674-07:00 [Info] memstats {"Alloc":57562904, "TotalAlloc":87491200, "Sys":75430136, "Lookups":294, "Mallocs":774425,"Frees":667186, "HeapAlloc":57562904, "HeapSys":65404928, "HeapIdle":6684672, "HeapInuse":58720256,"HeapReleased":0, "HeapObjects":107239,"GCSys":2285568, "LastGC":1460017989826693413,"PauseTotalNs":13455544, "PauseNs":[], "NumGC":6} 2016-04-07T01:35:12.782-07:00 [Info] Indexer::ReadMemstats Time Taken 322.093µs 2016-04-07T01:36:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:36:09.374-07:00 [Info] client load stats {} 2016-04-07T01:36:09.628-07:00 [Info] PeriodicStats = {"indexer_state":"Active","memory_quota":31457280000,"memory_used":55320576,"memory_used_storage":262144,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"504 25789169 1636391725061","uptime":"7m1.321638465s"} ==== StorageStats ==== 2016-04-07T01:36:09.675-07:00 [Info] memstats {"Alloc":50550160, "TotalAlloc":94225408, "Sys":75430136, "Lookups":342, "Mallocs":913014,"Frees":893207, "HeapAlloc":50550160, "HeapSys":65437696, "HeapIdle":12664832, "HeapInuse":52772864,"HeapReleased":0, "HeapObjects":19807,"GCSys":2285568, "LastGC":1460018110386037734,"PauseTotalNs":13976104, "PauseNs":[520560], "NumGC":7} 2016-04-07T01:36:13.007-07:00 [Info] Indexer::ReadMemstats Time Taken 168.156µs 2016-04-07T01:36:23.03-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:36:23.044-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:36:23.044-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:37:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:37:09.374-07:00 [Info] client load stats {} 2016-04-07T01:37:09.628-07:00 [Info] PeriodicStats = {"indexer_state":"Active","memory_quota":31457280000,"memory_used":61128704,"memory_used_storage":262144,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"576 29569793 1867639045453","uptime":"8m1.321942575s"} ==== StorageStats ==== 2016-04-07T01:37:09.675-07:00 [Info] memstats {"Alloc":57275832, "TotalAlloc":100951080, "Sys":75430136, "Lookups":391, "Mallocs":1051417,"Frees":946539, "HeapAlloc":57275832, "HeapSys":65437696, "HeapIdle":6856704, "HeapInuse":58580992,"HeapReleased":0, "HeapObjects":104878,"GCSys":2285568, "LastGC":1460018110386037734,"PauseTotalNs":13976104, "PauseNs":[], "NumGC":7} 2016-04-07T01:37:13.228-07:00 [Info] Indexer::ReadMemstats Time Taken 108.102µs 2016-04-07T01:38:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:38:09.374-07:00 [Info] client load stats {} 2016-04-07T01:38:09.629-07:00 [Info] PeriodicStats = {"indexer_state":"Active","memory_quota":31457280000,"memory_used":55427072,"memory_used_storage":262144,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"648 33205906 2084938077836","uptime":"9m1.322210497s"} ==== StorageStats ==== 2016-04-07T01:38:09.675-07:00 [Info] memstats {"Alloc":50564240, "TotalAlloc":108208008, "Sys":76515576, "Lookups":440, "Mallocs":1201189,"Frees":1181325, "HeapAlloc":50564240, "HeapSys":66486272, "HeapIdle":13639680, "HeapInuse":52846592,"HeapReleased":0, "HeapObjects":19864,"GCSys":2318336, "LastGC":1460018230826671574,"PauseTotalNs":14313968, "PauseNs":[337864], "NumGC":8} 2016-04-07T01:38:13.457-07:00 [Info] Indexer::ReadMemstats Time Taken 151.046µs 2016-04-07T01:39:08.553-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:39:08.567-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:39:08.567-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:39:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:39:09.374-07:00 [Info] client load stats {} 2016-04-07T01:39:09.629-07:00 [Info] PeriodicStats = {"indexer_state":"Active","memory_quota":31457280000,"memory_used":61161472,"memory_used_storage":262144,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"720 36955543 2315238478947","uptime":"10m1.322428937s"} ==== StorageStats ==== 2016-04-07T01:39:09.675-07:00 [Info] memstats {"Alloc":57274736, "TotalAlloc":114918504, "Sys":76515576, "Lookups":489, "Mallocs":1339496,"Frees":1234642, "HeapAlloc":57274736, "HeapSys":66486272, "HeapIdle":7905280, "HeapInuse":58580992,"HeapReleased":0, "HeapObjects":104854,"GCSys":2318336, "LastGC":1460018230826671574,"PauseTotalNs":14313968, "PauseNs":[], "NumGC":8} 2016-04-07T01:39:13.68-07:00 [Info] Indexer::ReadMemstats Time Taken 115.839µs 2016-04-07T01:40:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:40:09.374-07:00 [Info] client load stats {} 2016-04-07T01:40:09.629-07:00 [Info] PeriodicStats = {"indexer_state":"Active","memory_quota":31457280000,"memory_used":55451648,"memory_used_storage":262144,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"792 41039410 2578628259934","uptime":"11m1.322682808s"} ==== StorageStats ==== 2016-04-07T01:40:09.675-07:00 [Info] memstats {"Alloc":50553928, "TotalAlloc":122494536, "Sys":76515576, "Lookups":537, "Mallocs":1492583,"Frees":1472793, "HeapAlloc":50553928, "HeapSys":66519040, "HeapIdle":13647872, "HeapInuse":52871168,"HeapReleased":0, "HeapObjects":19790,"GCSys":2318336, "LastGC":1460018351393616308,"PauseTotalNs":14758651, "PauseNs":[444683], "NumGC":9} 2016-04-07T01:40:13.907-07:00 [Info] Indexer::ReadMemstats Time Taken 150.345µs 2016-04-07T01:40:20.565-07:00 [Info] clustMgrAgent::OnIndexCreate Notification Received for Create Index DefnId: 8343618505678776570 Name: myindex Using: forestdb Bucket: bucket-1 IsPrimary: false SecExprs: [`alt_email`] PartitionScheme: SINGLE PartitionKey: WhereExpr: 2016-04-07T01:40:20.566-07:00 [Info] Indexer::handleCreateIndex InstId: 8343618505678776570 Defn: DefnId: 8343618505678776570 Name: myindex Using: forestdb Bucket: bucket-1 IsPrimary: false SecExprs: [`alt_email`] PartitionScheme: SINGLE PartitionKey: WhereExpr: State: INDEX_STATE_CREATED Stream: NIL_STREAM PartitionContainer: &{map[1:{1 [:9105]}] 1} 2016-04-07T01:40:20.579-07:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 8343618505678776570 Partition: PartitionId: 1 Endpoints: [:9105] 2016-04-07T01:40:20.584-07:00 [Info] NewForestDBSlice(): file version ForestDB v2.x format 2016-04-07T01:40:20.586-07:00 [Info] ForestDBSlice:NewForestDBSlice Created New Slice Id 0 IndexInstId 8343618505678776570 WriterThreads 1 2016-04-07T01:40:20.586-07:00 [Info] Indexer::initPartnInstance Initialized Slice: Index: 8343618505678776570 Slice: SliceId: 0 File: /data/@2i/bucket-1_myindex_8343618505678776570_0.index Index: 8343618505678776570 2016-04-07T01:40:20.588-07:00 [Info] MutationMgr::handleUpdateIndexInstMap Message: MsgUpdateInstMap InstanceId: 8343618505678776570 Name: myindex Bucket: bucket-1 State: INDEX_STATE_CREATED Stream: NIL_STREAM 2016-04-07T01:40:20.588-07:00 [Info] MutationMgr::handleUpdateIndexPartnMap Message: MsgUpdatePartnMap InstanceId: 8343618505678776570 PartitionId: 0 Endpoints: [:9105] 2016-04-07T01:40:20.588-07:00 [Info] clustMgrAgent::OnIndexCreate Success for Create Index DefnId: 8343618505678776570 Name: myindex Using: forestdb Bucket: bucket-1 IsPrimary: false SecExprs: [`alt_email`] PartitionScheme: SINGLE PartitionKey: WhereExpr: 2016-04-07T01:40:20.6-07:00 [Info] messageListener.start(): message channel closed. Remove peer 172.23.99.241/22:indexer:MetadataProvider:3b:b6:59:b8:d2:13:80:ed as follower. 2016-04-07T01:40:23.974-07:00 [Info] {bucket,feeds} "bucket-1" created for dcp_seqno cache... 2016-04-07T01:40:23.999-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:40:24.012-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:40:24.012-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:40:30.698-07:00 [Info] clustMgrAgent::OnIndexBuild Notification Received for Build Index [8343618505678776570] 2016-04-07T01:40:30.698-07:00 [Info] Indexer::handleBuildIndex [8343618505678776570] 2016-04-07T01:40:30.716-07:00 [Info] Indexer::handleBuildIndex Added Index: [8343618505678776570] to Stream: MAINT_STREAM State: INDEX_STATE_INITIAL 2016-04-07T01:40:30.716-07:00 [Info] MutationMgr::handleUpdateIndexInstMap Message: MsgUpdateInstMap InstanceId: 8343618505678776570 Name: myindex Bucket: bucket-1 State: INDEX_STATE_INITIAL Stream: MAINT_STREAM 2016-04-07T01:40:30.716-07:00 [Info] Timekeeper::handleStreamOpen Stream MAINT_STREAM State Changed to ACTIVE 2016-04-07T01:40:30.716-07:00 [Info] StreamState::initBucketInStream New Bucket bucket-1 Added for Stream MAINT_STREAM 2016-04-07T01:40:30.716-07:00 [Info] Timekeeper::addIndextoStream IndexCount map[MAINT_STREAM:map[bucket-1:1]] 2016-04-07T01:40:30.716-07:00 [Info] Timekeeper::addIndextoStream add BuildInfo index 8343618505678776570 stream MAINT_STREAM bucket bucket-1 state INDEX_STATE_INITIAL 2016-04-07T01:40:30.716-07:00 [Info] Timekeeper::startTimer MAINT_STREAM bucket-1 2016-04-07T01:40:30.716-07:00 [Info] MutationMgr::handleOpenStream Message: MsgStreamUpdate Type: OPEN_STREAM Stream: MAINT_STREAM Bucket: bucket-1 BuildTS: [0=19535,1=19525,2=19535,3=19530,4=19535,5=19530,6=19535,7=19525,8=19535,9=19525,10=19535,11=19530,12=19535,13=19530,14=19535,15=19525,16=19535,17=19525,18=19536,19=19530,20=19535,21=19530,22=19535,23=19525,24=19535,25=19525,26=19535,27=19530,28=19535,29=19530,30=19535,31=19525,32=19535,33=19525,34=19535,35=19530,36=19535,37=19530,38=19535,39=19525,40=19535,41=19525,42=19535,43=19530,44=19535,45=19530,46=19535,47=19525,48=19535,49=19525,50=19535,51=19530,52=19535,53=19530,54=19535,55=19525,56=19535,57=19525,58=19535,59=19530,60=19535,61=19530,62=19535,63=19525,64=19530,65=19535,66=19525,67=19535,68=19525,69=19535,70=19530,71=19535,72=19530,73=19535,74=19525,75=19535,76=19525,77=19535,78=19530,79=19535,80=19530,81=19535,82=19525,83=19535,84=19525,85=19535,86=19530,87=19535,88=19530,89=19535,90=19525,91=19535,92=19525,93=19535,94=19530,95=19535,96=19530,97=19535,98=19525,99=19535,100=19525,101=19535,102=19530,103=19535,104=19530,105=19535,106=19525,107=19535,108=19525,109=19535,110=19530,111=19535,112=19530,113=19535,114=19525,115=19535,116=19525,117=19535,118=19530,119=19535,120=19530,121=19535,122=19525,123=19535,124=19525,125=19535,126=19530,127=19535,128=19535,129=19530,130=19535,131=19525,132=19535,133=19525,134=19535,135=19530,136=19535,137=19530,138=19535,139=19525,140=19535,141=19525,142=19535,143=19530,144=19535,145=19530,146=19535,147=19525,148=19535,149=19525,150=19535,151=19530,152=19535,153=19530,154=19535,155=19525,156=19535,157=19525,158=19535,159=19530,160=19535,161=19530,162=19535,163=19525,164=19535,165=19525,166=19535,167=19530,168=19535,169=19530,170=19535,171=19525,172=19535,173=19525,174=19535,175=19530,176=19535,177=19530,178=19535,179=19525,180=19535,181=19525,182=19535,183=19530,184=19535,185=19530,186=19535,187=19525,188=19535,189=19525,190=19535,191=19530,192=19525,193=19535,194=19530,195=19535,196=19530,197=19535,198=19525,199=19535,200=19525,201=19535,202=19530,203=19535,204=19530,205=19535,206=19525,207=19535,208=19525,209=19535,210=19530,211=19535,212=19530,213=19535,214=19525,215=19535,216=19525,217=19535,218=19530,219=19535,220=19530,221=19535,222=19525,223=19535,224=19525,225=19535,226=19530,227=19535,228=19530,229=19535,230=19525,231=19535,232=19525,233=19535,234=19530,235=19535,236=19530,237=19535,238=19525,239=19535,240=19525,241=19535,242=19530,243=19535,244=19530,245=19535,246=19525,247=19535,248=19525,249=19535,250=19530,251=19535,252=19530,253=19535,254=19525,255=19535,256=19530,257=19535,258=19525,259=19535,260=19525,261=19535,262=19530,263=19535,264=19530,265=19535,266=19525,267=19535,268=19525,269=19535,270=19530,271=19535,272=19530,273=19535,274=19525,275=19535,276=19525,277=19535,278=19530,279=19535,280=19530,281=19535,282=19525,283=19535,284=19525,285=19535,286=19530,287=19535,288=19530,289=19535,290=19525,291=19535,292=19525,293=19535,294=19530,295=19535,296=19530,297=19535,298=19525,299=19535,300=19525,301=19535,302=19530,303=19535,304=19530,305=19535,306=19525,307=19535,308=19525,309=19535,310=19530,311=19535,312=19530,313=19535,314=19525,315=19535,316=19525,317=19535,318=19530,319=19535,320=19535,321=19525,322=19535,323=19530,324=19535,325=19530,326=19535,327=19525,328=19535,329=19525,330=19535,331=19530,332=19535,333=19530,334=19535,335=19525,336=19535,337=19525,338=19535,339=19530,340=19535,341=19530,342=19535,343=19525,344=19535,345=19525,346=19535,347=19530,348=19535,349=19530,350=19535,351=19525,352=19535,353=19525,354=19535,355=19530,356=19535,357=19530,358=19535,359=19525,360=19535,361=19525,362=19535,363=19530,364=19535,365=19530,366=19535,367=19525,368=19535,369=19525,370=19535,371=19530,372=19535,373=19530,374=19535,375=19525,376=19535,377=19525,378=19535,379=19530,380=19535,381=19530,382=19535,383=19525,384=19525,385=19535,386=19530,387=19535,388=19530,389=19535,390=19525,391=19535,392=19525,393=19535,394=19530,395=19535,396=19530,397=19535,398=19525,399=19535,400=19525,401=19535,402=19530,403=19535,404=19530,405=19535,406=19525,407=19535,408=19525,409=19535,410=19530,411=19535,412=19530,413=19535,414=19525,415=19535,416=19525,417=19535,418=19530,419=19535,420=19530,421=19535,422=19525,423=19535,424=19525,425=19535,426=19530,427=19535,428=19530,429=19535,430=19525,431=19535,432=19525,433=19535,434=19530,435=19535,436=19530,437=19535,438=19525,439=19535,440=19525,441=19535,442=19530,443=19535,444=19530,445=19535,446=19525,447=19535,448=19535,449=19530,450=19535,451=19525,452=19535,453=19525,454=19535,455=19530,456=19535,457=19530,458=19535,459=19525,460=19535,461=19525,462=19535,463=19530,464=19535,465=19530,466=19535,467=19525,468=19535,469=19525,470=19535,471=19530,472=19535,473=19530,474=19535,475=19525,476=19535,477=19525,478=19535,479=19530,480=19535,481=19530,482=19535,483=19525,484=19535,485=19525,486=19535,487=19530,488=19535,489=19530,490=19535,491=19525,492=19535,493=19525,494=19535,495=19530,496=19535,497=19530,498=19535,499=19525,500=19535,501=19525,502=19535,503=19530,504=19535,505=19530,506=19535,507=19525,508=19535,509=19525,510=19535,511=19530,512=19535,513=19530,514=19535,515=19525,516=19535,517=19525,518=19535,519=19530,520=19535,521=19530,522=19535,523=19525,524=19535,525=19525,526=19535,527=19530,528=19535,529=19530,530=19535,531=19525,532=19535,533=19525,534=19535,535=19530,536=19535,537=19530,538=19535,539=19525,540=19535,541=19525,542=19535,543=19530,544=19535,545=19530,546=19535,547=19525,548=19535,549=19525,550=19535,551=19530,552=19535,553=19530,554=19535,555=19525,556=19535,557=19525,558=19535,559=19530,560=19535,561=19530,562=19535,563=19525,564=19535,565=19525,566=19535,567=19529,568=19535,569=19530,570=19535,571=19525,572=19535,573=19525,574=19535,575=19530,576=19525,577=19535,578=19530,579=19535,580=19530,581=19535,582=19525,583=19535,584=19525,585=19535,586=19530,587=19535,588=19530,589=19535,590=19525,591=19535,592=19525,593=19535,594=19530,595=19535,596=19530,597=19535,598=19525,599=19535,600=19525,601=19535,602=19530,603=19535,604=19530,605=19535,606=19525,607=19535,608=19525,609=19535,610=19530,611=19535,612=19530,613=19535,614=19525,615=19535,616=19525,617=19535,618=19530,619=19535,620=19530,621=19535,622=19525,623=19535,624=19525,625=19535,626=19530,627=19535,628=19530,629=19535,630=19525,631=19535,632=19525,633=19535,634=19530,635=19535,636=19530,637=19535,638=19525,639=19535,640=19535,641=19525,642=19535,643=19530,644=19535,645=19530,646=19535,647=19525,648=19535,649=19525,650=19535,651=19530,652=19535,653=19530,654=19535,655=19525,656=19535,657=19525,658=19535,659=19530,660=19535,661=19530,662=19535,663=19525,664=19535,665=19525,666=19535,667=19530,668=19535,669=19530,670=19535,671=19525,672=19535,673=19525,674=19535,675=19530,676=19535,677=19530,678=19535,679=19525,680=19535,681=19525,682=19535,683=19530,684=19535,685=19530,686=19535,687=19525,688=19535,689=19525,690=19535,691=19530,692=19535,693=19530,694=19535,695=19525,696=19535,697=19525,698=19535,699=19530,700=19535,701=19530,702=19535,703=19525,704=19530,705=19535,706=19525,707=19535,708=19525,709=19535,710=19530,711=19535,712=19530,713=19535,714=19525,715=19535,716=19525,717=19535,718=19530,719=19535,720=19530,721=19535,722=19525,723=19535,724=19525,725=19535,726=19530,727=19535,728=19530,729=19535,730=19525,731=19535,732=19525,733=19535,734=19530,735=19535,736=19530,737=19535,738=19525,739=19535,740=19525,741=19535,742=19530,743=19535,744=19530,745=19535,746=19525,747=19535,748=19525,749=19535,750=19530,751=19535,752=19530,753=19535,754=19525,755=19535,756=19525,757=19535,758=19530,759=19535,760=19530,761=19535,762=19525,763=19535,764=19525,765=19535,766=19530,767=19535,768=19525,769=19535,770=19530,771=19535,772=19530,773=19535,774=19525,775=19535,776=19525,777=19535,778=19530,779=19535,780=19530,781=19535,782=19525,783=19535,784=19525,785=19535,786=19530,787=19535,788=19530,789=19535,790=19525,791=19535,792=19525,793=19535,794=19530,795=19535,796=19530,797=19535,798=19525,799=19535,800=19525,801=19535,802=19530,803=19535,804=19530,805=19535,806=19525,807=19535,808=19525,809=19535,810=19530,811=19535,812=19530,813=19535,814=19525,815=19535,816=19525,817=19535,818=19530,819=19535,820=19530,821=19535,822=19525,823=19535,824=19525,825=19535,826=19530,827=19535,828=19530,8 29=19535,830=19525,831=19535,832=19535,833=19530,834=19535,835=19525,836=19535,837=19525,838=19535,839=19530,840=19535,841=19530,842=19535,843=19525,844=19535,845=19525,846=19535,847=19530,848=19535,849=19530,850=19535,851=19525,852=19535,853=19525,854=19535,855=19530,856=19535,857=19530,858=19535,859=19525,860=19535,861=19525,862=19535,863=19530,864=19535,865=19530,866=19535,867=19525,868=19535,869=19525,870=19535,871=19530,872=19535,873=19530,874=19535,875=19525,876=19535,877=19525,878=19535,879=19530,880=19535,881=19530,882=19535,883=19525,884=19535,885=19525,886=19535,887=19530,888=19535,889=19530,890=19535,891=19525,892=19535,893=19525,894=19535,895=19530,896=19530,897=19535,898=19525,899=19535,900=19525,901=19535,902=19530,903=19535,904=19530,905=19535,906=19525,907=19535,908=19525,909=19535,910=19530,911=19535,912=19530,913=19535,914=19525,915=19535,916=19525,917=19535,918=19530,919=19535,920=19530,921=19535,922=19525,923=19535,924=19525,925=19535,926=19530,927=19535,928=19530,929=19535,930=19525,931=19535,932=19525,933=19535,934=19530,935=19535,936=19530,937=19535,938=19525,939=19535,940=19525,941=19535,942=19530,943=19535,944=19530,945=19535,946=19525,947=19535,948=19525,949=19535,950=19530,951=19535,952=19530,953=19535,954=19525,955=19535,956=19525,957=19535,958=19530,959=19535,960=19535,961=19525,962=19535,963=19530,964=19535,965=19530,966=19535,967=19525,968=19535,969=19525,970=19535,971=19530,972=19535,973=19530,974=19535,975=19525,976=19535,977=19525,978=19535,979=19530,980=19535,981=19530,982=19535,983=19525,984=19535,985=19525,986=19535,987=19530,988=19535,989=19530,990=19535,991=19525,992=19535,993=19525,994=19535,995=19530,996=19535,997=19530,998=19535,999=19525,1000=19535,1001=19525,1002=19535,1003=19530,1004=19535,1005=19530,1006=19535,1007=19525,1008=19535,1009=19525,1010=19535,1011=19530,1012=19535,1013=19530,1014=19535,1015=19525,1016=19535,1017=19525,1018=19535,1019=19530,1020=19535,1021=19530,1022=19535,1023=19525] IndexList: [ InstId: 8343618505678776570 Defn: DefnId: 8343618505678776570 Name: myindex Using: forestdb Bucket: bucket-1 IsPrimary: false SecExprs: [`alt_email`] PartitionScheme: SINGLE PartitionKey: WhereExpr: State: INDEX_STATE_INITIAL Stream: MAINT_STREAM PartitionContainer: &{map[1:{1 [:9105]}] 1}] RestartTs: 2016-04-07T01:40:30.723-07:00 [Info] MutationMgr:: Set maxVbQueueLength 10000 2016-04-07T01:40:30.725-07:00 [Info] DATP[->dataport ":9105"] started ... 2016-04-07T01:40:30.725-07:00 [Info] MutationStreamReader::startWorkers Starting All Stream Workers MAINT_STREAM 2016-04-07T01:40:30.725-07:00 [Info] MutationStreamReader::startMutationStreamWorker Stream Worker 0 Started for Stream MAINT_STREAM. 2016-04-07T01:40:30.725-07:00 [Info] ClustMgr:handleUpdateTopologyForIndex &{47 [{8343618505678776570 {8343618505678776570 myindex forestdb bucket-1 6311cb70dafcf851cc4e85775abf6ceb false [`alt_email`] N1QL SINGLE true false [52:a9:25:ef:79:3b:5c:c0] false} 2 1 0xc823805600 []}] {true true false true} 0} 2016-04-07T01:40:30.725-07:00 [Info] clustMgrAgent::OnIndexBuild returns for Build Index [8343618505678776570] 2016-04-07T01:40:30.733-07:00 [Info] messageListener.start(): message channel closed. Remove peer 172.23.99.241/22:indexer:MetadataProvider:fb:8c:a2:54:db:fe:39:bf as follower. 2016-04-07T01:40:31.015-07:00 [Info] KVSender::sendMutationTopicRequest Projector 172.23.99.241:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 bucket-1 Instances [indexInstance: singlePartn: > ] 2016-04-07T01:40:31.031-07:00 [Info] DATP[->dataport ":9105"] new connection "172.23.99.241:53867" +1 2016-04-07T01:40:31.273-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 2 114798468367539 0 2016-04-07T01:40:31.273-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 3 84407419725585 0 2016-04-07T01:40:31.273-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 4 205127652194672 0 2016-04-07T01:40:31.273-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 0 173538236354674 0 2016-04-07T01:40:31.273-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1 197785722352298 0 2016-04-07T01:40:31.28-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 6 273930640396784 0 2016-04-07T01:40:31.28-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 7 174989311891452 0 2016-04-07T01:40:31.28-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 9 13136714059451 0 2016-04-07T01:40:31.28-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 5 142054507137807 0 2016-04-07T01:40:31.28-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 8 232332841968847 0 2016-04-07T01:40:31.28-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1 2016-04-07T01:40:31.281-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 10 237238456749906 0 2016-04-07T01:40:31.281-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 11 250052736560364 0 2016-04-07T01:40:31.281-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 12 41670096544201 0 2016-04-07T01:40:31.287-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 17 70107510098365 0 2016-04-07T01:40:31.287-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 18 67496329833860 0 2016-04-07T01:40:31.287-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 13 224183131074657 0 2016-04-07T01:40:31.287-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 14 68875940455786 0 2016-04-07T01:40:31.287-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 15 221944898372221 0 2016-04-07T01:40:31.287-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 16 269022798921091 0 2016-04-07T01:40:31.293-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 20 250109880549627 0 2016-04-07T01:40:31.293-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 21 1574555877059 0 2016-04-07T01:40:31.293-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 22 12711634423989 0 2016-04-07T01:40:31.293-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 23 132432538316653 0 2016-04-07T01:40:31.293-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 19 53061323761081 0 2016-04-07T01:40:31.294-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 24 89527764492150 0 2016-04-07T01:40:31.301-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 26 66132062009978 0 2016-04-07T01:40:31.301-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 27 254295434490798 0 2016-04-07T01:40:31.301-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 28 120866834039305 0 2016-04-07T01:40:31.301-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 25 265486015670846 0 2016-04-07T01:40:31.307-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 29 199902849166047 0 2016-04-07T01:40:31.307-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 30 226895076559947 0 2016-04-07T01:40:31.312-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 31 205360631692437 0 2016-04-07T01:40:31.317-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:40:31.317-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1 QueuedCount 92 2016-04-07T01:40:31.318-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 32 26237782892397 0 2016-04-07T01:40:31.318-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 33 248781905315179 0 2016-04-07T01:40:31.326-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 35 149692309343035 0 2016-04-07T01:40:31.326-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 34 234719077590186 0 2016-04-07T01:40:31.333-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 36 239193738188306 0 2016-04-07T01:40:31.333-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 37 97064532775008 0 2016-04-07T01:40:31.339-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 38 136201918118380 0 2016-04-07T01:40:31.339-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 39 17899108026766 0 2016-04-07T01:40:31.344-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 40 168926978002054 0 2016-04-07T01:40:31.344-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 41 229080663657283 0 2016-04-07T01:40:31.351-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 42 155001004486010 0 2016-04-07T01:40:31.356-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 43 118582436101158 0 2016-04-07T01:40:31.362-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 44 108981141646107 0 2016-04-07T01:40:31.362-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 45 132521440342036 0 2016-04-07T01:40:31.368-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 46 35000493830000 0 2016-04-07T01:40:31.368-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 47 13032987282281 0 2016-04-07T01:40:31.376-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 48 159809972435570 0 2016-04-07T01:40:31.376-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 49 117062754271781 0 2016-04-07T01:40:31.382-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 51 246639028062303 0 2016-04-07T01:40:31.382-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 50 75703174569082 0 2016-04-07T01:40:31.39-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 52 64411213515897 0 2016-04-07T01:40:31.39-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 53 7355728773417 0 2016-04-07T01:40:31.394-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 54 234431458298494 0 2016-04-07T01:40:31.403-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 55 133304060944530 0 2016-04-07T01:40:31.403-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 56 189814084071578 0 2016-04-07T01:40:31.412-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 57 20970358528723 0 2016-04-07T01:40:31.412-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 58 202701390181768 0 2016-04-07T01:40:31.42-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 59 155375689980811 0 2016-04-07T01:40:31.426-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 60 93799547720772 0 2016-04-07T01:40:31.432-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 61 165143515685486 0 2016-04-07T01:40:31.432-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 62 161636437037811 0 2016-04-07T01:40:31.438-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 63 189151989047856 0 2016-04-07T01:40:31.442-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 66 156195961386874 0 2016-04-07T01:40:31.442-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 67 40016158994246 0 2016-04-07T01:40:31.445-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 68 269121184071781 0 2016-04-07T01:40:31.448-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 69 25027512518570 0 2016-04-07T01:40:31.448-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 70 104406236421678 0 2016-04-07T01:40:31.452-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 71 131268827913510 0 2016-04-07T01:40:31.453-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 72 12281558960790 0 2016-04-07T01:40:31.457-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 73 277198371411930 0 2016-04-07T01:40:31.457-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 74 206561635061722 0 2016-04-07T01:40:31.462-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 75 194309319960235 0 2016-04-07T01:40:31.466-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 76 278623955763993 0 2016-04-07T01:40:31.469-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 77 46807069325328 0 2016-04-07T01:40:31.476-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 78 50799447075526 0 2016-04-07T01:40:31.481-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 79 246642305941102 0 2016-04-07T01:40:31.485-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 80 247426227428478 0 2016-04-07T01:40:31.491-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 81 208310322288528 0 2016-04-07T01:40:31.495-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 82 103942126922340 0 2016-04-07T01:40:31.495-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 65 97171924873612 0 2016-04-07T01:40:31.5-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 83 149219672538424 0 2016-04-07T01:40:31.5-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 84 183503875416385 0 2016-04-07T01:40:31.504-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 85 63384949598256 0 2016-04-07T01:40:31.509-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 64 280906836058649 0 2016-04-07T01:40:31.509-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 86 41083203422522 0 2016-04-07T01:40:31.512-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 87 44864530384623 0 2016-04-07T01:40:31.516-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 88 199214506439146 0 2016-04-07T01:40:31.516-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 89 92726710211218 0 2016-04-07T01:40:31.519-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 90 244110497980544 0 2016-04-07T01:40:31.519-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:40:31.52-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 91 259520501161352 0 2016-04-07T01:40:31.523-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 92 62078328436821 0 2016-04-07T01:40:31.525-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 93 168746410939873 0 2016-04-07T01:40:31.528-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 95 170721282385778 0 2016-04-07T01:40:31.528-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 94 106882606305708 0 2016-04-07T01:40:31.53-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 96 189753971513548 0 2016-04-07T01:40:31.53-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 97 74041053522787 0 2016-04-07T01:40:31.532-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 98 232327496149120 0 2016-04-07T01:40:31.535-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 99 83919878959652 0 2016-04-07T01:40:31.535-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 100 144842375441927 0 2016-04-07T01:40:31.538-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 101 172724382122929 0 2016-04-07T01:40:31.54-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 102 109680680442198 0 2016-04-07T01:40:31.54-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 103 76031267301763 0 2016-04-07T01:40:31.543-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 104 242627748363637 0 2016-04-07T01:40:31.543-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 105 65993134285291 0 2016-04-07T01:40:31.545-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 106 169569535389249 0 2016-04-07T01:40:31.547-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 107 165825428043343 0 2016-04-07T01:40:31.547-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 108 7970293356831 0 2016-04-07T01:40:31.549-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 109 29810420255054 0 2016-04-07T01:40:31.551-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 110 77251621067528 0 2016-04-07T01:40:31.551-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 111 15053035114465 0 2016-04-07T01:40:31.553-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 112 130950632191738 0 2016-04-07T01:40:31.555-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 113 58911113705016 0 2016-04-07T01:40:31.557-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 114 32108676904306 0 2016-04-07T01:40:31.562-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 115 159185114810778 0 2016-04-07T01:40:31.564-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 116 193351020663597 0 2016-04-07T01:40:31.564-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 117 79563897309974 0 2016-04-07T01:40:31.566-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 118 274777222404367 0 2016-04-07T01:40:31.568-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 119 238114280518572 0 2016-04-07T01:40:31.568-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 120 240840131708840 0 2016-04-07T01:40:31.57-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 121 279903550898275 0 2016-04-07T01:40:31.57-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 122 63879963736202 0 2016-04-07T01:40:31.574-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 125 102404139558897 0 2016-04-07T01:40:31.574-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 123 65811684106545 0 2016-04-07T01:40:31.577-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 124 174719590159456 0 2016-04-07T01:40:31.577-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 126 125081120933971 0 2016-04-07T01:40:31.581-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 127 254764106025487 0 2016-04-07T01:40:31.591-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 136 25745861623430 0 2016-04-07T01:40:31.592-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 137 205651435775961 0 2016-04-07T01:40:31.592-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 135 232328660690763 0 2016-04-07T01:40:31.595-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 139 89947697199670 0 2016-04-07T01:40:31.595-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 140 67527822498729 0 2016-04-07T01:40:31.596-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 138 103088336646075 0 2016-04-07T01:40:31.596-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 141 13860553079946 0 2016-04-07T01:40:31.598-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 142 102705338895038 0 2016-04-07T01:40:31.599-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 144 173860495614263 0 2016-04-07T01:40:31.599-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 132 233849902759709 0 2016-04-07T01:40:31.599-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 133 118293531734410 0 2016-04-07T01:40:31.599-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 143 213394107658811 0 2016-04-07T01:40:31.601-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 145 118563316403232 0 2016-04-07T01:40:31.603-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 146 161895606167044 0 2016-04-07T01:40:31.603-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 147 159412699485291 0 2016-04-07T01:40:31.603-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 134 165636926177000 0 2016-04-07T01:40:31.604-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 129 7629899133637 0 2016-04-07T01:40:31.604-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 148 11599150079109 0 2016-04-07T01:40:31.606-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 130 84345837499707 0 2016-04-07T01:40:31.606-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 150 64382358772680 0 2016-04-07T01:40:31.606-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 149 95744290618947 0 2016-04-07T01:40:31.607-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 151 167049782272626 0 2016-04-07T01:40:31.607-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 152 149696220894385 0 2016-04-07T01:40:31.609-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 153 233386657461169 0 2016-04-07T01:40:31.61-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 154 146917836148714 0 2016-04-07T01:40:31.612-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 155 264780240273636 0 2016-04-07T01:40:31.612-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 156 28305507833185 0 2016-04-07T01:40:31.613-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 158 88507150332505 0 2016-04-07T01:40:31.613-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 157 217414814195349 0 2016-04-07T01:40:31.615-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 159 13833582796449 0 2016-04-07T01:40:31.617-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 160 179029527057394 0 2016-04-07T01:40:31.618-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 161 212249672071299 0 2016-04-07T01:40:31.62-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 162 219071429127582 0 2016-04-07T01:40:31.621-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 163 258514398466416 0 2016-04-07T01:40:31.623-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 165 92501748246563 0 2016-04-07T01:40:31.623-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 164 240891157822705 0 2016-04-07T01:40:31.624-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 167 115618085097832 0 2016-04-07T01:40:31.624-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 166 137887628771016 0 2016-04-07T01:40:31.626-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 169 23044413905719 0 2016-04-07T01:40:31.626-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 168 83236147998816 0 2016-04-07T01:40:31.627-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 170 249126579544280 0 2016-04-07T01:40:31.629-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10000 2016-04-07T01:40:31.629-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 171 29556219393742 0 2016-04-07T01:40:31.631-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 172 114885524964151 0 2016-04-07T01:40:31.631-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 173 199955737308991 0 2016-04-07T01:40:31.632-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 174 181608637308626 0 2016-04-07T01:40:31.633-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 175 221894738875534 0 2016-04-07T01:40:31.633-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 176 85686262153531 0 2016-04-07T01:40:31.635-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 177 51304531796330 0 2016-04-07T01:40:31.636-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 178 18424568533583 0 2016-04-07T01:40:31.638-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 179 196574124762770 0 2016-04-07T01:40:31.64-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 181 185635866979782 0 2016-04-07T01:40:31.64-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 180 53380458540410 0 2016-04-07T01:40:31.641-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 182 173687551181314 0 2016-04-07T01:40:31.642-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 183 175909605484048 0 2016-04-07T01:40:31.644-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 184 143974481262642 0 2016-04-07T01:40:31.644-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 185 35674611852785 0 2016-04-07T01:40:31.646-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 131 30690109642333 0 2016-04-07T01:40:31.646-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 186 230755959735246 0 2016-04-07T01:40:31.647-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 187 242209913714396 0 2016-04-07T01:40:31.658-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 188 19113392408368 0 2016-04-07T01:40:31.658-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 189 166063344925017 0 2016-04-07T01:40:31.66-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 190 199723255433805 0 2016-04-07T01:40:31.661-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 191 191778231151867 0 2016-04-07T01:40:31.665-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 128 102740372584504 0 2016-04-07T01:40:31.669-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 201 82618898704633 0 2016-04-07T01:40:31.674-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 204 14126644674527 0 2016-04-07T01:40:31.675-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 200 209550984826791 0 2016-04-07T01:40:31.675-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 203 228263044972741 0 2016-04-07T01:40:31.675-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 205 224519767765986 0 2016-04-07T01:40:31.677-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 206 91331376439154 0 2016-04-07T01:40:31.683-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 202 217300551820138 0 2016-04-07T01:40:31.683-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 207 207792745464493 0 2016-04-07T01:40:31.685-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 208 187484416779277 0 2016-04-07T01:40:31.685-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 196 277729123979898 0 2016-04-07T01:40:31.685-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 209 254394519729954 0 2016-04-07T01:40:31.686-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 210 114648098841082 0 2016-04-07T01:40:31.691-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 213 159279384568991 0 2016-04-07T01:40:31.692-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 199 10667602957216 0 2016-04-07T01:40:31.692-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 212 126418086190543 0 2016-04-07T01:40:31.692-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 194 18823703024258 0 2016-04-07T01:40:31.694-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 211 10487221525014 0 2016-04-07T01:40:31.694-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 198 136205498691660 0 2016-04-07T01:40:31.701-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 216 229184097041823 0 2016-04-07T01:40:31.703-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 214 223825748604223 0 2016-04-07T01:40:31.706-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 215 79327313224585 0 2016-04-07T01:40:31.706-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 217 67835701164032 0 2016-04-07T01:40:31.707-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 218 27447744316213 0 2016-04-07T01:40:31.709-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 220 87243615200284 0 2016-04-07T01:40:31.714-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 222 140648280935069 0 2016-04-07T01:40:31.715-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 221 47137360112032 0 2016-04-07T01:40:31.715-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 219 224008698130764 0 2016-04-07T01:40:31.716-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 197 38762857161424 0 2016-04-07T01:40:31.718-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 224 106399440579574 0 2016-04-07T01:40:31.718-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 226 68557284084239 0 2016-04-07T01:40:31.721-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 225 8933158878333 0 2016-04-07T01:40:31.723-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 223 198073521984403 0 2016-04-07T01:40:31.724-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:40:31.724-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 193 264401761696415 0 2016-04-07T01:40:31.726-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 230 145666042509725 0 2016-04-07T01:40:31.727-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 229 188962208980737 0 2016-04-07T01:40:31.727-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 228 19801676400475 0 2016-04-07T01:40:31.73-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 227 44697846800034 0 2016-04-07T01:40:31.731-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 195 129541932363670 0 2016-04-07T01:40:31.733-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 233 112288671191154 0 2016-04-07T01:40:31.733-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 234 53570529956415 0 2016-04-07T01:40:31.734-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 232 18158929490307 0 2016-04-07T01:40:31.737-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 237 250441559752442 0 2016-04-07T01:40:31.737-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 238 86826263935261 0 2016-04-07T01:40:31.737-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 231 31960891932027 0 2016-04-07T01:40:31.74-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 241 88999456751274 0 2016-04-07T01:40:31.74-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 236 138131018818053 0 2016-04-07T01:40:31.744-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 242 265430450356543 0 2016-04-07T01:40:31.745-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 245 73183123015510 0 2016-04-07T01:40:31.745-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 192 241017762904693 0 2016-04-07T01:40:31.745-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 235 267328176912814 0 2016-04-07T01:40:31.746-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 246 194724623962947 0 2016-04-07T01:40:31.746-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 240 46904048593668 0 2016-04-07T01:40:31.749-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 249 146663740405018 0 2016-04-07T01:40:31.75-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 239 148838186622967 0 2016-04-07T01:40:31.751-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 253 253735140761604 0 2016-04-07T01:40:31.753-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 244 146792239241485 0 2016-04-07T01:40:31.753-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 250 179357602713857 0 2016-04-07T01:40:31.756-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 243 91488861363349 0 2016-04-07T01:40:31.758-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 248 207473333836538 0 2016-04-07T01:40:31.758-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 254 180831235341508 0 2016-04-07T01:40:31.76-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 247 42658685522707 0 2016-04-07T01:40:31.762-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 252 41137970376249 0 2016-04-07T01:40:31.762-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 251 101281701852002 0 2016-04-07T01:40:31.765-07:00 [Info] KVSender::sendMutationTopicRequest Success Projector 172.23.99.241:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 bucket-1 InstanceIds [8343618505678776570] 2016-04-07T01:40:31.765-07:00 [Info] KVSender::sendMutationTopicRequest Projector 172.23.99.245:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 bucket-1 Instances [indexInstance: singlePartn: > ] 2016-04-07T01:40:31.765-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 255 202527572927593 0 2016-04-07T01:40:31.781-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20000 2016-04-07T01:40:31.833-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10000 QueuedCount 3194 2016-04-07T01:40:31.834-07:00 [Info] DATP[->dataport ":9105"] new connection "172.23.99.245:38839" +2 2016-04-07T01:40:31.926-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:40:31.929-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30000 2016-04-07T01:40:32.036-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20000 QueuedCount 7040 2016-04-07T01:40:32.063-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 40000 2016-04-07T01:40:32.125-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:40:32.137-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 263 27260014232881 0 2016-04-07T01:40:32.137-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 265 246698879096207 0 2016-04-07T01:40:32.137-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 256 1546979460845 0 2016-04-07T01:40:32.137-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 264 129712421199272 0 2016-04-07T01:40:32.137-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 267 255687772971589 0 2016-04-07T01:40:32.137-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 270 262223004224953 0 2016-04-07T01:40:32.137-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 257 79544077497609 0 2016-04-07T01:40:32.137-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 268 32504167363520 0 2016-04-07T01:40:32.137-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 269 256426897432058 0 2016-04-07T01:40:32.137-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 262 895212588425 0 2016-04-07T01:40:32.137-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 259 115288224073358 0 2016-04-07T01:40:32.137-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 261 101945102375832 0 2016-04-07T01:40:32.137-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 258 85888777513931 0 2016-04-07T01:40:32.137-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 266 242907494698803 0 2016-04-07T01:40:32.137-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 260 113482490555676 0 2016-04-07T01:40:32.148-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 271 137921726772083 0 2016-04-07T01:40:32.148-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 272 146746836785032 0 2016-04-07T01:40:32.148-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 273 251166499214887 0 2016-04-07T01:40:32.148-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 274 146886425165105 0 2016-04-07T01:40:32.148-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 275 132865426336626 0 2016-04-07T01:40:32.148-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 276 89802449018928 0 2016-04-07T01:40:32.156-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 277 193080417098359 0 2016-04-07T01:40:32.156-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 279 95096824914441 0 2016-04-07T01:40:32.156-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 278 225236960413894 0 2016-04-07T01:40:32.167-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 280 81033548925769 0 2016-04-07T01:40:32.167-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 281 75935468159215 0 2016-04-07T01:40:32.173-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 282 214038713749476 0 2016-04-07T01:40:32.173-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 283 231390890158982 0 2016-04-07T01:40:32.181-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 285 97650412355571 0 2016-04-07T01:40:32.181-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 284 60686243476087 0 2016-04-07T01:40:32.188-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 286 149753918542890 0 2016-04-07T01:40:32.194-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 287 57694955057013 0 2016-04-07T01:40:32.194-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 288 20702222911674 0 2016-04-07T01:40:32.2-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 290 53427376355426 0 2016-04-07T01:40:32.2-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 289 75130560603798 0 2016-04-07T01:40:32.207-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 291 184618860117814 0 2016-04-07T01:40:32.207-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 292 154508766441666 0 2016-04-07T01:40:32.214-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30000 QueuedCount 10276 2016-04-07T01:40:32.215-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 294 205408868172307 0 2016-04-07T01:40:32.215-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 293 176970005731905 0 2016-04-07T01:40:32.222-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 295 52281852171828 0 2016-04-07T01:40:32.222-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 296 24543132806843 0 2016-04-07T01:40:32.225-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 297 62420413849067 0 2016-04-07T01:40:32.232-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 298 18882763864430 0 2016-04-07T01:40:32.232-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 299 150396509978555 0 2016-04-07T01:40:32.238-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 300 65543761980579 0 2016-04-07T01:40:32.238-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 301 56537105668682 0 2016-04-07T01:40:32.241-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 50000 2016-04-07T01:40:32.245-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 302 27891731578804 0 2016-04-07T01:40:32.245-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 303 29536263214701 0 2016-04-07T01:40:32.25-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 304 217073413973886 0 2016-04-07T01:40:32.256-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 305 129527931657206 0 2016-04-07T01:40:32.256-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 306 173430505812647 0 2016-04-07T01:40:32.262-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 307 60466047295224 0 2016-04-07T01:40:32.268-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 308 261638212856335 0 2016-04-07T01:40:32.268-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 309 45359109698225 0 2016-04-07T01:40:32.274-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 310 259124350695629 0 2016-04-07T01:40:32.274-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 311 7900171780873 0 2016-04-07T01:40:32.275-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 312 22005764754056 0 2016-04-07T01:40:32.281-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 314 123645683673392 0 2016-04-07T01:40:32.281-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 313 252688174513801 0 2016-04-07T01:40:32.288-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 315 240391270191713 0 2016-04-07T01:40:32.288-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 316 62539806496338 0 2016-04-07T01:40:32.288-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 317 7951947259292 0 2016-04-07T01:40:32.296-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 318 191538047081265 0 2016-04-07T01:40:32.296-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 319 264090322605651 0 2016-04-07T01:40:32.304-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 322 133803358863585 0 2016-04-07T01:40:32.304-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 323 210639052706213 0 2016-04-07T01:40:32.308-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 324 233623802774667 0 2016-04-07T01:40:32.312-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 325 178565120121885 0 2016-04-07T01:40:32.317-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 326 8905701079672 0 2016-04-07T01:40:32.317-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 327 115095453881658 0 2016-04-07T01:40:32.321-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 328 281009866095427 0 2016-04-07T01:40:32.321-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 329 50481111119864 0 2016-04-07T01:40:32.324-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:40:32.325-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 330 111980293109036 0 2016-04-07T01:40:32.33-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 331 121577495881284 0 2016-04-07T01:40:32.334-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 332 84899097656645 0 2016-04-07T01:40:32.339-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 333 115748511094308 0 2016-04-07T01:40:32.344-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 334 65024212746988 0 2016-04-07T01:40:32.348-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 335 218293362852103 0 2016-04-07T01:40:32.348-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 336 76390627012682 0 2016-04-07T01:40:32.35-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 337 161079301313387 0 2016-04-07T01:40:32.354-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 338 277841677476401 0 2016-04-07T01:40:32.357-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 339 258450357900235 0 2016-04-07T01:40:32.361-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 340 147416718275857 0 2016-04-07T01:40:32.361-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 341 213891240516525 0 2016-04-07T01:40:32.366-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 60000 2016-04-07T01:40:32.366-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 342 110940142156417 0 2016-04-07T01:40:32.369-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 320 144151456454454 0 2016-04-07T01:40:32.369-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 343 141078507077786 0 2016-04-07T01:40:32.373-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 321 107464450229706 0 2016-04-07T01:40:32.374-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 344 149475192345485 0 2016-04-07T01:40:32.375-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 345 131490923188363 0 2016-04-07T01:40:32.379-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 346 56499205482554 0 2016-04-07T01:40:32.382-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 347 223553175645689 0 2016-04-07T01:40:32.382-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 348 198787959814732 0 2016-04-07T01:40:32.385-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 349 76943470568236 0 2016-04-07T01:40:32.385-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 350 137832517455205 0 2016-04-07T01:40:32.387-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 351 236423867035575 0 2016-04-07T01:40:32.39-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 353 108165105037645 0 2016-04-07T01:40:32.39-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 352 93084725756287 0 2016-04-07T01:40:32.393-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 354 3867876969878 0 2016-04-07T01:40:32.395-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 355 280188491278034 0 2016-04-07T01:40:32.395-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 356 118582889367468 0 2016-04-07T01:40:32.398-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 357 188681393452350 0 2016-04-07T01:40:32.4-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 358 54586884726256 0 2016-04-07T01:40:32.4-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 359 211955265825501 0 2016-04-07T01:40:32.402-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 360 71634602888331 0 2016-04-07T01:40:32.402-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 361 179634547856301 0 2016-04-07T01:40:32.404-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 40000 QueuedCount 12336 2016-04-07T01:40:32.405-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 362 132935752981601 0 2016-04-07T01:40:32.405-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 363 116725244351635 0 2016-04-07T01:40:32.408-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 364 58251234397268 0 2016-04-07T01:40:32.408-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 365 64854067793169 0 2016-04-07T01:40:32.41-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 366 113813521847758 0 2016-04-07T01:40:32.41-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 367 30427538278801 0 2016-04-07T01:40:32.412-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 368 47945369333894 0 2016-04-07T01:40:32.412-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 369 261395256922666 0 2016-04-07T01:40:32.415-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 370 64670526806284 0 2016-04-07T01:40:32.417-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 373 159205846665341 0 2016-04-07T01:40:32.417-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 371 143877370104091 0 2016-04-07T01:40:32.417-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 372 192587362235355 0 2016-04-07T01:40:32.419-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 374 279531716457049 0 2016-04-07T01:40:32.419-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 375 131303116671924 0 2016-04-07T01:40:32.421-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 376 105679161862086 0 2016-04-07T01:40:32.423-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 377 57287959545345 0 2016-04-07T01:40:32.423-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 378 179411018314662 0 2016-04-07T01:40:32.432-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 379 225293860228645 0 2016-04-07T01:40:32.434-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 380 22485719731555 0 2016-04-07T01:40:32.434-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 381 250100011904387 0 2016-04-07T01:40:32.438-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 383 96787643646442 0 2016-04-07T01:40:32.438-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 382 159924366852801 0 2016-04-07T01:40:32.447-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 391 34628172633403 0 2016-04-07T01:40:32.448-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 386 11570794554064 0 2016-04-07T01:40:32.448-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 392 248514792083613 0 2016-04-07T01:40:32.448-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 393 203344104828641 0 2016-04-07T01:40:32.449-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 394 201676917574221 0 2016-04-07T01:40:32.451-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 390 107764607740190 0 2016-04-07T01:40:32.451-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 395 12607236770973 0 2016-04-07T01:40:32.453-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 396 234507562818907 0 2016-04-07T01:40:32.453-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 388 260859723691414 0 2016-04-07T01:40:32.453-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 397 111201551062485 0 2016-04-07T01:40:32.454-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 398 116590054896309 0 2016-04-07T01:40:32.456-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 399 67639299521934 0 2016-04-07T01:40:32.458-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 400 160993832250430 0 2016-04-07T01:40:32.458-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 401 245216097505569 0 2016-04-07T01:40:32.458-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 70000 2016-04-07T01:40:32.46-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 402 76567398954440 0 2016-04-07T01:40:32.461-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 403 17035286284947 0 2016-04-07T01:40:32.465-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 404 248537555008716 0 2016-04-07T01:40:32.465-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 389 9838367018341 0 2016-04-07T01:40:32.465-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 387 37239634745072 0 2016-04-07T01:40:32.467-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 405 374275972059 0 2016-04-07T01:40:32.469-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 406 225004082599477 0 2016-04-07T01:40:32.471-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 407 250498057665774 0 2016-04-07T01:40:32.473-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 408 182894201694341 0 2016-04-07T01:40:32.473-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 409 31029584782986 0 2016-04-07T01:40:32.475-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 410 139649877376138 0 2016-04-07T01:40:32.477-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 411 3518353137033 0 2016-04-07T01:40:32.479-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 412 254765561230686 0 2016-04-07T01:40:32.481-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 413 80220849914541 0 2016-04-07T01:40:32.482-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 414 138339190094840 0 2016-04-07T01:40:32.484-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 415 139394071499469 0 2016-04-07T01:40:32.486-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 416 107761158731494 0 2016-04-07T01:40:32.488-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 417 114702136248440 0 2016-04-07T01:40:32.49-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 418 96411606641571 0 2016-04-07T01:40:32.49-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 419 2655290973521 0 2016-04-07T01:40:32.492-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 420 215048844823944 0 2016-04-07T01:40:32.494-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 421 80219706067300 0 2016-04-07T01:40:32.496-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 422 236277540984918 0 2016-04-07T01:40:32.498-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 423 153171087128800 0 2016-04-07T01:40:32.5-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 424 209390039044360 0 2016-04-07T01:40:32.502-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 425 140252528342781 0 2016-04-07T01:40:32.503-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 426 105261294565838 0 2016-04-07T01:40:32.505-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 427 62626992333014 0 2016-04-07T01:40:32.505-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 428 217328489453964 0 2016-04-07T01:40:32.507-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 429 34948802404363 0 2016-04-07T01:40:32.508-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 430 214448270703440 0 2016-04-07T01:40:32.511-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 431 236337978062457 0 2016-04-07T01:40:32.512-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 432 180439943079880 0 2016-04-07T01:40:32.512-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 385 260565233876719 0 2016-04-07T01:40:32.514-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 433 166543065468965 0 2016-04-07T01:40:32.516-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 434 111551990533393 0 2016-04-07T01:40:32.519-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 435 165879388023084 0 2016-04-07T01:40:32.519-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 384 211278235814426 0 2016-04-07T01:40:32.521-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 436 224269957627790 0 2016-04-07T01:40:32.523-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 437 121980222524199 0 2016-04-07T01:40:32.525-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 438 26472436990691 0 2016-04-07T01:40:32.525-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 439 263090038207007 0 2016-04-07T01:40:32.526-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 440 238669376592694 0 2016-04-07T01:40:32.528-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 441 124504757089489 0 2016-04-07T01:40:32.528-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:40:32.53-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 443 219874331436897 0 2016-04-07T01:40:32.53-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 442 82096644648745 0 2016-04-07T01:40:32.531-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 444 175547076513295 0 2016-04-07T01:40:32.533-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 445 258202566303063 0 2016-04-07T01:40:32.535-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 446 212357857345047 0 2016-04-07T01:40:32.535-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 447 169612914546903 0 2016-04-07T01:40:32.54-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 450 52301573355900 0 2016-04-07T01:40:32.544-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 80000 2016-04-07T01:40:32.545-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 454 97253190393479 0 2016-04-07T01:40:32.547-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 456 3820380395648 0 2016-04-07T01:40:32.547-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 452 240951505692632 0 2016-04-07T01:40:32.549-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 455 188046195044087 0 2016-04-07T01:40:32.549-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 457 14379661487527 0 2016-04-07T01:40:32.551-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 458 35826637587626 0 2016-04-07T01:40:32.553-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 459 57537140445320 0 2016-04-07T01:40:32.555-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 460 53137009095894 0 2016-04-07T01:40:32.555-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 461 86693483966322 0 2016-04-07T01:40:32.556-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 462 112043093562810 0 2016-04-07T01:40:32.558-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 463 267022638433516 0 2016-04-07T01:40:32.56-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 464 252177993290728 0 2016-04-07T01:40:32.561-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 465 260364507682 0 2016-04-07T01:40:32.561-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 466 103596289392858 0 2016-04-07T01:40:32.561-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 453 17499729916943 0 2016-04-07T01:40:32.563-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 467 202628578448919 0 2016-04-07T01:40:32.564-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 468 131855587161488 0 2016-04-07T01:40:32.564-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 469 228642619865691 0 2016-04-07T01:40:32.566-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 470 55411065384231 0 2016-04-07T01:40:32.567-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 471 253894947411124 0 2016-04-07T01:40:32.569-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 472 122820154740668 0 2016-04-07T01:40:32.571-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 473 255757676493145 0 2016-04-07T01:40:32.571-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 474 182935058106234 0 2016-04-07T01:40:32.572-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 475 21770990846595 0 2016-04-07T01:40:32.573-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 477 156846789942576 0 2016-04-07T01:40:32.573-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 476 69081877792878 0 2016-04-07T01:40:32.577-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 479 169597542087816 0 2016-04-07T01:40:32.577-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 478 228785149186050 0 2016-04-07T01:40:32.578-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 482 271000096409149 0 2016-04-07T01:40:32.581-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 481 25128162586403 0 2016-04-07T01:40:32.582-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 480 32335002875345 0 2016-04-07T01:40:32.582-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 483 204674205649011 0 2016-04-07T01:40:32.584-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 451 73626665407999 0 2016-04-07T01:40:32.584-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 486 63287324269707 0 2016-04-07T01:40:32.587-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 485 188731967166384 0 2016-04-07T01:40:32.587-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 487 43834058726155 0 2016-04-07T01:40:32.589-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 484 278995094861168 0 2016-04-07T01:40:32.589-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 490 75550753124997 0 2016-04-07T01:40:32.591-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 448 229998737342737 0 2016-04-07T01:40:32.591-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 489 281349442854681 0 2016-04-07T01:40:32.593-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 491 66187222960312 0 2016-04-07T01:40:32.594-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 488 108485932818222 0 2016-04-07T01:40:32.594-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 494 178563709596228 0 2016-04-07T01:40:32.596-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 493 12561644398361 0 2016-04-07T01:40:32.598-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 498 226334980604911 0 2016-04-07T01:40:32.598-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 495 78568031446701 0 2016-04-07T01:40:32.6-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 497 280084319865020 0 2016-04-07T01:40:32.602-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 492 65920473004241 0 2016-04-07T01:40:32.603-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 499 151555235989109 0 2016-04-07T01:40:32.61-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 90000 2016-04-07T01:40:32.613-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 502 45947649780431 0 2016-04-07T01:40:32.613-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 496 172418809954194 0 2016-04-07T01:40:32.613-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 501 258518236170735 0 2016-04-07T01:40:32.616-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 506 224673953503960 0 2016-04-07T01:40:32.616-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 503 205558840653118 0 2016-04-07T01:40:32.616-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 50000 QueuedCount 23538 2016-04-07T01:40:32.619-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 505 55475257544811 0 2016-04-07T01:40:32.62-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 500 26600488475182 0 2016-04-07T01:40:32.62-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 510 226625287765610 0 2016-04-07T01:40:32.623-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 449 266187086754814 0 2016-04-07T01:40:32.623-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 509 172790174641434 0 2016-04-07T01:40:32.627-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 507 194984705667080 0 2016-04-07T01:40:32.63-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 504 38711381267561 0 2016-04-07T01:40:32.631-07:00 [Info] KVSender::sendMutationTopicRequest Success Projector 172.23.99.245:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 bucket-1 InstanceIds [8343618505678776570] 2016-04-07T01:40:32.631-07:00 [Info] KVSender::sendMutationTopicRequest Projector 172.23.99.246:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 bucket-1 Instances [indexInstance: singlePartn: > ] 2016-04-07T01:40:32.631-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 511 210748206588174 0 2016-04-07T01:40:32.634-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 508 140218838986220 0 2016-04-07T01:40:32.64-07:00 [Info] DATP[->dataport ":9105"] new connection "172.23.99.246:54142" +3 2016-04-07T01:40:32.678-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 100000 2016-04-07T01:40:32.731-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:40:32.751-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 110000 2016-04-07T01:40:32.827-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 60000 QueuedCount 41252 2016-04-07T01:40:32.832-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 120000 2016-04-07T01:40:32.898-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 130000 2016-04-07T01:40:32.924-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 515 141115925012637 0 2016-04-07T01:40:32.924-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 516 90758405621894 0 2016-04-07T01:40:32.924-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 517 240506127610227 0 2016-04-07T01:40:32.924-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 518 7693416181893 0 2016-04-07T01:40:32.924-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 519 138438729402825 0 2016-04-07T01:40:32.924-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 512 121182627168231 0 2016-04-07T01:40:32.924-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 513 171625535958830 0 2016-04-07T01:40:32.924-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 514 2417747867436 0 2016-04-07T01:40:32.929-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 522 59753070980287 0 2016-04-07T01:40:32.929-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 523 80400504718970 0 2016-04-07T01:40:32.929-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 520 154570785912287 0 2016-04-07T01:40:32.929-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 521 53770474621051 0 2016-04-07T01:40:32.933-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 524 229745376473076 0 2016-04-07T01:40:32.934-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 525 49004432144297 0 2016-04-07T01:40:32.934-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 526 214403966626793 0 2016-04-07T01:40:32.934-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 527 31617201205299 0 2016-04-07T01:40:32.939-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 528 231847482667629 0 2016-04-07T01:40:32.939-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 529 260484666472298 0 2016-04-07T01:40:32.939-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 530 250794711912013 0 2016-04-07T01:40:32.939-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 531 74523311916711 0 2016-04-07T01:40:32.939-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 532 206524842755330 0 2016-04-07T01:40:32.944-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 533 72875319483470 0 2016-04-07T01:40:32.944-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 536 40030380340463 0 2016-04-07T01:40:32.944-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 534 33717473206474 0 2016-04-07T01:40:32.944-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 535 112544349734855 0 2016-04-07T01:40:32.95-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 538 269226584366352 0 2016-04-07T01:40:32.95-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 539 157191786669833 0 2016-04-07T01:40:32.95-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 540 124122355073905 0 2016-04-07T01:40:32.95-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 537 211674968619301 0 2016-04-07T01:40:32.956-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 541 19677190933529 0 2016-04-07T01:40:32.956-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 542 78472735462906 0 2016-04-07T01:40:32.958-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 543 157414300767058 0 2016-04-07T01:40:32.963-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 544 26487224465524 0 2016-04-07T01:40:32.963-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 545 107997282218421 0 2016-04-07T01:40:32.964-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 140000 2016-04-07T01:40:32.968-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 546 232213232807794 0 2016-04-07T01:40:32.968-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 547 160989522243575 0 2016-04-07T01:40:32.973-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 548 200308776339811 0 2016-04-07T01:40:32.973-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 549 38586788803523 0 2016-04-07T01:40:32.973-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 550 271349735932303 0 2016-04-07T01:40:32.979-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 551 140098529754914 0 2016-04-07T01:40:32.979-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 552 218904289228598 0 2016-04-07T01:40:32.984-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 555 152627239777209 0 2016-04-07T01:40:32.984-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 553 277972108106165 0 2016-04-07T01:40:32.984-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 554 174970542016099 0 2016-04-07T01:40:32.988-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 556 63471972823774 0 2016-04-07T01:40:32.989-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 557 89395364103575 0 2016-04-07T01:40:32.989-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 558 168075938356822 0 2016-04-07T01:40:32.994-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 559 81283187259173 0 2016-04-07T01:40:32.994-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 560 7186543346804 0 2016-04-07T01:40:33.001-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 561 89554778090352 0 2016-04-07T01:40:33.001-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 562 237998568306492 0 2016-04-07T01:40:33.007-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 563 243538275545468 0 2016-04-07T01:40:33.007-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 564 19085982620903 0 2016-04-07T01:40:33.008-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 565 226748367877526 0 2016-04-07T01:40:33.012-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 566 29746640343830 0 2016-04-07T01:40:33.012-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 567 58884002826157 0 2016-04-07T01:40:33.016-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 568 73498504245710 0 2016-04-07T01:40:33.016-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 569 153332375190596 0 2016-04-07T01:40:33.019-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 570 232506388044413 0 2016-04-07T01:40:33.019-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 571 193462090158717 0 2016-04-07T01:40:33.022-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 572 171061328440539 0 2016-04-07T01:40:33.022-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 573 256287400090435 0 2016-04-07T01:40:33.025-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 574 4150222480714 0 2016-04-07T01:40:33.025-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 575 55569083272847 0 2016-04-07T01:40:33.028-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 150000 2016-04-07T01:40:33.033-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 578 4048496252386 0 2016-04-07T01:40:33.037-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 580 246158588696535 0 2016-04-07T01:40:33.038-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 579 231134217328070 0 2016-04-07T01:40:33.045-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 581 2234919860034 0 2016-04-07T01:40:33.045-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 582 187706073648641 0 2016-04-07T01:40:33.045-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 583 37201460083659 0 2016-04-07T01:40:33.052-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 70000 QueuedCount 50000 2016-04-07T01:40:33.058-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 584 218398890214187 0 2016-04-07T01:40:33.078-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 585 210741245741635 0 2016-04-07T01:40:33.08-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 160000 2016-04-07T01:40:33.084-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 586 203874114691866 0 2016-04-07T01:40:33.088-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 587 178063703342990 0 2016-04-07T01:40:33.091-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 588 62085308641635 0 2016-04-07T01:40:33.095-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 589 80533561546119 0 2016-04-07T01:40:33.099-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 590 1038551550053 0 2016-04-07T01:40:33.099-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 591 7253997924037 0 2016-04-07T01:40:33.102-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 576 243985402515740 0 2016-04-07T01:40:33.102-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 592 200695890500436 0 2016-04-07T01:40:33.106-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 593 226987473102021 0 2016-04-07T01:40:33.106-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 577 136675310130922 0 2016-04-07T01:40:33.109-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 595 249796873110263 0 2016-04-07T01:40:33.109-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 594 65954391558993 0 2016-04-07T01:40:33.113-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 596 135577706194787 0 2016-04-07T01:40:33.117-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 597 94344959119248 0 2016-04-07T01:40:33.117-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 598 134887353613640 0 2016-04-07T01:40:33.12-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 599 257761213065819 0 2016-04-07T01:40:33.124-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 600 244949116164538 0 2016-04-07T01:40:33.128-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 601 94174951702240 0 2016-04-07T01:40:33.128-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 602 7271558041305 0 2016-04-07T01:40:33.131-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 603 210321241078083 0 2016-04-07T01:40:33.134-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 604 69139912599804 0 2016-04-07T01:40:33.137-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 170000 2016-04-07T01:40:33.137-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 605 96568825453020 0 2016-04-07T01:40:33.141-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 606 253093415000216 0 2016-04-07T01:40:33.141-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 607 199001204584077 0 2016-04-07T01:40:33.147-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 608 225660949002060 0 2016-04-07T01:40:33.15-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 609 269643305555178 0 2016-04-07T01:40:33.153-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 611 203972799826097 0 2016-04-07T01:40:33.153-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 610 179143467787197 0 2016-04-07T01:40:33.156-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 612 123781682680502 0 2016-04-07T01:40:33.159-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 613 144403713888023 0 2016-04-07T01:40:33.16-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 614 38982631264451 0 2016-04-07T01:40:33.164-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 615 126964824615054 0 2016-04-07T01:40:33.167-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 616 159792905164818 0 2016-04-07T01:40:33.17-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 617 11230407112024 0 2016-04-07T01:40:33.174-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 618 239995581471961 0 2016-04-07T01:40:33.176-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 619 6437331176175 0 2016-04-07T01:40:33.176-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 620 239276861039369 0 2016-04-07T01:40:33.178-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 621 225150866062221 0 2016-04-07T01:40:33.181-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 622 199923360693944 0 2016-04-07T01:40:33.183-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 180000 2016-04-07T01:40:33.184-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 623 125483282256132 0 2016-04-07T01:40:33.187-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 624 65906415392530 0 2016-04-07T01:40:33.188-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 625 186797886188568 0 2016-04-07T01:40:33.188-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 626 96107373924028 0 2016-04-07T01:40:33.191-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 627 174110897621360 0 2016-04-07T01:40:33.193-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 628 94491709393761 0 2016-04-07T01:40:33.195-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 629 163033954045746 0 2016-04-07T01:40:33.195-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 630 232562582400634 0 2016-04-07T01:40:33.197-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 631 173811333357236 0 2016-04-07T01:40:33.197-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 632 21274276982049 0 2016-04-07T01:40:33.199-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 633 114843981245936 0 2016-04-07T01:40:33.201-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 634 65491767613405 0 2016-04-07T01:40:33.204-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 635 181299317990589 0 2016-04-07T01:40:33.206-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 636 144238425250200 0 2016-04-07T01:40:33.206-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 637 55454390737325 0 2016-04-07T01:40:33.211-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 638 40945871617517 0 2016-04-07T01:40:33.212-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 639 259281874819523 0 2016-04-07T01:40:33.215-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 642 119415251647932 0 2016-04-07T01:40:33.221-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 640 251936841014868 0 2016-04-07T01:40:33.221-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 646 148533452600260 0 2016-04-07T01:40:33.222-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 644 10706035109351 0 2016-04-07T01:40:33.222-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 648 33856805171736 0 2016-04-07T01:40:33.222-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 645 95658183018036 0 2016-04-07T01:40:33.224-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:40:33.225-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 651 103306647967420 0 2016-04-07T01:40:33.225-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 649 15787209778767 0 2016-04-07T01:40:33.225-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 652 54515296940328 0 2016-04-07T01:40:33.225-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 650 166529299148079 0 2016-04-07T01:40:33.226-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 190000 2016-04-07T01:40:33.227-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 654 194407104021084 0 2016-04-07T01:40:33.228-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 653 238970329742878 0 2016-04-07T01:40:33.228-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 647 187634409069380 0 2016-04-07T01:40:33.228-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 655 199465436418566 0 2016-04-07T01:40:33.228-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 656 45847015370197 0 2016-04-07T01:40:33.23-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 657 40675375786322 0 2016-04-07T01:40:33.232-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 658 190973403981823 0 2016-04-07T01:40:33.232-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 659 199462859760951 0 2016-04-07T01:40:33.237-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 660 87579210264307 0 2016-04-07T01:40:33.237-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 661 192471345456176 0 2016-04-07T01:40:33.241-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 662 199929294762064 0 2016-04-07T01:40:33.241-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 663 117806187388589 0 2016-04-07T01:40:33.248-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 643 248472582573469 0 2016-04-07T01:40:33.252-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 665 148274589114 0 2016-04-07T01:40:33.252-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 664 175279108970734 0 2016-04-07T01:40:33.252-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 666 275504016529819 0 2016-04-07T01:40:33.253-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 667 185267074649567 0 2016-04-07T01:40:33.254-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 668 235763381711287 0 2016-04-07T01:40:33.255-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 669 37845966580356 0 2016-04-07T01:40:33.256-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 670 241850425171506 0 2016-04-07T01:40:33.257-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 671 11076272302553 0 2016-04-07T01:40:33.258-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 672 143369547089555 0 2016-04-07T01:40:33.258-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 673 173745778139719 0 2016-04-07T01:40:33.26-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 675 103243387674262 0 2016-04-07T01:40:33.26-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 674 150065116242090 0 2016-04-07T01:40:33.263-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 676 237206038916630 0 2016-04-07T01:40:33.263-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 677 155271829866346 0 2016-04-07T01:40:33.267-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 80000 QueuedCount 50000 2016-04-07T01:40:33.268-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 678 127683916822270 0 2016-04-07T01:40:33.27-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 679 106647841526140 0 2016-04-07T01:40:33.271-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 680 122510453317696 0 2016-04-07T01:40:33.275-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 682 87089646620180 0 2016-04-07T01:40:33.275-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 681 10343420084800 0 2016-04-07T01:40:33.278-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 683 82186644251455 0 2016-04-07T01:40:33.279-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 200000 2016-04-07T01:40:33.283-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 684 241114200205308 0 2016-04-07T01:40:33.285-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 685 98258800457502 0 2016-04-07T01:40:33.286-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 686 148734543954869 0 2016-04-07T01:40:33.287-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 688 261967423039901 0 2016-04-07T01:40:33.287-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 687 134804313858666 0 2016-04-07T01:40:33.29-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 689 206750443299925 0 2016-04-07T01:40:33.292-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 690 177085962241439 0 2016-04-07T01:40:33.293-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 691 88501580709 0 2016-04-07T01:40:33.294-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 692 73888721951788 0 2016-04-07T01:40:33.295-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 693 79540881339667 0 2016-04-07T01:40:33.295-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 694 160193805742819 0 2016-04-07T01:40:33.296-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 695 253289163590388 0 2016-04-07T01:40:33.297-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 696 179874413826370 0 2016-04-07T01:40:33.298-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 698 38143970035414 0 2016-04-07T01:40:33.298-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 697 65785302909542 0 2016-04-07T01:40:33.299-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 699 246700486550243 0 2016-04-07T01:40:33.302-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 700 68833643641607 0 2016-04-07T01:40:33.303-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 701 37629904433202 0 2016-04-07T01:40:33.304-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 702 113539816738316 0 2016-04-07T01:40:33.311-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 641 165019081060876 0 2016-04-07T01:40:33.311-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 703 144524851334552 0 2016-04-07T01:40:33.316-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 706 236544566023273 0 2016-04-07T01:40:33.324-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 210000 2016-04-07T01:40:33.324-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 712 49698593952272 0 2016-04-07T01:40:33.326-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 710 221063529258034 0 2016-04-07T01:40:33.328-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 713 166141910319743 0 2016-04-07T01:40:33.33-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 714 52127078739365 0 2016-04-07T01:40:33.331-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 709 9863256903365 0 2016-04-07T01:40:33.333-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 715 274799394068621 0 2016-04-07T01:40:33.333-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 716 227939418672568 0 2016-04-07T01:40:33.333-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 707 194913087247124 0 2016-04-07T01:40:33.335-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 717 193423114030088 0 2016-04-07T01:40:33.337-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 711 180591749716087 0 2016-04-07T01:40:33.337-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 718 195883030944052 0 2016-04-07T01:40:33.339-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 719 254649393257417 0 2016-04-07T01:40:33.342-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 720 196350851010847 0 2016-04-07T01:40:33.342-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 721 136407670737912 0 2016-04-07T01:40:33.344-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 722 177810022134496 0 2016-04-07T01:40:33.346-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 723 13938188647258 0 2016-04-07T01:40:33.349-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 724 189132994515139 0 2016-04-07T01:40:33.349-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 704 219728197607365 0 2016-04-07T01:40:33.351-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 725 150738337981131 0 2016-04-07T01:40:33.353-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 726 85036339319173 0 2016-04-07T01:40:33.354-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 727 4565179148266 0 2016-04-07T01:40:33.354-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 728 93169746553477 0 2016-04-07T01:40:33.357-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 730 229206918190183 0 2016-04-07T01:40:33.36-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 729 230523707487381 0 2016-04-07T01:40:33.361-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 731 223861804382111 0 2016-04-07T01:40:33.363-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 708 196670764400477 0 2016-04-07T01:40:33.363-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 732 268553876943507 0 2016-04-07T01:40:33.368-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 733 139899607624654 0 2016-04-07T01:40:33.37-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 734 96147441510207 0 2016-04-07T01:40:33.372-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 735 128873993153374 0 2016-04-07T01:40:33.373-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 220000 2016-04-07T01:40:33.376-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 737 270045396322591 0 2016-04-07T01:40:33.376-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 736 78293995114228 0 2016-04-07T01:40:33.378-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 738 202923974948122 0 2016-04-07T01:40:33.38-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 739 125259162890706 0 2016-04-07T01:40:33.381-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 740 269470378244496 0 2016-04-07T01:40:33.383-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 741 276184179198262 0 2016-04-07T01:40:33.385-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 742 140432756945101 0 2016-04-07T01:40:33.409-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 743 56998013782375 0 2016-04-07T01:40:33.409-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 744 133766351217225 0 2016-04-07T01:40:33.411-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 745 188913981844397 0 2016-04-07T01:40:33.411-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 746 131499556058067 0 2016-04-07T01:40:33.412-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 705 192749176327537 0 2016-04-07T01:40:33.412-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 747 124245424955869 0 2016-04-07T01:40:33.412-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 748 217451534236849 0 2016-04-07T01:40:33.413-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 749 153141460310009 0 2016-04-07T01:40:33.413-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 750 53278236976843 0 2016-04-07T01:40:33.415-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 751 3391527528329 0 2016-04-07T01:40:33.418-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 230000 2016-04-07T01:40:33.421-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 754 101917232659935 0 2016-04-07T01:40:33.421-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 752 260614483832796 0 2016-04-07T01:40:33.422-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 753 141445068933992 0 2016-04-07T01:40:33.423-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 755 68174380939602 0 2016-04-07T01:40:33.426-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 756 232979345728759 0 2016-04-07T01:40:33.427-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 757 15352552746161 0 2016-04-07T01:40:33.429-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 758 92649922133892 0 2016-04-07T01:40:33.43-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 759 30055473586619 0 2016-04-07T01:40:33.43-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 760 87755165973455 0 2016-04-07T01:40:33.431-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 761 79579702200559 0 2016-04-07T01:40:33.437-07:00 [Info] KVSender::sendMutationTopicRequest Success Projector 172.23.99.246:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 bucket-1 InstanceIds [8343618505678776570] 2016-04-07T01:40:33.438-07:00 [Info] KVSender::sendMutationTopicRequest Projector 172.23.99.251:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 bucket-1 Instances [indexInstance: singlePartn: > ] 2016-04-07T01:40:33.439-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 762 259384051621340 0 2016-04-07T01:40:33.44-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 763 90615089879065 0 2016-04-07T01:40:33.441-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 764 250191008743136 0 2016-04-07T01:40:33.442-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 765 166416435741915 0 2016-04-07T01:40:33.444-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 766 6697396518271 0 2016-04-07T01:40:33.445-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 767 235394880216311 0 2016-04-07T01:40:33.448-07:00 [Info] DATP[->dataport ":9105"] new connection "172.23.99.251:38946" +4 2016-04-07T01:40:33.455-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 240000 2016-04-07T01:40:33.49-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 250000 2016-04-07T01:40:33.494-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 90000 QueuedCount 50000 2016-04-07T01:40:33.526-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 260000 2016-04-07T01:40:33.563-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 270000 2016-04-07T01:40:33.597-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 280000 2016-04-07T01:40:33.629-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 290000 2016-04-07T01:40:33.665-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 300000 2016-04-07T01:40:33.704-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 310000 2016-04-07T01:40:33.719-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 100000 QueuedCount 50000 2016-04-07T01:40:33.733-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 771 167823797448314 0 2016-04-07T01:40:33.733-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 769 119684723420151 0 2016-04-07T01:40:33.733-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 778 3296577238586 0 2016-04-07T01:40:33.733-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 768 167417654149734 0 2016-04-07T01:40:33.733-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 780 244656852814609 0 2016-04-07T01:40:33.733-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 781 250664656787493 0 2016-04-07T01:40:33.733-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 774 51247365818981 0 2016-04-07T01:40:33.733-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 770 15191440025051 0 2016-04-07T01:40:33.733-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 775 162464320565986 0 2016-04-07T01:40:33.733-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 779 61899254463015 0 2016-04-07T01:40:33.733-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 773 213094686189310 0 2016-04-07T01:40:33.733-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 772 238562371560618 0 2016-04-07T01:40:33.733-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 776 245171589508226 0 2016-04-07T01:40:33.733-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 777 272787077699191 0 2016-04-07T01:40:33.737-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 320000 2016-04-07T01:40:33.738-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 784 64241653497482 0 2016-04-07T01:40:33.738-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 785 183623795060367 0 2016-04-07T01:40:33.738-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 786 228927505262487 0 2016-04-07T01:40:33.738-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 787 151712931639347 0 2016-04-07T01:40:33.738-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 782 203419575454925 0 2016-04-07T01:40:33.738-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 783 196140345174845 0 2016-04-07T01:40:33.74-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 788 205273292708288 0 2016-04-07T01:40:33.746-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 789 23893093514858 0 2016-04-07T01:40:33.746-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 790 188068678920666 0 2016-04-07T01:40:33.746-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 791 188888847251819 0 2016-04-07T01:40:33.752-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 792 228627617608674 0 2016-04-07T01:40:33.759-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 793 72315079110491 0 2016-04-07T01:40:33.759-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 794 110052949541171 0 2016-04-07T01:40:33.765-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 795 249381772749464 0 2016-04-07T01:40:33.771-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 796 34278105207717 0 2016-04-07T01:40:33.772-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 330000 2016-04-07T01:40:33.777-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 797 85441703750098 0 2016-04-07T01:40:33.777-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 798 122874417009323 0 2016-04-07T01:40:33.783-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 799 58056103441527 0 2016-04-07T01:40:33.783-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 800 248357503161931 0 2016-04-07T01:40:33.789-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 801 36975030588681 0 2016-04-07T01:40:33.795-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 802 187167311958634 0 2016-04-07T01:40:33.795-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 803 223301528539750 0 2016-04-07T01:40:33.8-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 804 670994719036 0 2016-04-07T01:40:33.809-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 340000 2016-04-07T01:40:33.81-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 805 165819412165510 0 2016-04-07T01:40:33.815-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 806 2687353547616 0 2016-04-07T01:40:33.82-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 807 88150143575577 0 2016-04-07T01:40:33.82-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 808 87571342340667 0 2016-04-07T01:40:33.825-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 809 171222250480486 0 2016-04-07T01:40:33.831-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 810 39789329166822 0 2016-04-07T01:40:33.835-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 811 217237483189394 0 2016-04-07T01:40:33.84-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 812 131838331220227 0 2016-04-07T01:40:33.843-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 350000 2016-04-07T01:40:33.845-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 813 104293311727190 0 2016-04-07T01:40:33.85-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 814 110980075021296 0 2016-04-07T01:40:33.85-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 815 49597751941324 0 2016-04-07T01:40:33.855-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 816 120602021182050 0 2016-04-07T01:40:33.865-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 817 180248915543866 0 2016-04-07T01:40:33.872-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 819 105670988778675 0 2016-04-07T01:40:33.872-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 818 66176456419902 0 2016-04-07T01:40:33.877-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 820 202356965828438 0 2016-04-07T01:40:33.877-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 360000 2016-04-07T01:40:33.883-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 821 219363620991226 0 2016-04-07T01:40:33.888-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 822 89960844043157 0 2016-04-07T01:40:33.89-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 824 148658551504962 0 2016-04-07T01:40:33.89-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 823 172744867439657 0 2016-04-07T01:40:33.894-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 825 200628345437292 0 2016-04-07T01:40:33.894-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 826 148098586238302 0 2016-04-07T01:40:33.894-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 827 274896412369839 0 2016-04-07T01:40:33.898-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 828 177536559028766 0 2016-04-07T01:40:33.898-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 829 185761484060507 0 2016-04-07T01:40:33.901-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 830 221494619985881 0 2016-04-07T01:40:33.901-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 831 276230473126324 0 2016-04-07T01:40:33.909-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 835 147944333662853 0 2016-04-07T01:40:33.909-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 836 234289403388377 0 2016-04-07T01:40:33.912-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 370000 2016-04-07T01:40:33.915-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 839 96746711472548 0 2016-04-07T01:40:33.919-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 837 201160532552525 0 2016-04-07T01:40:33.919-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 838 15002169598331 0 2016-04-07T01:40:33.922-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 840 217108797649270 0 2016-04-07T01:40:33.928-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 841 236592863032118 0 2016-04-07T01:40:33.933-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 842 32401449636770 0 2016-04-07T01:40:33.936-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 843 164304736960988 0 2016-04-07T01:40:33.939-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 844 190392693114055 0 2016-04-07T01:40:33.94-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 845 131015819980566 0 2016-04-07T01:40:33.94-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 846 182117109023835 0 2016-04-07T01:40:33.942-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 380000 2016-04-07T01:40:33.943-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 834 103211447372532 0 2016-04-07T01:40:33.943-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 847 163938847128388 0 2016-04-07T01:40:33.946-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 848 194545985476477 0 2016-04-07T01:40:33.948-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 849 167084296056238 0 2016-04-07T01:40:33.951-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 850 44492449492906 0 2016-04-07T01:40:33.951-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 851 33181197746246 0 2016-04-07T01:40:33.952-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 110000 QueuedCount 50000 2016-04-07T01:40:33.957-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 852 263873911362631 0 2016-04-07T01:40:33.957-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 853 92134740943423 0 2016-04-07T01:40:33.96-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 832 266614626909633 0 2016-04-07T01:40:33.96-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 854 223056690805508 0 2016-04-07T01:40:33.962-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 855 38784593940824 0 2016-04-07T01:40:33.963-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 856 175379834807291 0 2016-04-07T01:40:33.964-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 857 227555696825491 0 2016-04-07T01:40:33.968-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 859 8951431726019 0 2016-04-07T01:40:33.968-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 858 2749136218894 0 2016-04-07T01:40:33.97-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 860 165997334633164 0 2016-04-07T01:40:33.973-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 861 38452082968622 0 2016-04-07T01:40:33.974-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 390000 2016-04-07T01:40:33.974-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 863 22663655592060 0 2016-04-07T01:40:33.974-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 862 125508309298686 0 2016-04-07T01:40:33.976-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 833 86776578222493 0 2016-04-07T01:40:33.977-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 864 154545738041888 0 2016-04-07T01:40:33.978-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 866 226040801415397 0 2016-04-07T01:40:33.978-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 865 231980311324454 0 2016-04-07T01:40:33.982-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 868 241285351122591 0 2016-04-07T01:40:33.982-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 867 19777323876132 0 2016-04-07T01:40:33.984-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 869 106659549173864 0 2016-04-07T01:40:33.986-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 870 146836865073731 0 2016-04-07T01:40:33.987-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 872 172989811505877 0 2016-04-07T01:40:33.988-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 871 194052109916259 0 2016-04-07T01:40:33.989-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 873 120357945555307 0 2016-04-07T01:40:33.99-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 874 158362999239636 0 2016-04-07T01:40:33.99-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:40:33.992-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 875 113490401094560 0 2016-04-07T01:40:33.994-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 876 51900481267766 0 2016-04-07T01:40:33.998-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 877 74182213670935 0 2016-04-07T01:40:34.001-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 878 54597902011918 0 2016-04-07T01:40:34.002-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 400000 2016-04-07T01:40:34.003-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 879 32786384975144 0 2016-04-07T01:40:34.006-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 880 236503058173374 0 2016-04-07T01:40:34.008-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 881 108282096649359 0 2016-04-07T01:40:34.01-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 882 178575496325316 0 2016-04-07T01:40:34.01-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 883 271934342714608 0 2016-04-07T01:40:34.012-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 884 228093218064383 0 2016-04-07T01:40:34.014-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 885 251100636858351 0 2016-04-07T01:40:34.015-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 886 233439669490785 0 2016-04-07T01:40:34.017-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 887 266092365192296 0 2016-04-07T01:40:34.02-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 888 109759040477315 0 2016-04-07T01:40:34.022-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 889 82273330059023 0 2016-04-07T01:40:34.022-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 890 100846065687918 0 2016-04-07T01:40:34.024-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 891 44023981224968 0 2016-04-07T01:40:34.026-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 892 48710024280890 0 2016-04-07T01:40:34.028-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 893 264419436137468 0 2016-04-07T01:40:34.03-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 410000 2016-04-07T01:40:34.03-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 894 195732571309128 0 2016-04-07T01:40:34.032-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 895 205777679358674 0 2016-04-07T01:40:34.037-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:40:34.041-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 901 22950834970492 0 2016-04-07T01:40:34.041-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 902 235590641162262 0 2016-04-07T01:40:34.042-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 900 1987592103907 0 2016-04-07T01:40:34.052-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:40:34.052-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:40:34.054-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 904 132235146944312 0 2016-04-07T01:40:34.054-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 903 28385848037620 0 2016-04-07T01:40:34.056-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 905 192784147146524 0 2016-04-07T01:40:34.056-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 906 86177601608847 0 2016-04-07T01:40:34.058-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 420000 2016-04-07T01:40:34.058-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 907 143131728023238 0 2016-04-07T01:40:34.06-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 908 135981876191609 0 2016-04-07T01:40:34.061-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 909 97846627359153 0 2016-04-07T01:40:34.064-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 911 273351771790701 0 2016-04-07T01:40:34.064-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 910 53599352098618 0 2016-04-07T01:40:34.065-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 912 258823183588264 0 2016-04-07T01:40:34.067-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 913 160254292913797 0 2016-04-07T01:40:34.069-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 914 263500206960949 0 2016-04-07T01:40:34.071-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 915 50555201443308 0 2016-04-07T01:40:34.073-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 916 173021682352387 0 2016-04-07T01:40:34.075-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 917 180526784133297 0 2016-04-07T01:40:34.077-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 918 206049995868418 0 2016-04-07T01:40:34.077-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 899 100324473505079 0 2016-04-07T01:40:34.079-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 919 8220690870647 0 2016-04-07T01:40:34.081-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 898 261017622669486 0 2016-04-07T01:40:34.081-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 920 166616051240724 0 2016-04-07T01:40:34.082-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 921 92225782367738 0 2016-04-07T01:40:34.084-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 922 270606774844465 0 2016-04-07T01:40:34.087-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 923 87709581930409 0 2016-04-07T01:40:34.089-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 924 274933650058684 0 2016-04-07T01:40:34.089-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 430000 2016-04-07T01:40:34.09-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 925 105766906956784 0 2016-04-07T01:40:34.092-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 926 11877639688319 0 2016-04-07T01:40:34.094-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 927 101450603738427 0 2016-04-07T01:40:34.096-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 928 280333745323232 0 2016-04-07T01:40:34.096-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 929 158287611910501 0 2016-04-07T01:40:34.097-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 930 257054954531733 0 2016-04-07T01:40:34.101-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 932 108399311735020 0 2016-04-07T01:40:34.101-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 931 135997251132730 0 2016-04-07T01:40:34.103-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 933 117248853635644 0 2016-04-07T01:40:34.105-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 934 165706037185806 0 2016-04-07T01:40:34.107-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 935 10702827088957 0 2016-04-07T01:40:34.109-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 936 75651540637428 0 2016-04-07T01:40:34.11-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 937 255339352436559 0 2016-04-07T01:40:34.112-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 938 182859683370318 0 2016-04-07T01:40:34.115-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 939 15016920434170 0 2016-04-07T01:40:34.117-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 896 265204963676224 0 2016-04-07T01:40:34.117-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 940 178452270845630 0 2016-04-07T01:40:34.117-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 897 27335519897227 0 2016-04-07T01:40:34.117-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 440000 2016-04-07T01:40:34.118-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 941 190311171192780 0 2016-04-07T01:40:34.121-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 942 194681604729004 0 2016-04-07T01:40:34.122-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 943 72515481479053 0 2016-04-07T01:40:34.126-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 944 79794311328115 0 2016-04-07T01:40:34.127-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 945 259655749339078 0 2016-04-07T01:40:34.129-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 946 281386233226459 0 2016-04-07T01:40:34.13-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 947 75539687088406 0 2016-04-07T01:40:34.132-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 948 159889912249187 0 2016-04-07T01:40:34.134-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 949 97802938485417 0 2016-04-07T01:40:34.138-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 950 56385540584519 0 2016-04-07T01:40:34.14-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 951 245987675728486 0 2016-04-07T01:40:34.142-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 952 214016412097970 0 2016-04-07T01:40:34.144-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 953 201501464222875 0 2016-04-07T01:40:34.146-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 954 140247494724867 0 2016-04-07T01:40:34.147-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 450000 2016-04-07T01:40:34.148-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 955 203232173954490 0 2016-04-07T01:40:34.15-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 956 77884454231334 0 2016-04-07T01:40:34.152-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 957 177390703796905 0 2016-04-07T01:40:34.154-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 958 52630394160475 0 2016-04-07T01:40:34.155-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 959 177540846417351 0 2016-04-07T01:40:34.162-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 962 99925294304439 0 2016-04-07T01:40:34.163-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 963 99265834758091 0 2016-04-07T01:40:34.164-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 965 201279298064564 0 2016-04-07T01:40:34.167-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 967 41089932396533 0 2016-04-07T01:40:34.167-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 966 167692882694406 0 2016-04-07T01:40:34.172-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 460000 2016-04-07T01:40:34.172-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 969 138068846362015 0 2016-04-07T01:40:34.172-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 968 147390448093405 0 2016-04-07T01:40:34.172-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 964 197031466237016 0 2016-04-07T01:40:34.174-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 970 189776016634130 0 2016-04-07T01:40:34.175-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 971 53122009238994 0 2016-04-07T01:40:34.176-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 973 87557228395401 0 2016-04-07T01:40:34.176-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 972 222812965050068 0 2016-04-07T01:40:34.178-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 974 18959651237754 0 2016-04-07T01:40:34.179-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 975 88643388481600 0 2016-04-07T01:40:34.18-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 976 76628705784698 0 2016-04-07T01:40:34.182-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 978 27718686300755 0 2016-04-07T01:40:34.182-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 977 144036042745893 0 2016-04-07T01:40:34.183-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 120000 QueuedCount 50000 2016-04-07T01:40:34.184-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 979 52257116888734 0 2016-04-07T01:40:34.185-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 980 80231177236796 0 2016-04-07T01:40:34.186-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 981 76538245942385 0 2016-04-07T01:40:34.188-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 983 15200823426233 0 2016-04-07T01:40:34.188-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 982 240881219650558 0 2016-04-07T01:40:34.189-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 984 245074169822026 0 2016-04-07T01:40:34.191-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 985 265842611968983 0 2016-04-07T01:40:34.192-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 986 219464851735090 0 2016-04-07T01:40:34.194-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 987 75446034953088 0 2016-04-07T01:40:34.195-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 988 99361797614379 0 2016-04-07T01:40:34.196-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 470000 2016-04-07T01:40:34.196-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 989 76778605281650 0 2016-04-07T01:40:34.217-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 990 148788822766681 0 2016-04-07T01:40:34.217-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 992 120398345588805 0 2016-04-07T01:40:34.217-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 991 53280366075082 0 2016-04-07T01:40:34.217-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 993 234256575607560 0 2016-04-07T01:40:34.222-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 994 146835025901698 0 2016-04-07T01:40:34.222-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 995 116792240774004 0 2016-04-07T01:40:34.223-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 998 62004301354551 0 2016-04-07T01:40:34.223-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 996 158062602457496 0 2016-04-07T01:40:34.223-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 997 180845047600116 0 2016-04-07T01:40:34.224-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 999 147462687857864 0 2016-04-07T01:40:34.225-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1000 131969516711878 0 2016-04-07T01:40:34.225-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1002 1420533371232 0 2016-04-07T01:40:34.225-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1001 166759762554854 0 2016-04-07T01:40:34.226-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1003 49963468408605 0 2016-04-07T01:40:34.226-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 961 58388910476743 0 2016-04-07T01:40:34.226-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1004 253157896376997 0 2016-04-07T01:40:34.227-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1005 125176074811179 0 2016-04-07T01:40:34.227-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1006 90610836447905 0 2016-04-07T01:40:34.23-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1010 208804800816664 0 2016-04-07T01:40:34.23-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1009 248067331504442 0 2016-04-07T01:40:34.23-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1007 110597737628189 0 2016-04-07T01:40:34.23-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 480000 2016-04-07T01:40:34.231-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1008 85189580463852 0 2016-04-07T01:40:34.231-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1013 46103663180683 0 2016-04-07T01:40:34.231-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1011 73721817224870 0 2016-04-07T01:40:34.231-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1014 27674876412373 0 2016-04-07T01:40:34.233-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1018 47530510739701 0 2016-04-07T01:40:34.233-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1015 34325260633545 0 2016-04-07T01:40:34.233-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1012 6864568681105 0 2016-04-07T01:40:34.234-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1017 210133914034229 0 2016-04-07T01:40:34.236-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1019 84780871345392 0 2016-04-07T01:40:34.237-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1016 222334993860403 0 2016-04-07T01:40:34.239-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1021 60615446903445 0 2016-04-07T01:40:34.241-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1020 225274629892100 0 2016-04-07T01:40:34.241-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1022 174110473726014 0 2016-04-07T01:40:34.243-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 960 26703928420217 0 2016-04-07T01:40:34.247-07:00 [Info] KVSender::sendMutationTopicRequest Success Projector 172.23.99.251:9999 Topic MAINT_STREAM_TOPIC_52:a9:25:ef:79:3b:5c:c0 bucket-1 InstanceIds [8343618505678776570] 2016-04-07T01:40:34.248-07:00 [Info] Indexer::sendStreamUpdateForBuildIndex Stream Request Success For Stream MAINT_STREAM Bucket bucket-1. 2016-04-07T01:40:34.249-07:00 [Info] Indexer::handleStreamRequestDone StreamId MAINT_STREAM Bucket bucket-1 2016-04-07T01:40:34.249-07:00 [Info] Timekeeper::handleStreamRequestDone StreamId MAINT_STREAM Bucket bucket-1 2016-04-07T01:40:34.249-07:00 [Info] Timekeeper::handleStreamRequestDone Call RepairStream to check for vbucket for repair. StreamId MAINT_STREAM bucket bucket-1 2016-04-07T01:40:34.249-07:00 [Info] Timekeeper::handleStreamRequestDone Call RepairMissingStreamBegin to check for vbucket for repair. StreamId MAINT_STREAM bucket bucket-1 2016-04-07T01:40:34.249-07:00 [Info] timekeeper.repairMissingStreamBegin stream MAINT_STREAM 2016-04-07T01:40:34.249-07:00 [Info] TK StreamBegin MAINT_STREAM bucket-1 1023 68563294351828 0 2016-04-07T01:40:34.252-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 490000 2016-04-07T01:40:34.281-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 500000 2016-04-07T01:40:34.306-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 510000 2016-04-07T01:40:34.332-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 520000 2016-04-07T01:40:34.357-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 530000 2016-04-07T01:40:34.38-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 540000 2016-04-07T01:40:34.402-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 550000 2016-04-07T01:40:34.427-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 560000 2016-04-07T01:40:34.454-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 570000 2016-04-07T01:40:34.456-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 130000 QueuedCount 50000 2016-04-07T01:40:34.478-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 580000 2016-04-07T01:40:34.506-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 590000 2016-04-07T01:40:34.53-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 600000 2016-04-07T01:40:34.556-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 610000 2016-04-07T01:40:34.581-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 620000 2016-04-07T01:40:34.607-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 630000 2016-04-07T01:40:34.631-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 640000 2016-04-07T01:40:34.653-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 650000 2016-04-07T01:40:34.681-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 660000 2016-04-07T01:40:34.705-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 670000 2016-04-07T01:40:34.707-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 140000 QueuedCount 50000 2016-04-07T01:40:34.728-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 680000 2016-04-07T01:40:34.749-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 690000 2016-04-07T01:40:34.783-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 700000 2016-04-07T01:40:34.805-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 710000 2016-04-07T01:40:34.832-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 720000 2016-04-07T01:40:34.855-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 730000 2016-04-07T01:40:34.883-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 740000 2016-04-07T01:40:34.905-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 750000 2016-04-07T01:40:34.926-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 760000 2016-04-07T01:40:34.95-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 770000 2016-04-07T01:40:34.978-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 780000 2016-04-07T01:40:34.993-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 150000 QueuedCount 50000 2016-04-07T01:40:35.005-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:40:35.005-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 790000 2016-04-07T01:40:35.028-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 800000 2016-04-07T01:40:35.049-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 810000 2016-04-07T01:40:35.069-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 820000 2016-04-07T01:40:35.096-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 830000 2016-04-07T01:40:35.118-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 840000 2016-04-07T01:40:35.139-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 850000 2016-04-07T01:40:35.165-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 860000 2016-04-07T01:40:35.188-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 870000 2016-04-07T01:40:35.21-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 880000 2016-04-07T01:40:35.229-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 890000 2016-04-07T01:40:35.249-07:00 [Info] StreamState::adjustNonSnapAlignedVbs Bucket bucket-1 StreamId MAINT_STREAM Vbuckets []. 2016-04-07T01:40:35.249-07:00 [Info] Timekeeper::repairStream Nothing to repair for Stream MAINT_STREAM and Bucket bucket-1 2016-04-07T01:40:35.299-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 900000 2016-04-07T01:40:35.314-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 910000 2016-04-07T01:40:35.326-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 160000 QueuedCount 50000 2016-04-07T01:40:35.33-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 920000 2016-04-07T01:40:35.344-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 930000 2016-04-07T01:40:35.362-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 940000 2016-04-07T01:40:35.386-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 950000 2016-04-07T01:40:35.412-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 960000 2016-04-07T01:40:35.434-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 970000 2016-04-07T01:40:35.455-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 980000 2016-04-07T01:40:35.476-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 990000 2016-04-07T01:40:35.498-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1000000 2016-04-07T01:40:35.52-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1010000 2016-04-07T01:40:35.541-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1020000 2016-04-07T01:40:35.562-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1030000 2016-04-07T01:40:35.586-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 170000 QueuedCount 50000 2016-04-07T01:40:35.842-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 180000 QueuedCount 50000 2016-04-07T01:40:36.121-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 190000 QueuedCount 50000 2016-04-07T01:40:36.249-07:00 [Info] timekeeper.repairMissingStreamBegin stream MAINT_STREAM done 2016-04-07T01:40:36.341-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:40:36.391-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 200000 QueuedCount 50000 2016-04-07T01:40:36.668-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 210000 QueuedCount 50000 2016-04-07T01:40:36.939-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 220000 QueuedCount 50000 2016-04-07T01:40:37.22-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 230000 QueuedCount 50000 2016-04-07T01:40:37.472-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 240000 QueuedCount 50000 2016-04-07T01:40:37.783-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 250000 QueuedCount 50000 2016-04-07T01:40:38.031-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 260000 QueuedCount 44371 2016-04-07T01:40:38.304-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 270000 QueuedCount 34371 2016-04-07T01:40:38.583-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 280000 QueuedCount 24371 2016-04-07T01:40:38.862-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 290000 QueuedCount 14371 2016-04-07T01:40:39.146-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 300000 QueuedCount 4371 2016-04-07T01:40:39.421-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 1.360085398s CommitTime 156.190215ms TotalFlushTime 7.353755213s TotalCommitTime 156.190215ms 2016-04-07T01:40:39.422-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 304371, 304371, 1 committed:true 2016-04-07T01:40:39.422-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 5711982616188364156 (SnapshotInfo: seqnos: 304371, 304371, 1 committed:true) SnapCreateDur 1.516914663s SnapOpenDur 122.616µs 2016-04-07T01:40:39.457-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1040000 2016-04-07T01:40:39.471-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1050000 2016-04-07T01:40:39.483-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1060000 2016-04-07T01:40:39.495-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1070000 2016-04-07T01:40:39.508-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1080000 2016-04-07T01:40:39.519-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1090000 2016-04-07T01:40:39.53-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1100000 2016-04-07T01:40:39.541-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1110000 2016-04-07T01:40:39.552-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 310000 QueuedCount 50000 2016-04-07T01:40:39.553-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1120000 2016-04-07T01:40:39.565-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1130000 2016-04-07T01:40:39.576-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1140000 2016-04-07T01:40:39.587-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1150000 2016-04-07T01:40:39.607-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:40:39.667-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:82372321, Disk:80707584, ExtraSnapshotData:0, Fragmentation:0%, mode : circular 2016-04-07T01:40:39.667-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:40:39.675-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1160000 2016-04-07T01:40:39.799-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1170000 2016-04-07T01:40:39.819-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 320000 QueuedCount 50000 2016-04-07T01:40:39.99-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1180000 2016-04-07T01:40:40.061-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1190000 2016-04-07T01:40:40.074-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1200000 2016-04-07T01:40:40.087-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1210000 2016-04-07T01:40:40.099-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1220000 2016-04-07T01:40:40.113-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1230000 2016-04-07T01:40:40.124-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1240000 2016-04-07T01:40:40.131-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 330000 QueuedCount 50000 2016-04-07T01:40:40.138-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1250000 2016-04-07T01:40:40.15-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1260000 2016-04-07T01:40:40.164-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1270000 2016-04-07T01:40:40.177-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1280000 2016-04-07T01:40:40.188-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1290000 2016-04-07T01:40:40.198-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1300000 2016-04-07T01:40:40.377-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1310000 2016-04-07T01:40:40.413-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 340000 QueuedCount 50000 2016-04-07T01:40:40.532-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1320000 2016-04-07T01:40:40.679-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 350000 QueuedCount 50000 2016-04-07T01:40:40.749-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1330000 2016-04-07T01:40:40.953-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 360000 QueuedCount 50000 2016-04-07T01:40:41.026-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1340000 2016-04-07T01:40:41.214-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1350000 2016-04-07T01:40:41.252-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 370000 QueuedCount 50000 2016-04-07T01:40:41.368-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1360000 2016-04-07T01:40:41.466-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1370000 2016-04-07T01:40:41.477-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1380000 2016-04-07T01:40:41.496-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:40:41.508-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1390000 2016-04-07T01:40:41.517-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 380000 QueuedCount 50000 2016-04-07T01:40:41.519-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1400000 2016-04-07T01:40:41.732-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1410000 2016-04-07T01:40:41.783-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 390000 QueuedCount 50000 2016-04-07T01:40:42.008-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1420000 2016-04-07T01:40:42.066-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 400000 QueuedCount 50000 2016-04-07T01:40:42.384-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 410000 QueuedCount 50000 2016-04-07T01:40:42.405-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1430000 2016-04-07T01:40:42.649-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1440000 2016-04-07T01:40:42.709-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 420000 QueuedCount 50000 2016-04-07T01:40:42.837-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1450000 2016-04-07T01:40:43-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 430000 QueuedCount 50000 2016-04-07T01:40:43.024-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1460000 2016-04-07T01:40:43.336-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 440000 QueuedCount 50000 2016-04-07T01:40:43.459-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1470000 2016-04-07T01:40:43.645-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1480000 2016-04-07T01:40:43.694-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 450000 QueuedCount 50000 2016-04-07T01:40:43.714-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1490000 2016-04-07T01:40:43.724-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1500000 2016-04-07T01:40:43.958-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 460000 QueuedCount 41022 2016-04-07T01:40:44.074-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:40:44.088-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:40:44.088-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:40:44.263-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 470000 QueuedCount 31022 2016-04-07T01:40:44.563-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 480000 QueuedCount 21022 2016-04-07T01:40:44.864-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 490000 QueuedCount 11022 2016-04-07T01:40:45.168-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 500000 QueuedCount 1022 2016-04-07T01:40:45.337-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 29.261µs CommitTime 149.595572ms TotalFlushTime 12.965011299s TotalCommitTime 305.785787ms 2016-04-07T01:40:45.337-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 501022, 501022, 2 committed:true 2016-04-07T01:40:45.337-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 501022, 501022, 2 committed:true) SnapCreateDur 151.875078ms SnapOpenDur 90.246µs 2016-04-07T01:40:45.377-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1510000 2016-04-07T01:40:45.387-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1520000 2016-04-07T01:40:45.397-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1530000 2016-04-07T01:40:45.408-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1540000 2016-04-07T01:40:45.418-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1550000 2016-04-07T01:40:45.429-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1560000 2016-04-07T01:40:45.577-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1570000 2016-04-07T01:40:45.609-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 510000 QueuedCount 50000 2016-04-07T01:40:45.648-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1580000 2016-04-07T01:40:45.66-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1590000 2016-04-07T01:40:45.673-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1600000 2016-04-07T01:40:45.684-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1610000 2016-04-07T01:40:45.695-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1620000 2016-04-07T01:40:45.705-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1630000 2016-04-07T01:40:45.716-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1640000 2016-04-07T01:40:45.729-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1650000 2016-04-07T01:40:45.742-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1660000 2016-04-07T01:40:45.752-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1670000 2016-04-07T01:40:45.763-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1680000 2016-04-07T01:40:45.775-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1690000 2016-04-07T01:40:45.79-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1700000 2016-04-07T01:40:45.803-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1710000 2016-04-07T01:40:45.815-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1720000 2016-04-07T01:40:45.826-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1730000 2016-04-07T01:40:45.838-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1740000 2016-04-07T01:40:45.849-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1750000 2016-04-07T01:40:45.861-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1760000 2016-04-07T01:40:45.872-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1770000 2016-04-07T01:40:45.883-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1780000 2016-04-07T01:40:45.895-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1790000 2016-04-07T01:40:45.905-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 520000 QueuedCount 50000 2016-04-07T01:40:45.907-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1800000 2016-04-07T01:40:45.92-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1810000 2016-04-07T01:40:45.931-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1820000 2016-04-07T01:40:45.941-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1830000 2016-04-07T01:40:45.953-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1840000 2016-04-07T01:40:45.966-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1850000 2016-04-07T01:40:45.977-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1860000 2016-04-07T01:40:45.988-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1870000 2016-04-07T01:40:45.998-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1880000 2016-04-07T01:40:46.01-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1890000 2016-04-07T01:40:46.021-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1900000 2016-04-07T01:40:46.032-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1910000 2016-04-07T01:40:46.042-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1920000 2016-04-07T01:40:46.054-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1930000 2016-04-07T01:40:46.065-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1940000 2016-04-07T01:40:46.079-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1950000 2016-04-07T01:40:46.092-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1960000 2016-04-07T01:40:46.105-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1970000 2016-04-07T01:40:46.115-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1980000 2016-04-07T01:40:46.126-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 1990000 2016-04-07T01:40:46.14-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2000000 2016-04-07T01:40:46.153-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2010000 2016-04-07T01:40:46.164-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2020000 2016-04-07T01:40:46.178-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2030000 2016-04-07T01:40:46.188-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 530000 QueuedCount 50000 2016-04-07T01:40:46.189-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2040000 2016-04-07T01:40:46.201-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2050000 2016-04-07T01:40:46.213-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2060000 2016-04-07T01:40:46.224-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2070000 2016-04-07T01:40:46.235-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2080000 2016-04-07T01:40:46.248-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2090000 2016-04-07T01:40:46.248-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:40:46.262-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2100000 2016-04-07T01:40:46.275-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2110000 2016-04-07T01:40:46.286-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2120000 2016-04-07T01:40:46.298-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2130000 2016-04-07T01:40:46.309-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2140000 2016-04-07T01:40:46.322-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2150000 2016-04-07T01:40:46.334-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2160000 2016-04-07T01:40:46.345-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2170000 2016-04-07T01:40:46.36-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2180000 2016-04-07T01:40:46.374-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2190000 2016-04-07T01:40:46.387-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2200000 2016-04-07T01:40:46.401-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2210000 2016-04-07T01:40:46.413-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2220000 2016-04-07T01:40:46.426-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2230000 2016-04-07T01:40:46.44-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2240000 2016-04-07T01:40:46.453-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2250000 2016-04-07T01:40:46.463-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 540000 QueuedCount 50000 2016-04-07T01:40:46.464-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2260000 2016-04-07T01:40:46.475-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2270000 2016-04-07T01:40:46.488-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2280000 2016-04-07T01:40:46.499-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2290000 2016-04-07T01:40:46.51-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2300000 2016-04-07T01:40:46.522-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2310000 2016-04-07T01:40:46.535-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2320000 2016-04-07T01:40:46.547-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2330000 2016-04-07T01:40:46.561-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2340000 2016-04-07T01:40:46.573-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2350000 2016-04-07T01:40:46.584-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2360000 2016-04-07T01:40:46.595-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2370000 2016-04-07T01:40:46.608-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2380000 2016-04-07T01:40:46.619-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2390000 2016-04-07T01:40:46.631-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2400000 2016-04-07T01:40:46.642-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2410000 2016-04-07T01:40:46.654-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2420000 2016-04-07T01:40:46.665-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2430000 2016-04-07T01:40:46.676-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2440000 2016-04-07T01:40:46.688-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2450000 2016-04-07T01:40:46.699-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2460000 2016-04-07T01:40:46.705-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 550000 QueuedCount 50000 2016-04-07T01:40:46.709-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2470000 2016-04-07T01:40:46.72-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2480000 2016-04-07T01:40:46.731-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2490000 2016-04-07T01:40:46.742-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2500000 2016-04-07T01:40:46.753-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2510000 2016-04-07T01:40:46.763-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2520000 2016-04-07T01:40:46.775-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2530000 2016-04-07T01:40:46.787-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2540000 2016-04-07T01:40:46.797-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2550000 2016-04-07T01:40:46.809-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2560000 2016-04-07T01:40:46.822-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2570000 2016-04-07T01:40:46.832-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2580000 2016-04-07T01:40:46.843-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2590000 2016-04-07T01:40:46.856-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2600000 2016-04-07T01:40:46.869-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2610000 2016-04-07T01:40:46.881-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2620000 2016-04-07T01:40:46.892-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2630000 2016-04-07T01:40:46.903-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2640000 2016-04-07T01:40:46.918-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2650000 2016-04-07T01:40:46.931-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2660000 2016-04-07T01:40:46.942-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2670000 2016-04-07T01:40:46.954-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2680000 2016-04-07T01:40:46.965-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2690000 2016-04-07T01:40:46.976-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 560000 QueuedCount 50000 2016-04-07T01:40:46.978-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2700000 2016-04-07T01:40:46.991-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2710000 2016-04-07T01:40:47.002-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2720000 2016-04-07T01:40:47.013-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2730000 2016-04-07T01:40:47.023-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2740000 2016-04-07T01:40:47.034-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2750000 2016-04-07T01:40:47.045-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2760000 2016-04-07T01:40:47.056-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2770000 2016-04-07T01:40:47.067-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2780000 2016-04-07T01:40:47.078-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2790000 2016-04-07T01:40:47.09-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2800000 2016-04-07T01:40:47.102-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2810000 2016-04-07T01:40:47.113-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2820000 2016-04-07T01:40:47.126-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2830000 2016-04-07T01:40:47.14-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2840000 2016-04-07T01:40:47.154-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2850000 2016-04-07T01:40:47.164-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2860000 2016-04-07T01:40:47.177-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2870000 2016-04-07T01:40:47.189-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2880000 2016-04-07T01:40:47.202-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2890000 2016-04-07T01:40:47.213-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2900000 2016-04-07T01:40:47.224-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2910000 2016-04-07T01:40:47.235-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2920000 2016-04-07T01:40:47.245-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 570000 QueuedCount 50000 2016-04-07T01:40:47.246-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2930000 2016-04-07T01:40:47.258-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2940000 2016-04-07T01:40:47.271-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2950000 2016-04-07T01:40:47.284-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2960000 2016-04-07T01:40:47.298-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2970000 2016-04-07T01:40:47.311-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2980000 2016-04-07T01:40:47.322-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 2990000 2016-04-07T01:40:47.334-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3000000 2016-04-07T01:40:47.346-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3010000 2016-04-07T01:40:47.357-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3020000 2016-04-07T01:40:47.368-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3030000 2016-04-07T01:40:47.378-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3040000 2016-04-07T01:40:47.39-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3050000 2016-04-07T01:40:47.403-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3060000 2016-04-07T01:40:47.415-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3070000 2016-04-07T01:40:47.426-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3080000 2016-04-07T01:40:47.44-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3090000 2016-04-07T01:40:47.452-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3100000 2016-04-07T01:40:47.464-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3110000 2016-04-07T01:40:47.475-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3120000 2016-04-07T01:40:47.486-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3130000 2016-04-07T01:40:47.497-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3140000 2016-04-07T01:40:47.502-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 580000 QueuedCount 50000 2016-04-07T01:40:47.51-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3150000 2016-04-07T01:40:47.523-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3160000 2016-04-07T01:40:47.537-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3170000 2016-04-07T01:40:47.551-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3180000 2016-04-07T01:40:47.562-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3190000 2016-04-07T01:40:47.573-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3200000 2016-04-07T01:40:47.587-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3210000 2016-04-07T01:40:47.601-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3220000 2016-04-07T01:40:47.613-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3230000 2016-04-07T01:40:47.624-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3240000 2016-04-07T01:40:47.635-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3250000 2016-04-07T01:40:47.646-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3260000 2016-04-07T01:40:47.66-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3270000 2016-04-07T01:40:47.673-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3280000 2016-04-07T01:40:47.684-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3290000 2016-04-07T01:40:47.699-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3300000 2016-04-07T01:40:47.711-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3310000 2016-04-07T01:40:47.722-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3320000 2016-04-07T01:40:47.734-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3330000 2016-04-07T01:40:47.748-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3340000 2016-04-07T01:40:47.762-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3350000 2016-04-07T01:40:47.773-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3360000 2016-04-07T01:40:47.783-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 590000 QueuedCount 50000 2016-04-07T01:40:47.784-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3370000 2016-04-07T01:40:47.795-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3380000 2016-04-07T01:40:47.806-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3390000 2016-04-07T01:40:47.818-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3400000 2016-04-07T01:40:47.83-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3410000 2016-04-07T01:40:47.842-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3420000 2016-04-07T01:40:47.855-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3430000 2016-04-07T01:40:47.866-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3440000 2016-04-07T01:40:47.877-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3450000 2016-04-07T01:40:47.89-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3460000 2016-04-07T01:40:47.903-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3470000 2016-04-07T01:40:47.914-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3480000 2016-04-07T01:40:47.926-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3490000 2016-04-07T01:40:47.939-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3500000 2016-04-07T01:40:47.951-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3510000 2016-04-07T01:40:47.964-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3520000 2016-04-07T01:40:47.975-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3530000 2016-04-07T01:40:47.987-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3540000 2016-04-07T01:40:47.998-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3550000 2016-04-07T01:40:48.011-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3560000 2016-04-07T01:40:48.023-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3570000 2016-04-07T01:40:48.037-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3580000 2016-04-07T01:40:48.049-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3590000 2016-04-07T01:40:48.06-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3600000 2016-04-07T01:40:48.072-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3610000 2016-04-07T01:40:48.084-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3620000 2016-04-07T01:40:48.095-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3630000 2016-04-07T01:40:48.106-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3640000 2016-04-07T01:40:48.117-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3650000 2016-04-07T01:40:48.132-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3660000 2016-04-07T01:40:48.133-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 600000 QueuedCount 50000 2016-04-07T01:40:48.148-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3670000 2016-04-07T01:40:48.16-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3680000 2016-04-07T01:40:48.172-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3690000 2016-04-07T01:40:48.182-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3700000 2016-04-07T01:40:48.197-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3710000 2016-04-07T01:40:48.209-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3720000 2016-04-07T01:40:48.221-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3730000 2016-04-07T01:40:48.232-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3740000 2016-04-07T01:40:48.242-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3750000 2016-04-07T01:40:48.257-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3760000 2016-04-07T01:40:48.27-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3770000 2016-04-07T01:40:48.283-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3780000 2016-04-07T01:40:48.293-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3790000 2016-04-07T01:40:48.304-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3800000 2016-04-07T01:40:48.315-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3810000 2016-04-07T01:40:48.328-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3820000 2016-04-07T01:40:48.341-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3830000 2016-04-07T01:40:48.353-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3840000 2016-04-07T01:40:48.363-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3850000 2016-04-07T01:40:48.376-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3860000 2016-04-07T01:40:48.387-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3870000 2016-04-07T01:40:48.399-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3880000 2016-04-07T01:40:48.412-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3890000 2016-04-07T01:40:48.423-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3900000 2016-04-07T01:40:48.434-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3910000 2016-04-07T01:40:48.446-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3920000 2016-04-07T01:40:48.457-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 610000 QueuedCount 50000 2016-04-07T01:40:48.46-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3930000 2016-04-07T01:40:48.475-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3940000 2016-04-07T01:40:48.476-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:40:48.486-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3950000 2016-04-07T01:40:48.498-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3960000 2016-04-07T01:40:48.509-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3970000 2016-04-07T01:40:48.52-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3980000 2016-04-07T01:40:48.531-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 3990000 2016-04-07T01:40:48.542-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4000000 2016-04-07T01:40:48.552-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4010000 2016-04-07T01:40:48.565-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4020000 2016-04-07T01:40:48.576-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4030000 2016-04-07T01:40:48.586-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4040000 2016-04-07T01:40:48.597-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4050000 2016-04-07T01:40:48.608-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4060000 2016-04-07T01:40:48.621-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4070000 2016-04-07T01:40:48.632-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4080000 2016-04-07T01:40:48.645-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4090000 2016-04-07T01:40:48.656-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4100000 2016-04-07T01:40:48.666-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4110000 2016-04-07T01:40:48.677-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4120000 2016-04-07T01:40:48.688-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4130000 2016-04-07T01:40:48.698-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4140000 2016-04-07T01:40:48.709-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4150000 2016-04-07T01:40:48.786-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 620000 QueuedCount 50000 2016-04-07T01:40:48.858-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4160000 2016-04-07T01:40:49.04-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 630000 QueuedCount 50000 2016-04-07T01:40:49.095-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:40:49.109-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:40:49.109-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:40:49.285-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4170000 2016-04-07T01:40:49.341-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 640000 QueuedCount 50000 2016-04-07T01:40:49.531-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4180000 2016-04-07T01:40:49.666-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 650000 QueuedCount 50000 2016-04-07T01:40:49.808-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4190000 2016-04-07T01:40:49.989-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 660000 QueuedCount 50000 2016-04-07T01:40:50.173-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4200000 2016-04-07T01:40:50.316-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 670000 QueuedCount 50000 2016-04-07T01:40:50.449-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4210000 2016-04-07T01:40:50.607-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4220000 2016-04-07T01:40:50.623-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 680000 QueuedCount 50000 2016-04-07T01:40:50.821-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4230000 2016-04-07T01:40:50.928-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 690000 QueuedCount 50000 2016-04-07T01:40:50.947-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4240000 2016-04-07T01:40:51.25-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 700000 QueuedCount 40955 2016-04-07T01:40:51.576-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 710000 QueuedCount 30955 2016-04-07T01:40:51.859-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 720000 QueuedCount 20955 2016-04-07T01:40:52.188-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 730000 QueuedCount 10955 2016-04-07T01:40:52.517-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 740000 QueuedCount 955 2016-04-07T01:40:52.742-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 31.059µs CommitTime 159.529919ms TotalFlushTime 20.019461016s TotalCommitTime 465.315706ms 2016-04-07T01:40:52.742-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 740955, 740955, 3 committed:true 2016-04-07T01:40:52.742-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 740955, 740955, 3 committed:true) SnapCreateDur 163.649823ms SnapOpenDur 62.93µs 2016-04-07T01:40:52.783-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4250000 2016-04-07T01:40:52.797-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4260000 2016-04-07T01:40:52.809-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4270000 2016-04-07T01:40:52.927-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4280000 2016-04-07T01:40:53.032-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 750000 QueuedCount 50000 2016-04-07T01:40:53.114-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4290000 2016-04-07T01:40:53.271-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4300000 2016-04-07T01:40:53.399-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 760000 QueuedCount 50000 2016-04-07T01:40:53.484-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4310000 2016-04-07T01:40:53.552-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4320000 2016-04-07T01:40:53.676-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4330000 2016-04-07T01:40:53.688-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4340000 2016-04-07T01:40:53.703-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4350000 2016-04-07T01:40:53.705-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 770000 QueuedCount 50000 2016-04-07T01:40:53.714-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4360000 2016-04-07T01:40:53.725-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4370000 2016-04-07T01:40:53.737-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4380000 2016-04-07T01:40:53.749-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4390000 2016-04-07T01:40:53.76-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4400000 2016-04-07T01:40:53.773-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4410000 2016-04-07T01:40:53.786-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4420000 2016-04-07T01:40:53.799-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4430000 2016-04-07T01:40:53.81-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4440000 2016-04-07T01:40:53.82-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4450000 2016-04-07T01:40:53.834-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4460000 2016-04-07T01:40:53.846-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4470000 2016-04-07T01:40:53.857-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4480000 2016-04-07T01:40:53.867-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4490000 2016-04-07T01:40:53.878-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4500000 2016-04-07T01:40:53.88-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:40:53.891-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4510000 2016-04-07T01:40:53.903-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4520000 2016-04-07T01:40:53.914-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4530000 2016-04-07T01:40:53.924-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4540000 2016-04-07T01:40:53.935-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4550000 2016-04-07T01:40:54.028-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4560000 2016-04-07T01:40:54.069-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 780000 QueuedCount 49935 2016-04-07T01:40:54.146-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4570000 2016-04-07T01:40:54.222-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4580000 2016-04-07T01:40:54.272-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4590000 2016-04-07T01:40:54.287-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:40:54.377-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:40:54.377-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:40:54.377-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4600000 2016-04-07T01:40:54.413-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4610000 2016-04-07T01:40:54.426-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4620000 2016-04-07T01:40:54.439-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4630000 2016-04-07T01:40:54.45-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4640000 2016-04-07T01:40:54.461-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4650000 2016-04-07T01:40:54.472-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4660000 2016-04-07T01:40:54.473-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 790000 QueuedCount 50000 2016-04-07T01:40:54.483-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4670000 2016-04-07T01:40:54.496-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4680000 2016-04-07T01:40:54.506-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4690000 2016-04-07T01:40:54.517-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4700000 2016-04-07T01:40:54.528-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4710000 2016-04-07T01:40:54.541-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4720000 2016-04-07T01:40:54.553-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4730000 2016-04-07T01:40:54.569-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4740000 2016-04-07T01:40:54.586-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4750000 2016-04-07T01:40:54.599-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4760000 2016-04-07T01:40:54.613-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4770000 2016-04-07T01:40:54.624-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4780000 2016-04-07T01:40:54.635-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4790000 2016-04-07T01:40:54.647-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4800000 2016-04-07T01:40:54.659-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4810000 2016-04-07T01:40:54.672-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4820000 2016-04-07T01:40:54.684-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4830000 2016-04-07T01:40:54.699-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4840000 2016-04-07T01:40:54.713-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4850000 2016-04-07T01:40:54.727-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4860000 2016-04-07T01:40:54.74-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4870000 2016-04-07T01:40:54.751-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4880000 2016-04-07T01:40:54.765-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4890000 2016-04-07T01:40:54.778-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4900000 2016-04-07T01:40:54.793-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4910000 2016-04-07T01:40:54.8-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 800000 QueuedCount 50000 2016-04-07T01:40:54.808-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4920000 2016-04-07T01:40:54.821-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4930000 2016-04-07T01:40:54.837-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4940000 2016-04-07T01:40:54.854-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4950000 2016-04-07T01:40:54.87-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4960000 2016-04-07T01:40:54.884-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4970000 2016-04-07T01:40:54.896-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4980000 2016-04-07T01:40:54.911-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 4990000 2016-04-07T01:40:54.926-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5000000 2016-04-07T01:40:54.942-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5010000 2016-04-07T01:40:54.958-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5020000 2016-04-07T01:40:54.976-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5030000 2016-04-07T01:40:54.99-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5040000 2016-04-07T01:40:55.004-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5050000 2016-04-07T01:40:55.019-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5060000 2016-04-07T01:40:55.033-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5070000 2016-04-07T01:40:55.049-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5080000 2016-04-07T01:40:55.064-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5090000 2016-04-07T01:40:55.077-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5100000 2016-04-07T01:40:55.094-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5110000 2016-04-07T01:40:55.11-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5120000 2016-04-07T01:40:55.126-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 810000 QueuedCount 50000 2016-04-07T01:40:55.14-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5130000 2016-04-07T01:40:55.164-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5140000 2016-04-07T01:40:55.188-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5150000 2016-04-07T01:40:55.21-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5160000 2016-04-07T01:40:55.231-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5170000 2016-04-07T01:40:55.254-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5180000 2016-04-07T01:40:55.276-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5190000 2016-04-07T01:40:55.297-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5200000 2016-04-07T01:40:55.315-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5210000 2016-04-07T01:40:55.327-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5220000 2016-04-07T01:40:55.339-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5230000 2016-04-07T01:40:55.353-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5240000 2016-04-07T01:40:55.366-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5250000 2016-04-07T01:40:55.382-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5260000 2016-04-07T01:40:55.398-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5270000 2016-04-07T01:40:55.412-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5280000 2016-04-07T01:40:55.423-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5290000 2016-04-07T01:40:55.436-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5300000 2016-04-07T01:40:55.449-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5310000 2016-04-07T01:40:55.465-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5320000 2016-04-07T01:40:55.472-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 820000 QueuedCount 50000 2016-04-07T01:40:55.478-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5330000 2016-04-07T01:40:55.489-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5340000 2016-04-07T01:40:55.503-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5350000 2016-04-07T01:40:55.515-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5360000 2016-04-07T01:40:55.53-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5370000 2016-04-07T01:40:55.544-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5380000 2016-04-07T01:40:55.554-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5390000 2016-04-07T01:40:55.564-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5400000 2016-04-07T01:40:55.575-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5410000 2016-04-07T01:40:55.586-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5420000 2016-04-07T01:40:55.598-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5430000 2016-04-07T01:40:55.611-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5440000 2016-04-07T01:40:55.623-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5450000 2016-04-07T01:40:55.633-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5460000 2016-04-07T01:40:55.643-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5470000 2016-04-07T01:40:55.655-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5480000 2016-04-07T01:40:55.666-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5490000 2016-04-07T01:40:55.678-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5500000 2016-04-07T01:40:55.689-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5510000 2016-04-07T01:40:55.699-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5520000 2016-04-07T01:40:55.708-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5530000 2016-04-07T01:40:55.718-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5540000 2016-04-07T01:40:55.729-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5550000 2016-04-07T01:40:55.738-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5560000 2016-04-07T01:40:55.748-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5570000 2016-04-07T01:40:55.758-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5580000 2016-04-07T01:40:55.768-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5590000 2016-04-07T01:40:55.78-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5600000 2016-04-07T01:40:55.791-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5610000 2016-04-07T01:40:55.795-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 830000 QueuedCount 50000 2016-04-07T01:40:55.801-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5620000 2016-04-07T01:40:55.812-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5630000 2016-04-07T01:40:55.822-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5640000 2016-04-07T01:40:55.833-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5650000 2016-04-07T01:40:55.846-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5660000 2016-04-07T01:40:55.859-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5670000 2016-04-07T01:40:55.87-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5680000 2016-04-07T01:40:55.881-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5690000 2016-04-07T01:40:55.891-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5700000 2016-04-07T01:40:55.901-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5710000 2016-04-07T01:40:55.913-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5720000 2016-04-07T01:40:55.923-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5730000 2016-04-07T01:40:55.934-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5740000 2016-04-07T01:40:55.943-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5750000 2016-04-07T01:40:55.953-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5760000 2016-04-07T01:40:55.963-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5770000 2016-04-07T01:40:55.973-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5780000 2016-04-07T01:40:55.982-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5790000 2016-04-07T01:40:55.992-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5800000 2016-04-07T01:40:56.002-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5810000 2016-04-07T01:40:56.013-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5820000 2016-04-07T01:40:56.023-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5830000 2016-04-07T01:40:56.033-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5840000 2016-04-07T01:40:56.043-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5850000 2016-04-07T01:40:56.054-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5860000 2016-04-07T01:40:56.063-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5870000 2016-04-07T01:40:56.073-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5880000 2016-04-07T01:40:56.083-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5890000 2016-04-07T01:40:56.093-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5900000 2016-04-07T01:40:56.103-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5910000 2016-04-07T01:40:56.113-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5920000 2016-04-07T01:40:56.114-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 840000 QueuedCount 50000 2016-04-07T01:40:56.124-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5930000 2016-04-07T01:40:56.134-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5940000 2016-04-07T01:40:56.144-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5950000 2016-04-07T01:40:56.155-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5960000 2016-04-07T01:40:56.167-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5970000 2016-04-07T01:40:56.178-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5980000 2016-04-07T01:40:56.189-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 5990000 2016-04-07T01:40:56.201-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6000000 2016-04-07T01:40:56.211-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6010000 2016-04-07T01:40:56.221-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6020000 2016-04-07T01:40:56.232-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6030000 2016-04-07T01:40:56.243-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6040000 2016-04-07T01:40:56.253-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6050000 2016-04-07T01:40:56.265-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6060000 2016-04-07T01:40:56.277-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6070000 2016-04-07T01:40:56.287-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6080000 2016-04-07T01:40:56.3-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6090000 2016-04-07T01:40:56.314-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6100000 2016-04-07T01:40:56.327-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6110000 2016-04-07T01:40:56.339-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6120000 2016-04-07T01:40:56.349-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6130000 2016-04-07T01:40:56.359-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6140000 2016-04-07T01:40:56.457-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 850000 QueuedCount 49515 2016-04-07T01:40:56.76-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 860000 QueuedCount 39515 2016-04-07T01:40:57.008-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 870000 QueuedCount 29515 2016-04-07T01:40:57.293-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 880000 QueuedCount 19515 2016-04-07T01:40:57.58-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 890000 QueuedCount 9515 2016-04-07T01:40:58.007-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.596µs CommitTime 133.581517ms TotalFlushTime 25.00164388s TotalCommitTime 598.897223ms 2016-04-07T01:40:58.007-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 899515, 899515, 4 committed:true 2016-04-07T01:40:58.007-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 899515, 899515, 4 committed:true) SnapCreateDur 139.780996ms SnapOpenDur 75.217µs 2016-04-07T01:40:58.038-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 900000 QueuedCount 50000 2016-04-07T01:40:58.04-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6150000 2016-04-07T01:40:58.371-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 910000 QueuedCount 50000 2016-04-07T01:40:58.694-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 920000 QueuedCount 50000 2016-04-07T01:40:58.973-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 930000 QueuedCount 50000 2016-04-07T01:40:59.301-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 940000 QueuedCount 50000 2016-04-07T01:40:59.302-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:40:59.317-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:40:59.317-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:40:59.446-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:40:59.548-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6160000 2016-04-07T01:40:59.562-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6170000 2016-04-07T01:40:59.574-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6180000 2016-04-07T01:40:59.585-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6190000 2016-04-07T01:40:59.595-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6200000 2016-04-07T01:40:59.606-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6210000 2016-04-07T01:40:59.619-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6220000 2016-04-07T01:40:59.635-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6230000 2016-04-07T01:40:59.648-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6240000 2016-04-07T01:40:59.653-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 950000 QueuedCount 50000 2016-04-07T01:40:59.659-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6250000 2016-04-07T01:40:59.669-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6260000 2016-04-07T01:40:59.68-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6270000 2016-04-07T01:40:59.691-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6280000 2016-04-07T01:40:59.703-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6290000 2016-04-07T01:40:59.714-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6300000 2016-04-07T01:40:59.724-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6310000 2016-04-07T01:40:59.736-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6320000 2016-04-07T01:40:59.747-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6330000 2016-04-07T01:40:59.758-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6340000 2016-04-07T01:40:59.77-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6350000 2016-04-07T01:40:59.783-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6360000 2016-04-07T01:40:59.794-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6370000 2016-04-07T01:40:59.804-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6380000 2016-04-07T01:40:59.815-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6390000 2016-04-07T01:40:59.826-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6400000 2016-04-07T01:40:59.838-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6410000 2016-04-07T01:40:59.852-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6420000 2016-04-07T01:40:59.864-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6430000 2016-04-07T01:40:59.875-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6440000 2016-04-07T01:40:59.885-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6450000 2016-04-07T01:40:59.901-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6460000 2016-04-07T01:40:59.915-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6470000 2016-04-07T01:40:59.928-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6480000 2016-04-07T01:40:59.939-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6490000 2016-04-07T01:40:59.952-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6500000 2016-04-07T01:40:59.963-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6510000 2016-04-07T01:40:59.974-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 960000 QueuedCount 50000 2016-04-07T01:40:59.977-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6520000 2016-04-07T01:40:59.988-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6530000 2016-04-07T01:40:59.999-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6540000 2016-04-07T01:41:00.01-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6550000 2016-04-07T01:41:00.021-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6560000 2016-04-07T01:41:00.034-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6570000 2016-04-07T01:41:00.047-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6580000 2016-04-07T01:41:00.058-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6590000 2016-04-07T01:41:00.068-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6600000 2016-04-07T01:41:00.079-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6610000 2016-04-07T01:41:00.09-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6620000 2016-04-07T01:41:00.101-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6630000 2016-04-07T01:41:00.112-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6640000 2016-04-07T01:41:00.123-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6650000 2016-04-07T01:41:00.134-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6660000 2016-04-07T01:41:00.145-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6670000 2016-04-07T01:41:00.156-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6680000 2016-04-07T01:41:00.167-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6690000 2016-04-07T01:41:00.181-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6700000 2016-04-07T01:41:00.194-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6710000 2016-04-07T01:41:00.207-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6720000 2016-04-07T01:41:00.22-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6730000 2016-04-07T01:41:00.233-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6740000 2016-04-07T01:41:00.244-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6750000 2016-04-07T01:41:00.257-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6760000 2016-04-07T01:41:00.27-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6770000 2016-04-07T01:41:00.284-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6780000 2016-04-07T01:41:00.285-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 970000 QueuedCount 50000 2016-04-07T01:41:00.298-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6790000 2016-04-07T01:41:00.309-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6800000 2016-04-07T01:41:00.32-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6810000 2016-04-07T01:41:00.332-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6820000 2016-04-07T01:41:00.344-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6830000 2016-04-07T01:41:00.355-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6840000 2016-04-07T01:41:00.366-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6850000 2016-04-07T01:41:00.378-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6860000 2016-04-07T01:41:00.39-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6870000 2016-04-07T01:41:00.402-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6880000 2016-04-07T01:41:00.415-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6890000 2016-04-07T01:41:00.427-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6900000 2016-04-07T01:41:00.438-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6910000 2016-04-07T01:41:00.449-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6920000 2016-04-07T01:41:00.459-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6930000 2016-04-07T01:41:00.47-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6940000 2016-04-07T01:41:00.481-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6950000 2016-04-07T01:41:00.491-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6960000 2016-04-07T01:41:00.503-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6970000 2016-04-07T01:41:00.516-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6980000 2016-04-07T01:41:00.528-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 6990000 2016-04-07T01:41:00.542-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7000000 2016-04-07T01:41:00.555-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7010000 2016-04-07T01:41:00.568-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7020000 2016-04-07T01:41:00.579-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7030000 2016-04-07T01:41:00.589-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 980000 QueuedCount 50000 2016-04-07T01:41:00.59-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7040000 2016-04-07T01:41:00.601-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7050000 2016-04-07T01:41:00.613-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7060000 2016-04-07T01:41:00.623-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7070000 2016-04-07T01:41:00.634-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7080000 2016-04-07T01:41:00.646-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7090000 2016-04-07T01:41:00.657-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7100000 2016-04-07T01:41:00.668-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7110000 2016-04-07T01:41:00.679-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7120000 2016-04-07T01:41:00.689-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7130000 2016-04-07T01:41:00.701-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7140000 2016-04-07T01:41:00.711-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7150000 2016-04-07T01:41:00.722-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7160000 2016-04-07T01:41:00.736-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7170000 2016-04-07T01:41:00.751-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7180000 2016-04-07T01:41:00.762-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7190000 2016-04-07T01:41:00.773-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7200000 2016-04-07T01:41:00.785-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7210000 2016-04-07T01:41:00.795-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7220000 2016-04-07T01:41:00.806-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7230000 2016-04-07T01:41:00.819-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7240000 2016-04-07T01:41:00.832-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7250000 2016-04-07T01:41:00.846-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7260000 2016-04-07T01:41:00.856-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7270000 2016-04-07T01:41:00.868-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7280000 2016-04-07T01:41:00.882-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7290000 2016-04-07T01:41:00.884-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 990000 QueuedCount 41385 2016-04-07T01:41:00.893-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7300000 2016-04-07T01:41:00.904-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7310000 2016-04-07T01:41:00.915-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7320000 2016-04-07T01:41:00.927-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7330000 2016-04-07T01:41:00.938-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7340000 2016-04-07T01:41:00.949-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7350000 2016-04-07T01:41:00.959-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7360000 2016-04-07T01:41:00.971-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7370000 2016-04-07T01:41:00.983-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7380000 2016-04-07T01:41:00.995-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7390000 2016-04-07T01:41:01.008-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7400000 2016-04-07T01:41:01.019-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7410000 2016-04-07T01:41:01.031-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7420000 2016-04-07T01:41:01.042-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7430000 2016-04-07T01:41:01.179-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1000000 QueuedCount 31385 2016-04-07T01:41:01.439-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1010000 QueuedCount 21385 2016-04-07T01:41:01.742-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1020000 QueuedCount 11385 2016-04-07T01:41:02.041-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1030000 QueuedCount 1385 2016-04-07T01:41:02.274-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 24.378µs CommitTime 167.784778ms TotalFlushTime 28.979512028s TotalCommitTime 766.682001ms 2016-04-07T01:41:02.274-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 1031385, 1031385, 5 committed:true 2016-04-07T01:41:02.274-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 1031385, 1031385, 5 committed:true) SnapCreateDur 179.43994ms SnapOpenDur 55.282µs 2016-04-07T01:41:02.289-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:02.311-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7440000 2016-04-07T01:41:02.33-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7450000 2016-04-07T01:41:02.342-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7460000 2016-04-07T01:41:02.353-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7470000 2016-04-07T01:41:02.364-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7480000 2016-04-07T01:41:02.377-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7490000 2016-04-07T01:41:02.39-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7500000 2016-04-07T01:41:02.403-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7510000 2016-04-07T01:41:02.414-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7520000 2016-04-07T01:41:02.425-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7530000 2016-04-07T01:41:02.436-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7540000 2016-04-07T01:41:02.448-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7550000 2016-04-07T01:41:02.461-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7560000 2016-04-07T01:41:02.474-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7570000 2016-04-07T01:41:02.485-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7580000 2016-04-07T01:41:02.497-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7590000 2016-04-07T01:41:02.509-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7600000 2016-04-07T01:41:02.52-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7610000 2016-04-07T01:41:02.531-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7620000 2016-04-07T01:41:02.541-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7630000 2016-04-07T01:41:02.553-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7640000 2016-04-07T01:41:02.566-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7650000 2016-04-07T01:41:02.571-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1040000 QueuedCount 50000 2016-04-07T01:41:02.579-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7660000 2016-04-07T01:41:02.592-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7670000 2016-04-07T01:41:02.602-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7680000 2016-04-07T01:41:02.615-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7690000 2016-04-07T01:41:02.626-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7700000 2016-04-07T01:41:02.637-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7710000 2016-04-07T01:41:02.649-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7720000 2016-04-07T01:41:02.662-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7730000 2016-04-07T01:41:02.673-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7740000 2016-04-07T01:41:02.683-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7750000 2016-04-07T01:41:02.695-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7760000 2016-04-07T01:41:02.708-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7770000 2016-04-07T01:41:02.72-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7780000 2016-04-07T01:41:02.731-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7790000 2016-04-07T01:41:02.742-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7800000 2016-04-07T01:41:02.754-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7810000 2016-04-07T01:41:02.768-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7820000 2016-04-07T01:41:02.782-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7830000 2016-04-07T01:41:02.794-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7840000 2016-04-07T01:41:02.807-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7850000 2016-04-07T01:41:02.818-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7860000 2016-04-07T01:41:02.831-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7870000 2016-04-07T01:41:02.844-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7880000 2016-04-07T01:41:02.856-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7890000 2016-04-07T01:41:02.867-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7900000 2016-04-07T01:41:02.877-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7910000 2016-04-07T01:41:02.888-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7920000 2016-04-07T01:41:02.898-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1050000 QueuedCount 50000 2016-04-07T01:41:02.899-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7930000 2016-04-07T01:41:02.91-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7940000 2016-04-07T01:41:02.92-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7950000 2016-04-07T01:41:02.931-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7960000 2016-04-07T01:41:02.942-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7970000 2016-04-07T01:41:02.955-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7980000 2016-04-07T01:41:02.967-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 7990000 2016-04-07T01:41:02.978-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8000000 2016-04-07T01:41:02.989-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8010000 2016-04-07T01:41:03.002-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8020000 2016-04-07T01:41:03.013-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8030000 2016-04-07T01:41:03.024-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8040000 2016-04-07T01:41:03.035-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8050000 2016-04-07T01:41:03.046-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8060000 2016-04-07T01:41:03.057-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8070000 2016-04-07T01:41:03.067-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8080000 2016-04-07T01:41:03.078-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8090000 2016-04-07T01:41:03.09-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8100000 2016-04-07T01:41:03.104-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8110000 2016-04-07T01:41:03.117-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8120000 2016-04-07T01:41:03.128-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8130000 2016-04-07T01:41:03.141-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8140000 2016-04-07T01:41:03.155-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8150000 2016-04-07T01:41:03.168-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8160000 2016-04-07T01:41:03.177-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1060000 QueuedCount 50000 2016-04-07T01:41:03.179-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8170000 2016-04-07T01:41:03.191-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8180000 2016-04-07T01:41:03.202-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8190000 2016-04-07T01:41:03.214-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8200000 2016-04-07T01:41:03.225-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8210000 2016-04-07T01:41:03.236-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8220000 2016-04-07T01:41:03.247-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8230000 2016-04-07T01:41:03.258-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8240000 2016-04-07T01:41:03.268-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8250000 2016-04-07T01:41:03.282-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8260000 2016-04-07T01:41:03.295-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8270000 2016-04-07T01:41:03.307-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8280000 2016-04-07T01:41:03.318-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8290000 2016-04-07T01:41:03.329-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8300000 2016-04-07T01:41:03.34-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8310000 2016-04-07T01:41:03.494-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1070000 QueuedCount 50000 2016-04-07T01:41:03.637-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8320000 2016-04-07T01:41:03.841-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1080000 QueuedCount 50000 2016-04-07T01:41:03.914-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8330000 2016-04-07T01:41:04.187-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1090000 QueuedCount 50000 2016-04-07T01:41:04.219-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8340000 2016-04-07T01:41:04.435-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8350000 2016-04-07T01:41:04.559-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1100000 QueuedCount 50000 2016-04-07T01:41:04.651-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8360000 2016-04-07T01:41:04.778-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8370000 2016-04-07T01:41:04.911-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1110000 QueuedCount 50000 2016-04-07T01:41:04.981-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:04.994-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8380000 2016-04-07T01:41:05.292-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1120000 QueuedCount 50000 2016-04-07T01:41:05.389-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8390000 2016-04-07T01:41:05.718-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1130000 QueuedCount 44700 2016-04-07T01:41:06.152-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1140000 QueuedCount 34700 2016-04-07T01:41:06.52-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1150000 QueuedCount 24700 2016-04-07T01:41:06.95-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1160000 QueuedCount 14700 2016-04-07T01:41:07.391-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1170000 QueuedCount 4700 2016-04-07T01:41:07.858-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.828µs CommitTime 267.440736ms TotalFlushTime 34.17451719s TotalCommitTime 1.034122737s 2016-04-07T01:41:07.858-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 1174700, 1174700, 6 committed:true 2016-04-07T01:41:07.858-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 1174700, 1174700, 6 committed:true) SnapCreateDur 277.049675ms SnapOpenDur 72.686µs 2016-04-07T01:41:07.872-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:07.891-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8400000 2016-04-07T01:41:07.902-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8410000 2016-04-07T01:41:07.915-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8420000 2016-04-07T01:41:07.927-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8430000 2016-04-07T01:41:07.94-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8440000 2016-04-07T01:41:07.953-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8450000 2016-04-07T01:41:07.964-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8460000 2016-04-07T01:41:07.975-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8470000 2016-04-07T01:41:07.988-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8480000 2016-04-07T01:41:08-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8490000 2016-04-07T01:41:08.013-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8500000 2016-04-07T01:41:08.025-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8510000 2016-04-07T01:41:08.036-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8520000 2016-04-07T01:41:08.045-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1180000 QueuedCount 50000 2016-04-07T01:41:08.048-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8530000 2016-04-07T01:41:08.061-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8540000 2016-04-07T01:41:08.073-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8550000 2016-04-07T01:41:08.085-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8560000 2016-04-07T01:41:08.097-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8570000 2016-04-07T01:41:08.108-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8580000 2016-04-07T01:41:08.119-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8590000 2016-04-07T01:41:08.132-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8600000 2016-04-07T01:41:08.145-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8610000 2016-04-07T01:41:08.158-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8620000 2016-04-07T01:41:08.17-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8630000 2016-04-07T01:41:08.182-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8640000 2016-04-07T01:41:08.193-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8650000 2016-04-07T01:41:08.204-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8660000 2016-04-07T01:41:08.216-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8670000 2016-04-07T01:41:08.366-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8680000 2016-04-07T01:41:08.439-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1190000 QueuedCount 50000 2016-04-07T01:41:08.794-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8690000 2016-04-07T01:41:08.847-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1200000 QueuedCount 50000 2016-04-07T01:41:09.204-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1210000 QueuedCount 50000 2016-04-07T01:41:09.219-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8700000 2016-04-07T01:41:09.341-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:41:09.358-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:41:09.358-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:41:09.373-07:00 [Info] connected with 1 indexers 2016-04-07T01:41:09.373-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T01:41:09.374-07:00 [Info] client load stats {} 2016-04-07T01:41:09.59-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1220000 QueuedCount 50000 2016-04-07T01:41:09.616-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8710000 2016-04-07T01:41:09.63-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":7429646,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":5349578451,"bucket-1:myindex:avg_ts_items_count":151731,"bucket-1:myindex:build_progress":5,"bucket-1:myindex:data_size":299618641,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":385495040,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":50079,"bucket-1:myindex:frag_percent":22,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":118415640,"bucket-1:myindex:items_count":1031385,"bucket-1:myindex:num_commits":6,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":1221815,"bucket-1:myindex:num_docs_pending":11607872,"bucket-1:myindex:num_docs_processed":1162057,"bucket-1:myindex:num_docs_queued":7230071,"bucket-1:myindex:num_flush_queued":1271894,"bucket-1:myindex:num_items_flushed":1221815,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":6,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":5584158208,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"6 1034122737 189744314164652239","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"1221816 5964014454 920646677197392","bucket-1:myindex:timings/storage_info":"25 220358 3812279166","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"5 1135189 322972722985","bucket-1:myindex:timings/storage_meta_set":"6 1661448 597405504228","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"6 345821 20755108487","bucket-1:myindex:timings/storage_set":"2443630 29268935339 917190584572204661","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":8701462,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"11 47851106 1647055584353002","bucket-1:ts_queue_size":101,"indexer_state":"Active","memory_quota":31457280000,"memory_used":529563648,"memory_used_storage":468606976,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"911 65679346 8581173512558","uptime":"12m1.322929186s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T01:41:09.667-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:324943292, Disk:482791424, ExtraSnapshotData:0, Fragmentation:32.69489144861032%, mode : circular 2016-04-07T01:41:09.668-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:41:09.675-07:00 [Info] memstats {"Alloc":57280168, "TotalAlloc":129220776, "Sys":76515576, "Lookups":586, "Mallocs":1631073,"Frees":1526130, "HeapAlloc":57280168, "HeapSys":66486272, "HeapIdle":7847936, "HeapInuse":58638336,"HeapReleased":0, "HeapObjects":104943,"GCSys":2318336, "LastGC":1460018351393616308,"PauseTotalNs":14758651, "PauseNs":[], "NumGC":9} 2016-04-07T01:41:09.922-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8720000 2016-04-07T01:41:09.929-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1230000 QueuedCount 50000 2016-04-07T01:41:10.259-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8730000 2016-04-07T01:41:10.299-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8740000 2016-04-07T01:41:10.31-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8750000 2016-04-07T01:41:10.321-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8760000 2016-04-07T01:41:10.333-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8770000 2016-04-07T01:41:10.345-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8780000 2016-04-07T01:41:10.358-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8790000 2016-04-07T01:41:10.365-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1240000 QueuedCount 50000 2016-04-07T01:41:10.372-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8800000 2016-04-07T01:41:10.385-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8810000 2016-04-07T01:41:10.396-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8820000 2016-04-07T01:41:10.408-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8830000 2016-04-07T01:41:10.429-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8840000 2016-04-07T01:41:10.445-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8850000 2016-04-07T01:41:10.458-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8860000 2016-04-07T01:41:10.472-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8870000 2016-04-07T01:41:10.485-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8880000 2016-04-07T01:41:10.496-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8890000 2016-04-07T01:41:10.51-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8900000 2016-04-07T01:41:10.525-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8910000 2016-04-07T01:41:10.539-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8920000 2016-04-07T01:41:10.551-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8930000 2016-04-07T01:41:10.561-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8940000 2016-04-07T01:41:10.572-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8950000 2016-04-07T01:41:10.583-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8960000 2016-04-07T01:41:10.594-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8970000 2016-04-07T01:41:10.606-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8980000 2016-04-07T01:41:10.62-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 8990000 2016-04-07T01:41:10.633-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9000000 2016-04-07T01:41:10.644-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9010000 2016-04-07T01:41:10.657-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9020000 2016-04-07T01:41:10.67-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9030000 2016-04-07T01:41:10.681-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9040000 2016-04-07T01:41:10.695-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9050000 2016-04-07T01:41:10.708-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9060000 2016-04-07T01:41:10.719-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9070000 2016-04-07T01:41:10.732-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9080000 2016-04-07T01:41:10.744-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9090000 2016-04-07T01:41:10.759-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9100000 2016-04-07T01:41:10.773-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9110000 2016-04-07T01:41:10.775-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1250000 QueuedCount 50000 2016-04-07T01:41:10.787-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9120000 2016-04-07T01:41:10.798-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9130000 2016-04-07T01:41:10.809-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9140000 2016-04-07T01:41:10.819-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9150000 2016-04-07T01:41:10.835-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9160000 2016-04-07T01:41:10.846-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9170000 2016-04-07T01:41:10.861-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9180000 2016-04-07T01:41:10.872-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9190000 2016-04-07T01:41:10.883-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9200000 2016-04-07T01:41:10.896-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9210000 2016-04-07T01:41:10.908-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9220000 2016-04-07T01:41:10.92-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9230000 2016-04-07T01:41:10.932-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9240000 2016-04-07T01:41:10.944-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9250000 2016-04-07T01:41:10.955-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9260000 2016-04-07T01:41:10.966-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9270000 2016-04-07T01:41:11.012-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:11.115-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9280000 2016-04-07T01:41:11.162-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1260000 QueuedCount 50000 2016-04-07T01:41:11.54-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1270000 QueuedCount 49232 2016-04-07T01:41:11.894-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1280000 QueuedCount 39232 2016-04-07T01:41:12.227-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1290000 QueuedCount 29232 2016-04-07T01:41:12.566-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1300000 QueuedCount 19232 2016-04-07T01:41:12.901-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1310000 QueuedCount 9232 2016-04-07T01:41:13.416-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 28.848µs CommitTime 214.253114ms TotalFlushTime 39.399370218s TotalCommitTime 1.248375851s 2016-04-07T01:41:13.416-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 1319232, 1319232, 7 committed:true 2016-04-07T01:41:13.416-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 1319232, 1319232, 7 committed:true) SnapCreateDur 223.728823ms SnapOpenDur 83.207µs 2016-04-07T01:41:13.418-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:13.422-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:13.425-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:13.43-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:13.432-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1320000 QueuedCount 41830 2016-04-07T01:41:13.434-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9290000 2016-04-07T01:41:13.449-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9300000 2016-04-07T01:41:13.46-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9310000 2016-04-07T01:41:13.639-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9320000 2016-04-07T01:41:13.841-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1330000 QueuedCount 50000 2016-04-07T01:41:13.886-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9330000 2016-04-07T01:41:14.04-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9340000 2016-04-07T01:41:14.105-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:14.196-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9350000 2016-04-07T01:41:14.243-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1340000 QueuedCount 50000 2016-04-07T01:41:14.359-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:41:14.372-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:41:14.372-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:41:14.62-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1350000 QueuedCount 50000 2016-04-07T01:41:14.865-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:14.892-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9360000 2016-04-07T01:41:14.974-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1360000 QueuedCount 50000 2016-04-07T01:41:15.244-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:15.409-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9370000 2016-04-07T01:41:15.417-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1370000 QueuedCount 50000 2016-04-07T01:41:15.487-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:15.644-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:15.655-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9380000 2016-04-07T01:41:15.832-07:00 [Info] Indexer::ReadMemstats Time Taken 13.296394ms 2016-04-07T01:41:15.834-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1380000 QueuedCount 50000 2016-04-07T01:41:16.049-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:16.084-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9390000 2016-04-07T01:41:16.097-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9400000 2016-04-07T01:41:16.212-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1390000 QueuedCount 50000 2016-04-07T01:41:16.576-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1400000 QueuedCount 42813 2016-04-07T01:41:16.897-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1410000 QueuedCount 32813 2016-04-07T01:41:17.23-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1420000 QueuedCount 22813 2016-04-07T01:41:17.542-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1430000 QueuedCount 12813 2016-04-07T01:41:17.812-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1440000 QueuedCount 2813 2016-04-07T01:41:18.128-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.538µs CommitTime 188.511607ms TotalFlushTime 43.827171281s TotalCommitTime 1.436887458s 2016-04-07T01:41:18.128-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 1442813, 1442813, 8 committed:true 2016-04-07T01:41:18.128-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 1442813, 1442813, 8 committed:true) SnapCreateDur 197.300534ms SnapOpenDur 103.549µs 2016-04-07T01:41:18.133-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:18.139-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:18.141-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:18.143-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:18.151-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:18.167-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9410000 2016-04-07T01:41:18.178-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9420000 2016-04-07T01:41:18.189-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9430000 2016-04-07T01:41:18.201-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9440000 2016-04-07T01:41:18.211-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9450000 2016-04-07T01:41:18.397-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1450000 QueuedCount 50000 2016-04-07T01:41:18.557-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:18.775-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1460000 QueuedCount 50000 2016-04-07T01:41:18.87-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9460000 2016-04-07T01:41:19.157-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1470000 QueuedCount 50000 2016-04-07T01:41:19.236-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9470000 2016-04-07T01:41:19.379-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:41:19.395-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:41:19.395-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:41:19.532-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1480000 QueuedCount 50000 2016-04-07T01:41:19.633-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9480000 2016-04-07T01:41:19.913-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1490000 QueuedCount 50000 2016-04-07T01:41:20.029-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9490000 2016-04-07T01:41:20.243-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1500000 QueuedCount 50000 2016-04-07T01:41:20.487-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9500000 2016-04-07T01:41:20.608-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1510000 QueuedCount 50000 2016-04-07T01:41:20.763-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9510000 2016-04-07T01:41:20.833-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:20.976-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1520000 QueuedCount 50000 2016-04-07T01:41:20.979-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9520000 2016-04-07T01:41:21.335-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1530000 QueuedCount 50000 2016-04-07T01:41:21.405-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9530000 2016-04-07T01:41:21.703-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1540000 QueuedCount 50000 2016-04-07T01:41:21.802-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9540000 2016-04-07T01:41:21.816-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9550000 2016-04-07T01:41:21.826-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9560000 2016-04-07T01:41:22.069-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1550000 QueuedCount 50000 2016-04-07T01:41:22.448-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1560000 QueuedCount 50000 2016-04-07T01:41:22.458-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:41:22.827-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1570000 QueuedCount 50000 2016-04-07T01:41:22.844-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9570000 2016-04-07T01:41:23.164-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1580000 QueuedCount 50000 2016-04-07T01:41:23.331-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9580000 2016-04-07T01:41:23.553-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1590000 QueuedCount 50000 2016-04-07T01:41:23.932-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1600000 QueuedCount 50000 2016-04-07T01:41:24.239-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9590000 2016-04-07T01:41:24.306-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1610000 QueuedCount 50000 2016-04-07T01:41:24.69-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1620000 QueuedCount 50000 2016-04-07T01:41:24.696-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9600000 2016-04-07T01:41:25.005-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9610000 2016-04-07T01:41:25.072-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1630000 QueuedCount 50000 2016-04-07T01:41:25.467-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1640000 QueuedCount 50000 2016-04-07T01:41:25.58-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9620000 2016-04-07T01:41:25.861-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1650000 QueuedCount 50000 2016-04-07T01:41:25.976-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9630000 2016-04-07T01:41:26.251-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1660000 QueuedCount 50000 2016-04-07T01:41:26.492-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9640000 2016-04-07T01:41:26.578-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1670000 QueuedCount 50000 2016-04-07T01:41:26.889-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9650000 2016-04-07T01:41:26.955-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1680000 QueuedCount 50000 2016-04-07T01:41:27.196-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9660000 2016-04-07T01:41:27.32-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1690000 QueuedCount 50000 2016-04-07T01:41:27.531-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9670000 2016-04-07T01:41:27.635-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9680000 2016-04-07T01:41:27.647-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9690000 2016-04-07T01:41:27.657-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9700000 2016-04-07T01:41:27.673-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1700000 QueuedCount 50000 2016-04-07T01:41:27.895-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9710000 2016-04-07T01:41:28.024-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1710000 QueuedCount 50000 2016-04-07T01:41:28.394-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1720000 QueuedCount 50000 2016-04-07T01:41:28.761-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1730000 QueuedCount 50000 2016-04-07T01:41:29.106-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1740000 QueuedCount 41802 2016-04-07T01:41:29.412-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1750000 QueuedCount 31802 2016-04-07T01:41:29.414-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:41:29.428-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:41:29.428-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:41:29.772-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1760000 QueuedCount 21802 2016-04-07T01:41:30.095-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1770000 QueuedCount 11802 2016-04-07T01:41:30.419-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1780000 QueuedCount 1802 2016-04-07T01:41:30.734-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 16.404µs CommitTime 245.111519ms TotalFlushTime 55.874911446s TotalCommitTime 1.681998977s 2016-04-07T01:41:30.734-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 1781802, 1781802, 9 committed:true 2016-04-07T01:41:30.734-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 1781802, 1781802, 9 committed:true) SnapCreateDur 253.307832ms SnapOpenDur 62.252µs 2016-04-07T01:41:30.782-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9720000 2016-04-07T01:41:31.067-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1790000 QueuedCount 50000 2016-04-07T01:41:31.268-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9730000 2016-04-07T01:41:31.404-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1800000 QueuedCount 50000 2016-04-07T01:41:31.635-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9740000 2016-04-07T01:41:31.772-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1810000 QueuedCount 50000 2016-04-07T01:41:32.031-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9750000 2016-04-07T01:41:32.169-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1820000 QueuedCount 50000 2016-04-07T01:41:32.487-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9760000 2016-04-07T01:41:32.571-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1830000 QueuedCount 50000 2016-04-07T01:41:32.915-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9770000 2016-04-07T01:41:32.969-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1840000 QueuedCount 50000 2016-04-07T01:41:33.31-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9780000 2016-04-07T01:41:33.354-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1850000 QueuedCount 50000 2016-04-07T01:41:33.675-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9790000 2016-04-07T01:41:33.75-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1860000 QueuedCount 50000 2016-04-07T01:41:33.921-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9800000 2016-04-07T01:41:34.108-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9810000 2016-04-07T01:41:34.119-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9820000 2016-04-07T01:41:34.129-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9830000 2016-04-07T01:41:34.141-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9840000 2016-04-07T01:41:34.142-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1870000 QueuedCount 50000 2016-04-07T01:41:34.151-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9850000 2016-04-07T01:41:34.182-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9860000 2016-04-07T01:41:34.5-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1880000 QueuedCount 50000 2016-04-07T01:41:34.52-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9870000 2016-04-07T01:41:34.736-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9880000 2016-04-07T01:41:34.891-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1890000 QueuedCount 50000 2016-04-07T01:41:34.894-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9890000 2016-04-07T01:41:34.99-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9900000 2016-04-07T01:41:35.284-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1900000 QueuedCount 50000 2016-04-07T01:41:35.657-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1910000 QueuedCount 45194 2016-04-07T01:41:35.988-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1920000 QueuedCount 35194 2016-04-07T01:41:36.331-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1930000 QueuedCount 25194 2016-04-07T01:41:36.655-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1940000 QueuedCount 15194 2016-04-07T01:41:36.971-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1950000 QueuedCount 5194 2016-04-07T01:41:37.374-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 41.304µs CommitTime 252.416269ms TotalFlushTime 1m2.100433939s TotalCommitTime 1.934415246s 2016-04-07T01:41:37.374-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 1955194, 1955194, 10 committed:true 2016-04-07T01:41:37.374-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 1955194, 1955194, 10 committed:true) SnapCreateDur 260.905036ms SnapOpenDur 61.249µs 2016-04-07T01:41:37.57-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1960000 QueuedCount 50000 2016-04-07T01:41:37.903-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9910000 2016-04-07T01:41:37.961-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1970000 QueuedCount 50000 2016-04-07T01:41:38.3-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9920000 2016-04-07T01:41:38.36-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1980000 QueuedCount 50000 2016-04-07T01:41:38.638-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9930000 2016-04-07T01:41:38.709-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 1990000 QueuedCount 50000 2016-04-07T01:41:38.943-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9940000 2016-04-07T01:41:39.108-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2000000 QueuedCount 50000 2016-04-07T01:41:39.37-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9950000 2016-04-07T01:41:39.519-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2010000 QueuedCount 50000 2016-04-07T01:41:39.668-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:519883876, Disk:980201472, ExtraSnapshotData:0, Fragmentation:46.96152874171566%, mode : circular 2016-04-07T01:41:39.668-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:41:39.798-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9960000 2016-04-07T01:41:39.942-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2020000 QueuedCount 50000 2016-04-07T01:41:40.105-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9970000 2016-04-07T01:41:40.259-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9980000 2016-04-07T01:41:40.323-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2030000 QueuedCount 50000 2016-04-07T01:41:40.505-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 9990000 2016-04-07T01:41:40.6-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10000000 2016-04-07T01:41:40.739-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2040000 QueuedCount 50000 2016-04-07T01:41:40.908-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10010000 2016-04-07T01:41:41.034-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10020000 2016-04-07T01:41:41.151-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2050000 QueuedCount 50000 2016-04-07T01:41:41.25-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10030000 2016-04-07T01:41:41.261-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10040000 2016-04-07T01:41:41.273-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10050000 2016-04-07T01:41:41.283-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10060000 2016-04-07T01:41:41.556-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2060000 QueuedCount 50000 2016-04-07T01:41:41.97-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2070000 QueuedCount 49002 2016-04-07T01:41:42.271-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2080000 QueuedCount 39002 2016-04-07T01:41:42.631-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2090000 QueuedCount 29002 2016-04-07T01:41:42.98-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2100000 QueuedCount 19002 2016-04-07T01:41:43.317-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2110000 QueuedCount 9002 2016-04-07T01:41:43.917-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 41.204µs CommitTime 298.831329ms TotalFlushTime 1m8.170679026s TotalCommitTime 2.233246575s 2016-04-07T01:41:43.917-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 2119002, 2119002, 11 committed:true 2016-04-07T01:41:43.917-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 2119002, 2119002, 11 committed:true) SnapCreateDur 307.962223ms SnapOpenDur 54.095µs 2016-04-07T01:41:43.95-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2120000 QueuedCount 50000 2016-04-07T01:41:44.284-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10070000 2016-04-07T01:41:44.355-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2130000 QueuedCount 50000 2016-04-07T01:41:44.47-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:41:44.485-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:41:44.485-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:41:44.753-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2140000 QueuedCount 50000 2016-04-07T01:41:44.801-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10080000 2016-04-07T01:41:45.159-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2150000 QueuedCount 50000 2016-04-07T01:41:45.257-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10090000 2016-04-07T01:41:45.534-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10100000 2016-04-07T01:41:45.573-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2160000 QueuedCount 50000 2016-04-07T01:41:45.78-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10110000 2016-04-07T01:41:45.933-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2170000 QueuedCount 50000 2016-04-07T01:41:46.115-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10120000 2016-04-07T01:41:46.331-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10130000 2016-04-07T01:41:46.345-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2180000 QueuedCount 50000 2016-04-07T01:41:46.697-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10140000 2016-04-07T01:41:46.767-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2190000 QueuedCount 50000 2016-04-07T01:41:46.945-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10150000 2016-04-07T01:41:47.184-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2200000 QueuedCount 50000 2016-04-07T01:41:47.372-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10160000 2016-04-07T01:41:47.595-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2210000 QueuedCount 50000 2016-04-07T01:41:47.616-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10170000 2016-04-07T01:41:47.714-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10180000 2016-04-07T01:41:47.726-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10190000 2016-04-07T01:41:47.736-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10200000 2016-04-07T01:41:47.749-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10210000 2016-04-07T01:41:47.76-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10220000 2016-04-07T01:41:47.985-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2220000 QueuedCount 50000 2016-04-07T01:41:48.372-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2230000 QueuedCount 47663 2016-04-07T01:41:48.74-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2240000 QueuedCount 37663 2016-04-07T01:41:49.046-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2250000 QueuedCount 27663 2016-04-07T01:41:49.399-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2260000 QueuedCount 17663 2016-04-07T01:41:49.488-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:41:49.502-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:41:49.502-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:41:49.772-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2270000 QueuedCount 7663 2016-04-07T01:41:50.359-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 35.081µs CommitTime 294.631477ms TotalFlushTime 1m14.162097239s TotalCommitTime 2.527878052s 2016-04-07T01:41:50.359-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 2277663, 2277663, 12 committed:true 2016-04-07T01:41:50.36-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 2277663, 2277663, 12 committed:true) SnapCreateDur 303.2678ms SnapOpenDur 96.343µs 2016-04-07T01:41:50.482-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2280000 QueuedCount 50000 2016-04-07T01:41:50.848-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10230000 2016-04-07T01:41:50.897-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2290000 QueuedCount 50000 2016-04-07T01:41:51.262-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2300000 QueuedCount 50000 2016-04-07T01:41:51.335-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10240000 2016-04-07T01:41:51.692-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2310000 QueuedCount 50000 2016-04-07T01:41:51.851-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10250000 2016-04-07T01:41:52.087-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2320000 QueuedCount 50000 2016-04-07T01:41:52.189-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10260000 2016-04-07T01:41:52.458-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2330000 QueuedCount 50000 2016-04-07T01:41:52.555-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10270000 2016-04-07T01:41:52.847-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2340000 QueuedCount 50000 2016-04-07T01:41:52.952-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10280000 2016-04-07T01:41:53.239-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2350000 QueuedCount 50000 2016-04-07T01:41:53.286-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10290000 2016-04-07T01:41:53.647-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2360000 QueuedCount 50000 2016-04-07T01:41:53.652-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10300000 2016-04-07T01:41:53.901-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10310000 2016-04-07T01:41:54.045-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2370000 QueuedCount 50000 2016-04-07T01:41:54.086-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10320000 2016-04-07T01:41:54.242-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10330000 2016-04-07T01:41:54.391-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2380000 QueuedCount 50000 2016-04-07T01:41:54.488-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10340000 2016-04-07T01:41:54.507-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:41:54.521-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:41:54.521-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:41:54.647-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10350000 2016-04-07T01:41:54.805-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2390000 QueuedCount 50000 2016-04-07T01:41:54.83-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10360000 2016-04-07T01:41:54.904-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10370000 2016-04-07T01:41:54.916-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10380000 2016-04-07T01:41:55.175-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2400000 QueuedCount 47225 2016-04-07T01:41:55.522-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2410000 QueuedCount 37225 2016-04-07T01:41:55.886-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2420000 QueuedCount 27225 2016-04-07T01:41:56.23-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2430000 QueuedCount 17225 2016-04-07T01:41:56.581-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2440000 QueuedCount 7225 2016-04-07T01:41:57.151-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 30.21µs CommitTime 325.42027ms TotalFlushTime 1m20.448364696s TotalCommitTime 2.853298322s 2016-04-07T01:41:57.151-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 2447225, 2447225, 13 committed:true 2016-04-07T01:41:57.151-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 2447225, 2447225, 13 committed:true) SnapCreateDur 334.779807ms SnapOpenDur 75.552µs 2016-04-07T01:41:57.277-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2450000 QueuedCount 50000 2016-04-07T01:41:57.283-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10390000 2016-04-07T01:41:57.723-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2460000 QueuedCount 50000 2016-04-07T01:41:57.831-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10400000 2016-04-07T01:41:58.157-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2470000 QueuedCount 50000 2016-04-07T01:41:58.438-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10410000 2016-04-07T01:41:58.507-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2480000 QueuedCount 50000 2016-04-07T01:41:58.865-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10420000 2016-04-07T01:41:58.926-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2490000 QueuedCount 50000 2016-04-07T01:41:59.352-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10430000 2016-04-07T01:41:59.356-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2500000 QueuedCount 50000 2016-04-07T01:41:59.718-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10440000 2016-04-07T01:41:59.774-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2510000 QueuedCount 50000 2016-04-07T01:42:00.025-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10450000 2016-04-07T01:42:00.204-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2520000 QueuedCount 50000 2016-04-07T01:42:00.39-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10460000 2016-04-07T01:42:00.612-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2530000 QueuedCount 50000 2016-04-07T01:42:00.726-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10470000 2016-04-07T01:42:00.973-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10480000 2016-04-07T01:42:01.022-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2540000 QueuedCount 50000 2016-04-07T01:42:01.278-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10490000 2016-04-07T01:42:01.365-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10500000 2016-04-07T01:42:01.391-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10510000 2016-04-07T01:42:01.418-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10520000 2016-04-07T01:42:01.453-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10530000 2016-04-07T01:42:01.55-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2550000 QueuedCount 49701 2016-04-07T01:42:01.648-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10540000 2016-04-07T01:42:01.673-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10550000 2016-04-07T01:42:01.699-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10560000 2016-04-07T01:42:01.732-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10570000 2016-04-07T01:42:02.182-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2560000 QueuedCount 50000 2016-04-07T01:42:02.544-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2570000 QueuedCount 50000 2016-04-07T01:42:02.96-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2580000 QueuedCount 43745 2016-04-07T01:42:03.36-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2590000 QueuedCount 33745 2016-04-07T01:42:03.756-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2600000 QueuedCount 23745 2016-04-07T01:42:04.18-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2610000 QueuedCount 13745 2016-04-07T01:42:04.551-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:42:04.565-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:42:04.565-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:42:04.607-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2620000 QueuedCount 3745 2016-04-07T01:42:05.141-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 45.106µs CommitTime 359.898001ms TotalFlushTime 1m27.892203908s TotalCommitTime 3.213196323s 2016-04-07T01:42:05.141-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 2623745, 2623745, 14 committed:true 2016-04-07T01:42:05.141-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 2623745, 2623745, 14 committed:true) SnapCreateDur 368.045982ms SnapOpenDur 88.143µs 2016-04-07T01:42:05.351-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10580000 2016-04-07T01:42:05.407-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2630000 QueuedCount 50000 2016-04-07T01:42:05.763-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2640000 QueuedCount 50000 2016-04-07T01:42:05.928-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10590000 2016-04-07T01:42:06.207-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2650000 QueuedCount 50000 2016-04-07T01:42:06.385-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10600000 2016-04-07T01:42:06.624-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2660000 QueuedCount 50000 2016-04-07T01:42:06.872-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10610000 2016-04-07T01:42:07.043-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2670000 QueuedCount 50000 2016-04-07T01:42:07.299-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10620000 2016-04-07T01:42:07.461-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2680000 QueuedCount 50000 2016-04-07T01:42:07.726-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10630000 2016-04-07T01:42:07.878-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2690000 QueuedCount 50000 2016-04-07T01:42:08.062-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10640000 2016-04-07T01:42:08.293-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2700000 QueuedCount 50000 2016-04-07T01:42:08.4-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10650000 2016-04-07T01:42:08.499-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10660000 2016-04-07T01:42:08.56-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10670000 2016-04-07T01:42:08.718-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2710000 QueuedCount 50000 2016-04-07T01:42:08.84-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10680000 2016-04-07T01:42:08.854-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10690000 2016-04-07T01:42:08.866-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10700000 2016-04-07T01:42:09.015-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10710000 2016-04-07T01:42:09.097-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2720000 QueuedCount 50000 2016-04-07T01:42:09.262-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10720000 2016-04-07T01:42:09.373-07:00 [Info] connected with 1 indexers 2016-04-07T01:42:09.373-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T01:42:09.374-07:00 [Info] client load stats {} 2016-04-07T01:42:09.511-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2730000 QueuedCount 50000 2016-04-07T01:42:09.575-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:42:09.588-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:42:09.588-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:42:09.632-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":7934723,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":7391071645,"bucket-1:myindex:avg_ts_items_count":173559,"bucket-1:myindex:build_progress":13,"bucket-1:myindex:data_size":683035687,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":1612050432,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":50044,"bucket-1:myindex:frag_percent":57,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":280225217,"bucket-1:myindex:items_count":2623745,"bucket-1:myindex:num_commits":14,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":2732315,"bucket-1:myindex:num_docs_pending":9386977,"bucket-1:myindex:num_docs_processed":2623745,"bucket-1:myindex:num_docs_queued":7989278,"bucket-1:myindex:num_flush_queued":2782359,"bucket-1:myindex:num_items_flushed":2732315,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":14,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":7989982123,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"14 3213196323 806511760281216077","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"2732316 14727934319 1011587110343187","bucket-1:myindex:timings/storage_info":"57 468793 6971525299","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"13 4497338 1795616331000","bucket-1:myindex:timings/storage_meta_set":"14 4689179 1802386263227","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"14 786574 46068733638","bucket-1:myindex:timings/storage_set":"5464630 76042322313 2989253433884019725","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":10717039,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"23 75485361 2016747916720547","bucket-1:ts_queue_size":281,"indexer_state":"Active","memory_quota":31457280000,"memory_used":10707675136,"memory_used_storage":1742671872,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"1054 96311927 15359304274501","uptime":"13m1.324122011s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T01:42:09.669-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:698000617, Disk:1612050432, ExtraSnapshotData:0, Fragmentation:56.70106820826806%, mode : circular 2016-04-07T01:42:09.669-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:42:09.676-07:00 [Info] memstats {"Alloc":8645574456, "TotalAlloc":11072244352, "Sys":9127152256, "Lookups":954, "Mallocs":275820631,"Frees":87921559, "HeapAlloc":8645574456, "HeapSys":8692826112, "HeapIdle":1048576, "HeapInuse":8691777536,"HeapReleased":0, "HeapObjects":187899072,"GCSys":273225728, "LastGC":1460018454401908589,"PauseTotalNs":21186687, "PauseNs":[808090, 958454, 554148, 719439, 657045, 1245341, 1485519], "NumGC":16} 2016-04-07T01:42:09.956-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2740000 QueuedCount 50000 2016-04-07T01:42:10.373-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2750000 QueuedCount 46509 2016-04-07T01:42:10.75-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2760000 QueuedCount 36509 2016-04-07T01:42:11.149-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2770000 QueuedCount 26509 2016-04-07T01:42:11.531-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2780000 QueuedCount 16509 2016-04-07T01:42:11.916-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2790000 QueuedCount 6509 2016-04-07T01:42:12.55-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 37.257µs CommitTime 386.458097ms TotalFlushTime 1m34.762074293s TotalCommitTime 3.59965442s 2016-04-07T01:42:12.55-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 2796509, 2796509, 15 committed:true 2016-04-07T01:42:12.55-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 2796509, 2796509, 15 committed:true) SnapCreateDur 394.41914ms SnapOpenDur 70.015µs 2016-04-07T01:42:12.677-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2800000 QueuedCount 50000 2016-04-07T01:42:12.688-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10730000 2016-04-07T01:42:13.091-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2810000 QueuedCount 50000 2016-04-07T01:42:13.175-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10740000 2016-04-07T01:42:13.453-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10750000 2016-04-07T01:42:13.509-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2820000 QueuedCount 50000 2016-04-07T01:42:13.848-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10760000 2016-04-07T01:42:13.938-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2830000 QueuedCount 50000 2016-04-07T01:42:14.215-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10770000 2016-04-07T01:42:14.371-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2840000 QueuedCount 50000 2016-04-07T01:42:14.553-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10780000 2016-04-07T01:42:14.797-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2850000 QueuedCount 50000 2016-04-07T01:42:14.979-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10790000 2016-04-07T01:42:15.154-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2860000 QueuedCount 50000 2016-04-07T01:42:15.406-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10800000 2016-04-07T01:42:15.583-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2870000 QueuedCount 50000 2016-04-07T01:42:15.593-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10810000 2016-04-07T01:42:15.867-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10820000 2016-04-07T01:42:16.008-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2880000 QueuedCount 50000 2016-04-07T01:42:16.206-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10830000 2016-04-07T01:42:16.219-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10840000 2016-04-07T01:42:16.23-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10850000 2016-04-07T01:42:16.242-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10860000 2016-04-07T01:42:16.256-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10870000 2016-04-07T01:42:16.267-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10880000 2016-04-07T01:42:16.438-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2890000 QueuedCount 50000 2016-04-07T01:42:16.445-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10890000 2016-04-07T01:42:16.632-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10900000 2016-04-07T01:42:16.878-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2900000 QueuedCount 50000 2016-04-07T01:42:16.879-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10910000 2016-04-07T01:42:17.01-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10920000 2016-04-07T01:42:17.023-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10930000 2016-04-07T01:42:17.033-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10940000 2016-04-07T01:42:17.227-07:00 [Info] Indexer::ReadMemstats Time Taken 15.622068ms 2016-04-07T01:42:17.327-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2910000 QueuedCount 50000 2016-04-07T01:42:17.73-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2920000 QueuedCount 40354 2016-04-07T01:42:18.12-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2930000 QueuedCount 30354 2016-04-07T01:42:18.522-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2940000 QueuedCount 20354 2016-04-07T01:42:18.849-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2950000 QueuedCount 10354 2016-04-07T01:42:19.226-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2960000 QueuedCount 354 2016-04-07T01:42:19.669-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 33.402µs CommitTime 427.631751ms TotalFlushTime 1m41.301272906s TotalCommitTime 4.027286171s 2016-04-07T01:42:19.669-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 2960354, 2960354, 16 committed:true 2016-04-07T01:42:19.669-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 2960354, 2960354, 16 committed:true) SnapCreateDur 437.533429ms SnapOpenDur 60.371µs 2016-04-07T01:42:19.731-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10950000 2016-04-07T01:42:20.048-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2970000 QueuedCount 50000 2016-04-07T01:42:20.158-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10960000 2016-04-07T01:42:20.475-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2980000 QueuedCount 50000 2016-04-07T01:42:20.825-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10970000 2016-04-07T01:42:20.888-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 2990000 QueuedCount 50000 2016-04-07T01:42:21.161-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10980000 2016-04-07T01:42:21.3-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3000000 QueuedCount 50000 2016-04-07T01:42:21.497-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 10990000 2016-04-07T01:42:21.72-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3010000 QueuedCount 50000 2016-04-07T01:42:21.894-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11000000 2016-04-07T01:42:22.137-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3020000 QueuedCount 50000 2016-04-07T01:42:22.23-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11010000 2016-04-07T01:42:22.484-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3030000 QueuedCount 50000 2016-04-07T01:42:22.569-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11020000 2016-04-07T01:42:22.899-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3040000 QueuedCount 50000 2016-04-07T01:42:22.994-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11030000 2016-04-07T01:42:23.182-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11040000 2016-04-07T01:42:23.343-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11050000 2016-04-07T01:42:23.343-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3050000 QueuedCount 50000 2016-04-07T01:42:23.357-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11060000 2016-04-07T01:42:23.368-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11070000 2016-04-07T01:42:23.379-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11080000 2016-04-07T01:42:23.389-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11090000 2016-04-07T01:42:23.775-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3060000 QueuedCount 50000 2016-04-07T01:42:24.209-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3070000 QueuedCount 50000 2016-04-07T01:42:24.626-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:42:24.628-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3080000 QueuedCount 48884 2016-04-07T01:42:24.639-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:42:24.639-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:42:25.027-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3090000 QueuedCount 38884 2016-04-07T01:42:25.422-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3100000 QueuedCount 28884 2016-04-07T01:42:25.814-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3110000 QueuedCount 18884 2016-04-07T01:42:26.151-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3120000 QueuedCount 8884 2016-04-07T01:42:26.926-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.77µs CommitTime 386.458439ms TotalFlushTime 1m48.007233071s TotalCommitTime 4.41374461s 2016-04-07T01:42:26.926-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 3128884, 3128884, 17 committed:true 2016-04-07T01:42:26.926-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 3128884, 3128884, 17 committed:true) SnapCreateDur 394.416463ms SnapOpenDur 55.127µs 2016-04-07T01:42:26.962-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3130000 QueuedCount 50000 2016-04-07T01:42:26.963-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11100000 2016-04-07T01:42:26.975-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11110000 2016-04-07T01:42:27.212-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11120000 2016-04-07T01:42:27.356-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3140000 QueuedCount 50000 2016-04-07T01:42:27.774-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3150000 QueuedCount 50000 2016-04-07T01:42:27.788-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11130000 2016-04-07T01:42:28.202-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3160000 QueuedCount 50000 2016-04-07T01:42:28.306-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11140000 2016-04-07T01:42:28.634-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3170000 QueuedCount 50000 2016-04-07T01:42:28.673-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11150000 2016-04-07T01:42:28.986-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3180000 QueuedCount 50000 2016-04-07T01:42:29.008-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11160000 2016-04-07T01:42:29.376-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11170000 2016-04-07T01:42:29.393-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3190000 QueuedCount 50000 2016-04-07T01:42:29.644-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:42:29.653-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11180000 2016-04-07T01:42:29.659-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:42:29.659-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:42:29.822-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3200000 QueuedCount 50000 2016-04-07T01:42:30.079-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11190000 2016-04-07T01:42:30.264-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3210000 QueuedCount 50000 2016-04-07T01:42:30.386-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11200000 2016-04-07T01:42:30.634-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11210000 2016-04-07T01:42:30.706-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3220000 QueuedCount 50000 2016-04-07T01:42:30.737-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11220000 2016-04-07T01:42:30.974-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11230000 2016-04-07T01:42:31.074-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11240000 2016-04-07T01:42:31.146-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3230000 QueuedCount 50000 2016-04-07T01:42:31.173-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11250000 2016-04-07T01:42:31.183-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11260000 2016-04-07T01:42:31.193-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11270000 2016-04-07T01:42:31.586-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3240000 QueuedCount 50000 2016-04-07T01:42:31.964-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3250000 QueuedCount 40412 2016-04-07T01:42:32.33-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3260000 QueuedCount 30412 2016-04-07T01:42:32.655-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3270000 QueuedCount 20412 2016-04-07T01:42:33.022-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3280000 QueuedCount 10412 2016-04-07T01:42:33.381-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3290000 QueuedCount 412 2016-04-07T01:42:33.798-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.954µs CommitTime 397.493004ms TotalFlushTime 1m54.315525419s TotalCommitTime 4.811237614s 2016-04-07T01:42:33.798-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 3290412, 3290412, 18 committed:true 2016-04-07T01:42:33.798-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 3290412, 3290412, 18 committed:true) SnapCreateDur 406.057943ms SnapOpenDur 85.09µs 2016-04-07T01:42:33.834-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11280000 2016-04-07T01:42:34.196-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3300000 QueuedCount 50000 2016-04-07T01:42:34.383-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11290000 2016-04-07T01:42:34.618-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3310000 QueuedCount 50000 2016-04-07T01:42:35.046-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3320000 QueuedCount 50000 2016-04-07T01:42:35.049-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11300000 2016-04-07T01:42:35.486-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3330000 QueuedCount 50000 2016-04-07T01:42:35.509-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11310000 2016-04-07T01:42:35.875-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11320000 2016-04-07T01:42:35.947-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3340000 QueuedCount 50000 2016-04-07T01:42:36.333-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11330000 2016-04-07T01:42:36.391-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3350000 QueuedCount 50000 2016-04-07T01:42:36.772-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3360000 QueuedCount 50000 2016-04-07T01:42:36.79-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11340000 2016-04-07T01:42:37.218-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3370000 QueuedCount 50000 2016-04-07T01:42:37.398-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11350000 2016-04-07T01:42:37.586-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11360000 2016-04-07T01:42:37.658-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3380000 QueuedCount 50000 2016-04-07T01:42:37.861-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11370000 2016-04-07T01:42:38.113-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3390000 QueuedCount 50000 2016-04-07T01:42:38.136-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11380000 2016-04-07T01:42:38.323-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11390000 2016-04-07T01:42:38.396-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11400000 2016-04-07T01:42:38.407-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11410000 2016-04-07T01:42:38.417-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11420000 2016-04-07T01:42:38.427-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11430000 2016-04-07T01:42:38.567-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3400000 QueuedCount 50000 2016-04-07T01:42:38.989-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3410000 QueuedCount 47296 2016-04-07T01:42:39.362-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3420000 QueuedCount 37296 2016-04-07T01:42:39.67-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:872550096, Disk:2328555520, ExtraSnapshotData:0, Fragmentation:62.52826748146422%, mode : circular 2016-04-07T01:42:39.67-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:42:39.685-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:42:39.699-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:42:39.699-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:42:39.75-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3430000 QueuedCount 27296 2016-04-07T01:42:40.127-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3440000 QueuedCount 17296 2016-04-07T01:42:40.439-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3450000 QueuedCount 7296 2016-04-07T01:42:41.112-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.156µs CommitTime 376.135817ms TotalFlushTime 2m1.075480122s TotalCommitTime 5.187373431s 2016-04-07T01:42:41.112-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 3457296, 3457296, 19 committed:true 2016-04-07T01:42:41.112-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 3457296, 3457296, 19 committed:true) SnapCreateDur 385.250678ms SnapOpenDur 52.616µs 2016-04-07T01:42:41.222-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3460000 QueuedCount 50000 2016-04-07T01:42:41.622-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3470000 QueuedCount 50000 2016-04-07T01:42:41.636-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11440000 2016-04-07T01:42:42.018-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3480000 QueuedCount 50000 2016-04-07T01:42:42.064-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11450000 2016-04-07T01:42:42.403-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3490000 QueuedCount 50000 2016-04-07T01:42:42.583-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11460000 2016-04-07T01:42:42.841-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3500000 QueuedCount 50000 2016-04-07T01:42:42.949-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11470000 2016-04-07T01:42:43.267-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3510000 QueuedCount 50000 2016-04-07T01:42:43.467-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11480000 2016-04-07T01:42:43.707-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3520000 QueuedCount 50000 2016-04-07T01:42:43.985-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11490000 2016-04-07T01:42:44.168-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3530000 QueuedCount 50000 2016-04-07T01:42:44.202-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11500000 2016-04-07T01:42:44.358-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11510000 2016-04-07T01:42:44.619-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3540000 QueuedCount 50000 2016-04-07T01:42:44.702-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:42:44.715-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:42:44.716-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:42:44.724-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11520000 2016-04-07T01:42:45.067-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3550000 QueuedCount 50000 2016-04-07T01:42:45.508-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3560000 QueuedCount 50000 2016-04-07T01:42:45.963-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3570000 QueuedCount 49997 2016-04-07T01:42:46.309-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3580000 QueuedCount 40274 2016-04-07T01:42:46.683-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3590000 QueuedCount 30274 2016-04-07T01:42:47.07-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3600000 QueuedCount 20274 2016-04-07T01:42:47.442-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3610000 QueuedCount 10274 2016-04-07T01:42:47.81-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3620000 QueuedCount 274 2016-04-07T01:42:48.398-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 38.865µs CommitTime 573.121881ms TotalFlushTime 2m7.622268563s TotalCommitTime 5.760495312s 2016-04-07T01:42:48.398-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 3620274, 3620274, 20 committed:true 2016-04-07T01:42:48.398-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 3620274, 3620274, 20 committed:true) SnapCreateDur 581.626992ms SnapOpenDur 61.046µs 2016-04-07T01:42:48.428-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11530000 2016-04-07T01:42:48.44-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11540000 2016-04-07T01:42:48.453-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11550000 2016-04-07T01:42:48.464-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11560000 2016-04-07T01:42:48.476-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11570000 2016-04-07T01:42:48.489-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11580000 2016-04-07T01:42:48.501-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11590000 2016-04-07T01:42:48.514-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11600000 2016-04-07T01:42:48.526-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11610000 2016-04-07T01:42:48.536-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11620000 2016-04-07T01:42:48.546-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11630000 2016-04-07T01:42:48.556-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11640000 2016-04-07T01:42:48.566-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11650000 2016-04-07T01:42:48.656-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11660000 2016-04-07T01:42:48.729-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3630000 QueuedCount 50000 2016-04-07T01:42:49.121-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3640000 QueuedCount 50000 2016-04-07T01:42:49.203-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11670000 2016-04-07T01:42:49.512-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3650000 QueuedCount 50000 2016-04-07T01:42:49.629-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11680000 2016-04-07T01:42:49.938-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3660000 QueuedCount 50000 2016-04-07T01:42:50.116-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11690000 2016-04-07T01:42:50.349-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3670000 QueuedCount 50000 2016-04-07T01:42:50.574-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11700000 2016-04-07T01:42:50.79-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3680000 QueuedCount 50000 2016-04-07T01:42:51.24-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3690000 QueuedCount 50000 2016-04-07T01:42:51.616-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3700000 QueuedCount 50000 2016-04-07T01:42:52.056-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3710000 QueuedCount 50000 2016-04-07T01:42:52.499-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3720000 QueuedCount 50000 2016-04-07T01:42:52.944-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3730000 QueuedCount 50000 2016-04-07T01:42:53.357-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3740000 QueuedCount 42169 2016-04-07T01:42:53.737-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3750000 QueuedCount 32169 2016-04-07T01:42:54.136-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3760000 QueuedCount 22169 2016-04-07T01:42:54.506-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3770000 QueuedCount 12169 2016-04-07T01:42:54.739-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:42:54.753-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:42:54.753-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:42:54.84-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3780000 QueuedCount 2169 2016-04-07T01:42:55.376-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.275µs CommitTime 389.765794ms TotalFlushTime 2m14.041573955s TotalCommitTime 6.150261106s 2016-04-07T01:42:55.376-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 3782169, 3782169, 21 committed:true 2016-04-07T01:42:55.376-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 3782169, 3782169, 21 committed:true) SnapCreateDur 398.05054ms SnapOpenDur 56.531µs 2016-04-07T01:42:55.41-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11710000 2016-04-07T01:42:55.611-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3790000 QueuedCount 50000 2016-04-07T01:42:55.689-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11720000 2016-04-07T01:42:55.991-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3800000 QueuedCount 50000 2016-04-07T01:42:56.375-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3810000 QueuedCount 50000 2016-04-07T01:42:56.384-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11730000 2016-04-07T01:42:56.767-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3820000 QueuedCount 50000 2016-04-07T01:42:56.782-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11740000 2016-04-07T01:42:57.178-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3830000 QueuedCount 50000 2016-04-07T01:42:57.299-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11750000 2016-04-07T01:42:57.613-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3840000 QueuedCount 50000 2016-04-07T01:42:57.727-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11760000 2016-04-07T01:42:58.04-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3850000 QueuedCount 50000 2016-04-07T01:42:58.153-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11770000 2016-04-07T01:42:58.407-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3860000 QueuedCount 50000 2016-04-07T01:42:58.428-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11780000 2016-04-07T01:42:58.767-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11790000 2016-04-07T01:42:58.847-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3870000 QueuedCount 50000 2016-04-07T01:42:59.101-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11800000 2016-04-07T01:42:59.274-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3880000 QueuedCount 50000 2016-04-07T01:42:59.289-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11810000 2016-04-07T01:42:59.537-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11820000 2016-04-07T01:42:59.548-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11830000 2016-04-07T01:42:59.701-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3890000 QueuedCount 50000 2016-04-07T01:43:00.12-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3900000 QueuedCount 48536 2016-04-07T01:43:00.497-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3910000 QueuedCount 38536 2016-04-07T01:43:00.862-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3920000 QueuedCount 28536 2016-04-07T01:43:01.233-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3930000 QueuedCount 18536 2016-04-07T01:43:01.602-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3940000 QueuedCount 8536 2016-04-07T01:43:02.275-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 53.634µs CommitTime 356.01299ms TotalFlushTime 2m20.424177458s TotalCommitTime 6.506274096s 2016-04-07T01:43:02.275-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 3948536, 3948536, 22 committed:true 2016-04-07T01:43:02.275-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 3948536, 3948536, 22 committed:true) SnapCreateDur 364.907254ms SnapOpenDur 45.33µs 2016-04-07T01:43:02.313-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3950000 QueuedCount 50000 2016-04-07T01:43:02.682-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3960000 QueuedCount 50000 2016-04-07T01:43:02.697-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11840000 2016-04-07T01:43:03.065-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3970000 QueuedCount 50000 2016-04-07T01:43:03.153-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11850000 2016-04-07T01:43:03.489-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3980000 QueuedCount 50000 2016-04-07T01:43:03.522-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11860000 2016-04-07T01:43:03.902-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 3990000 QueuedCount 50000 2016-04-07T01:43:03.919-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11870000 2016-04-07T01:43:04.353-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4000000 QueuedCount 50000 2016-04-07T01:43:04.376-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11880000 2016-04-07T01:43:04.743-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11890000 2016-04-07T01:43:04.778-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:43:04.786-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4010000 QueuedCount 50000 2016-04-07T01:43:04.793-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:43:04.793-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:43:05.05-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11900000 2016-04-07T01:43:05.145-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4020000 QueuedCount 50000 2016-04-07T01:43:05.327-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11910000 2016-04-07T01:43:05.584-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4030000 QueuedCount 50000 2016-04-07T01:43:05.603-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11920000 2016-04-07T01:43:05.85-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11930000 2016-04-07T01:43:06.017-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4040000 QueuedCount 50000 2016-04-07T01:43:06.126-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11940000 2016-04-07T01:43:06.283-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11950000 2016-04-07T01:43:06.461-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4050000 QueuedCount 50000 2016-04-07T01:43:06.559-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11960000 2016-04-07T01:43:06.661-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11970000 2016-04-07T01:43:06.673-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11980000 2016-04-07T01:43:06.684-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 11990000 2016-04-07T01:43:06.696-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12000000 2016-04-07T01:43:06.708-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12010000 2016-04-07T01:43:06.882-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4060000 QueuedCount 50000 2016-04-07T01:43:07.296-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4070000 QueuedCount 50000 2016-04-07T01:43:07.703-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4080000 QueuedCount 47321 2016-04-07T01:43:08.087-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4090000 QueuedCount 37321 2016-04-07T01:43:08.42-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4100000 QueuedCount 27321 2016-04-07T01:43:08.791-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4110000 QueuedCount 17321 2016-04-07T01:43:09.158-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4120000 QueuedCount 7321 2016-04-07T01:43:09.373-07:00 [Info] connected with 1 indexers 2016-04-07T01:43:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T01:43:09.374-07:00 [Info] client load stats {} 2016-04-07T01:43:09.632-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":7882058,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":7003670208,"bucket-1:myindex:avg_ts_items_count":164773,"bucket-1:myindex:build_progress":19,"bucket-1:myindex:data_size":1046349976,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":2737967104,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":0,"bucket-1:myindex:frag_percent":61,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":428434335,"bucket-1:myindex:items_count":3948536,"bucket-1:myindex:num_commits":22,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":4127321,"bucket-1:myindex:num_docs_pending":7980619,"bucket-1:myindex:num_docs_processed":3948536,"bucket-1:myindex:num_docs_queued":8070845,"bucket-1:myindex:num_flush_queued":4127321,"bucket-1:myindex:num_items_flushed":4127321,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":22,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":6899441763,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"22 6506274096 2194690815317090410","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"4127321 22746812664 1062391338718380","bucket-1:myindex:timings/storage_info":"92 766156 11768530212","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"22 7536664 2910824531246","bucket-1:myindex:timings/storage_meta_set":"23 7830474 2940628126314","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"22 1145076 63158382970","bucket-1:myindex:timings/storage_set":"8254642 122454625852 5240504388302695428","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":12009379,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"35 85330733 2024845940033921","bucket-1:ts_queue_size":439,"indexer_state":"Active","memory_quota":31457280000,"memory_used":10627844096,"memory_used_storage":2738417664,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"1196 125461490 21502736519946","uptime":"14m1.325480033s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T01:43:09.67-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:1060819539, Disk:2737967104, ExtraSnapshotData:0, Fragmentation:61.255212400097555%, mode : circular 2016-04-07T01:43:09.67-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:43:09.676-07:00 [Info] memstats {"Alloc":6194658248, "TotalAlloc":11956951352, "Sys":9853772104, "Lookups":1403, "Mallocs":303149343,"Frees":170339399, "HeapAlloc":6194658248, "HeapSys":9385836544, "HeapIdle":1792745472, "HeapInuse":7593091072,"HeapReleased":0, "HeapObjects":132809944,"GCSys":296335360, "LastGC":1460018522138384238,"PauseTotalNs":22534959, "PauseNs":[1348272], "NumGC":17} 2016-04-07T01:43:09.853-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.096µs CommitTime 394.742464ms TotalFlushTime 2m27.436736367s TotalCommitTime 6.90101656s 2016-04-07T01:43:09.853-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 4127321, 4127321, 23 committed:true 2016-04-07T01:43:09.853-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 4127321, 4127321, 23 committed:true) SnapCreateDur 403.105681ms SnapOpenDur 76.255µs 2016-04-07T01:43:09.889-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12020000 2016-04-07T01:43:09.904-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12030000 2016-04-07T01:43:09.915-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12040000 2016-04-07T01:43:09.952-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4130000 QueuedCount 30162 2016-04-07T01:43:10.31-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4140000 QueuedCount 20162 2016-04-07T01:43:10.694-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4150000 QueuedCount 10162 2016-04-07T01:43:11.021-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4160000 QueuedCount 162 2016-04-07T01:43:11.343-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.824µs CommitTime 253.916729ms TotalFlushTime 2m28.635434024s TotalCommitTime 7.154933289s 2016-04-07T01:43:11.343-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 4160162, 4160162, 24 committed:true 2016-04-07T01:43:11.343-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 4160162, 4160162, 24 committed:true) SnapCreateDur 262.917905ms SnapOpenDur 46.121µs 2016-04-07T01:43:11.346-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:43:11.348-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:43:11.354-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:43:11.356-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:43:11.36-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:43:11.363-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:43:11.367-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:43:11.374-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:43:11.386-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12050000 2016-04-07T01:43:11.398-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12060000 2016-04-07T01:43:11.408-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12070000 2016-04-07T01:43:11.672-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4170000 QueuedCount 50000 2016-04-07T01:43:11.812-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:43:11.829-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12080000 2016-04-07T01:43:12.042-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4180000 QueuedCount 50000 2016-04-07T01:43:12.196-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12090000 2016-04-07T01:43:12.265-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:43:12.432-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4190000 QueuedCount 50000 2016-04-07T01:43:12.756-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:43:12.849-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4200000 QueuedCount 50000 2016-04-07T01:43:12.862-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12100000 2016-04-07T01:43:13.17-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12110000 2016-04-07T01:43:13.256-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4210000 QueuedCount 50000 2016-04-07T01:43:13.681-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4220000 QueuedCount 50000 2016-04-07T01:43:14.025-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:43:14.094-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4230000 QueuedCount 50000 2016-04-07T01:43:14.107-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12120000 2016-04-07T01:43:14.447-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:43:14.465-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4240000 QueuedCount 50000 2016-04-07T01:43:14.476-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12130000 2016-04-07T01:43:14.816-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:43:14.83-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:43:14.83-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:43:14.924-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4250000 QueuedCount 50000 2016-04-07T01:43:14.932-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12140000 2016-04-07T01:43:15.007-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:43:15.376-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4260000 QueuedCount 50000 2016-04-07T01:43:15.8-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4270000 QueuedCount 40470 2016-04-07T01:43:16.22-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4280000 QueuedCount 30470 2016-04-07T01:43:16.613-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4290000 QueuedCount 20470 2016-04-07T01:43:17.001-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4300000 QueuedCount 10470 2016-04-07T01:43:17.39-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4310000 QueuedCount 470 2016-04-07T01:43:17.796-07:00 [Info] Indexer::ReadMemstats Time Taken 17.241245ms 2016-04-07T01:43:17.827-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 35.675µs CommitTime 415.990034ms TotalFlushTime 2m34.558398889s TotalCommitTime 7.570923323s 2016-04-07T01:43:17.827-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 4310470, 4310470, 25 committed:true 2016-04-07T01:43:17.827-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 4310470, 4310470, 25 committed:true) SnapCreateDur 424.52071ms SnapOpenDur 70.774µs 2016-04-07T01:43:17.845-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:43:17.858-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12150000 2016-04-07T01:43:17.871-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12160000 2016-04-07T01:43:17.881-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12170000 2016-04-07T01:43:18.183-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4320000 QueuedCount 50000 2016-04-07T01:43:18.271-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12180000 2016-04-07T01:43:18.59-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4330000 QueuedCount 50000 2016-04-07T01:43:18.849-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12190000 2016-04-07T01:43:19.005-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4340000 QueuedCount 50000 2016-04-07T01:43:19.366-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12200000 2016-04-07T01:43:19.419-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4350000 QueuedCount 50000 2016-04-07T01:43:19.862-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4360000 QueuedCount 50000 2016-04-07T01:43:19.975-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12210000 2016-04-07T01:43:20.304-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4370000 QueuedCount 50000 2016-04-07T01:43:20.613-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12220000 2016-04-07T01:43:20.693-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4380000 QueuedCount 50000 2016-04-07T01:43:20.951-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12230000 2016-04-07T01:43:21.126-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4390000 QueuedCount 50000 2016-04-07T01:43:21.467-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12240000 2016-04-07T01:43:21.561-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4400000 QueuedCount 50000 2016-04-07T01:43:21.775-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12250000 2016-04-07T01:43:22.027-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4410000 QueuedCount 50000 2016-04-07T01:43:22.232-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12260000 2016-04-07T01:43:22.481-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4420000 QueuedCount 50000 2016-04-07T01:43:22.599-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12270000 2016-04-07T01:43:22.61-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12280000 2016-04-07T01:43:22.62-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12290000 2016-04-07T01:43:22.631-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12300000 2016-04-07T01:43:22.642-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12310000 2016-04-07T01:43:22.654-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12320000 2016-04-07T01:43:22.862-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12330000 2016-04-07T01:43:22.92-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4430000 QueuedCount 50000 2016-04-07T01:43:23.2-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12340000 2016-04-07T01:43:23.363-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4440000 QueuedCount 50000 2016-04-07T01:43:23.809-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4450000 QueuedCount 50000 2016-04-07T01:43:23.989-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:43:24.107-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12350000 2016-04-07T01:43:24.176-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4460000 QueuedCount 50000 2016-04-07T01:43:24.642-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4470000 QueuedCount 50000 2016-04-07T01:43:24.655-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12360000 2016-04-07T01:43:25.087-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4480000 QueuedCount 50000 2016-04-07T01:43:25.292-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12370000 2016-04-07T01:43:25.524-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4490000 QueuedCount 50000 2016-04-07T01:43:25.719-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12380000 2016-04-07T01:43:25.956-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4500000 QueuedCount 50000 2016-04-07T01:43:26.058-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12390000 2016-04-07T01:43:26.212-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12400000 2016-04-07T01:43:26.388-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4510000 QueuedCount 50000 2016-04-07T01:43:26.807-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4520000 QueuedCount 43887 2016-04-07T01:43:27.184-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4530000 QueuedCount 33887 2016-04-07T01:43:27.554-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4540000 QueuedCount 23887 2016-04-07T01:43:27.878-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4550000 QueuedCount 13887 2016-04-07T01:43:28.242-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4560000 QueuedCount 3887 2016-04-07T01:43:28.771-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 33.951µs CommitTime 371.842302ms TotalFlushTime 2m44.872874399s TotalCommitTime 7.942765625s 2016-04-07T01:43:28.771-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 4563887, 4563887, 26 committed:true 2016-04-07T01:43:28.771-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 4563887, 4563887, 26 committed:true) SnapCreateDur 379.793067ms SnapOpenDur 54.749µs 2016-04-07T01:43:28.788-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:43:28.806-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12410000 2016-04-07T01:43:28.817-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12420000 2016-04-07T01:43:28.987-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4570000 QueuedCount 50000 2016-04-07T01:43:29.056-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12430000 2016-04-07T01:43:29.383-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4580000 QueuedCount 50000 2016-04-07T01:43:29.483-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12440000 2016-04-07T01:43:29.794-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4590000 QueuedCount 50000 2016-04-07T01:43:29.873-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:43:29.879-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12450000 2016-04-07T01:43:29.888-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:43:29.888-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:43:30.242-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4600000 QueuedCount 50000 2016-04-07T01:43:30.338-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12460000 2016-04-07T01:43:30.525-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12470000 2016-04-07T01:43:30.683-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4610000 QueuedCount 50000 2016-04-07T01:43:30.891-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12480000 2016-04-07T01:43:31.079-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12490000 2016-04-07T01:43:31.153-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4620000 QueuedCount 50000 2016-04-07T01:43:31.239-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:43:31.264-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12500000 2016-04-07T01:43:31.274-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12510000 2016-04-07T01:43:31.601-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4630000 QueuedCount 50000 2016-04-07T01:43:31.905-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12520000 2016-04-07T01:43:32.061-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4640000 QueuedCount 50000 2016-04-07T01:43:32.492-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4650000 QueuedCount 50000 2016-04-07T01:43:32.693-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12530000 2016-04-07T01:43:32.872-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4660000 QueuedCount 50000 2016-04-07T01:43:33.323-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4670000 QueuedCount 50000 2016-04-07T01:43:33.33-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12540000 2016-04-07T01:43:33.773-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4680000 QueuedCount 50000 2016-04-07T01:43:33.968-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12550000 2016-04-07T01:43:33.977-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12560000 2016-04-07T01:43:34.241-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4690000 QueuedCount 50000 2016-04-07T01:43:34.687-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4700000 QueuedCount 50000 2016-04-07T01:43:34.7-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12570000 2016-04-07T01:43:34.9-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:43:34.916-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:43:34.916-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:43:35.161-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4710000 QueuedCount 50000 2016-04-07T01:43:35.188-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12580000 2016-04-07T01:43:35.627-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4720000 QueuedCount 50000 2016-04-07T01:43:35.734-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12590000 2016-04-07T01:43:36.098-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4730000 QueuedCount 50000 2016-04-07T01:43:36.311-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12600000 2016-04-07T01:43:36.499-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4740000 QueuedCount 50000 2016-04-07T01:43:36.888-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12610000 2016-04-07T01:43:36.943-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4750000 QueuedCount 50000 2016-04-07T01:43:37.225-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12620000 2016-04-07T01:43:37.397-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4760000 QueuedCount 50000 2016-04-07T01:43:37.412-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12630000 2016-04-07T01:43:37.422-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12640000 2016-04-07T01:43:37.431-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12650000 2016-04-07T01:43:37.857-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4770000 QueuedCount 50000 2016-04-07T01:43:38.314-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4780000 QueuedCount 50000 2016-04-07T01:43:38.727-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4790000 QueuedCount 40610 2016-04-07T01:43:39.147-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4800000 QueuedCount 30610 2016-04-07T01:43:39.553-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4810000 QueuedCount 20610 2016-04-07T01:43:39.671-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:1240284703, Disk:2737967104, ExtraSnapshotData:0, Fragmentation:54.700525759129064%, mode : circular 2016-04-07T01:43:39.671-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:43:39.964-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4820000 QueuedCount 10610 2016-04-07T01:43:40.311-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4830000 QueuedCount 610 2016-04-07T01:43:40.865-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.233µs CommitTime 475.042279ms TotalFlushTime 2m56.212091323s TotalCommitTime 8.417807904s 2016-04-07T01:43:40.865-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 4830610, 4830610, 27 committed:true 2016-04-07T01:43:40.865-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 4830610, 4830610, 27 committed:true) SnapCreateDur 483.043302ms SnapOpenDur 64.432µs 2016-04-07T01:43:40.891-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12660000 2016-04-07T01:43:40.902-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12670000 2016-04-07T01:43:40.914-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12680000 2016-04-07T01:43:40.925-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12690000 2016-04-07T01:43:40.937-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12700000 2016-04-07T01:43:40.948-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12710000 2016-04-07T01:43:40.96-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12720000 2016-04-07T01:43:40.971-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12730000 2016-04-07T01:43:40.983-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12740000 2016-04-07T01:43:40.993-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12750000 2016-04-07T01:43:41.004-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12760000 2016-04-07T01:43:41.015-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12770000 2016-04-07T01:43:41.028-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12780000 2016-04-07T01:43:41.039-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12790000 2016-04-07T01:43:41.26-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4840000 QueuedCount 50000 2016-04-07T01:43:41.678-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4850000 QueuedCount 50000 2016-04-07T01:43:42.104-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4860000 QueuedCount 50000 2016-04-07T01:43:42.541-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4870000 QueuedCount 50000 2016-04-07T01:43:42.986-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4880000 QueuedCount 50000 2016-04-07T01:43:43.386-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4890000 QueuedCount 50000 2016-04-07T01:43:43.84-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4900000 QueuedCount 50000 2016-04-07T01:43:44.314-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4910000 QueuedCount 50000 2016-04-07T01:43:44.771-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4920000 QueuedCount 50000 2016-04-07T01:43:44.936-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:43:44.951-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:43:44.951-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:43:45.232-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4930000 QueuedCount 40953 2016-04-07T01:43:45.649-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4940000 QueuedCount 30953 2016-04-07T01:43:46.041-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5010 Milliseconds Vbucket 668 2016-04-07T01:43:46.066-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4950000 QueuedCount 20953 2016-04-07T01:43:46.071-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5040 Milliseconds Vbucket 668 2016-04-07T01:43:46.101-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5070 Milliseconds Vbucket 668 2016-04-07T01:43:46.131-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5100 Milliseconds Vbucket 668 2016-04-07T01:43:46.161-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5130 Milliseconds Vbucket 668 2016-04-07T01:43:46.191-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5160 Milliseconds Vbucket 668 2016-04-07T01:43:46.221-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5190 Milliseconds Vbucket 668 2016-04-07T01:43:46.251-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5220 Milliseconds Vbucket 668 2016-04-07T01:43:46.281-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5250 Milliseconds Vbucket 668 2016-04-07T01:43:46.311-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5280 Milliseconds Vbucket 668 2016-04-07T01:43:46.341-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5310 Milliseconds Vbucket 668 2016-04-07T01:43:46.371-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5340 Milliseconds Vbucket 668 2016-04-07T01:43:46.401-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5370 Milliseconds Vbucket 668 2016-04-07T01:43:46.431-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5400 Milliseconds Vbucket 668 2016-04-07T01:43:46.461-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5430 Milliseconds Vbucket 668 2016-04-07T01:43:46.491-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5460 Milliseconds Vbucket 668 2016-04-07T01:43:46.503-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4960000 QueuedCount 10953 2016-04-07T01:43:46.521-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5490 Milliseconds Vbucket 668 2016-04-07T01:43:46.551-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5520 Milliseconds Vbucket 668 2016-04-07T01:43:46.581-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5550 Milliseconds Vbucket 668 2016-04-07T01:43:46.611-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5580 Milliseconds Vbucket 668 2016-04-07T01:43:46.641-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5610 Milliseconds Vbucket 668 2016-04-07T01:43:46.671-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5640 Milliseconds Vbucket 668 2016-04-07T01:43:46.701-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5670 Milliseconds Vbucket 668 2016-04-07T01:43:46.731-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5700 Milliseconds Vbucket 668 2016-04-07T01:43:46.761-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5730 Milliseconds Vbucket 668 2016-04-07T01:43:46.791-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5760 Milliseconds Vbucket 668 2016-04-07T01:43:46.821-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5790 Milliseconds Vbucket 668 2016-04-07T01:43:46.851-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5820 Milliseconds Vbucket 668 2016-04-07T01:43:46.881-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5850 Milliseconds Vbucket 668 2016-04-07T01:43:46.91-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4970000 QueuedCount 953 2016-04-07T01:43:46.911-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5880 Milliseconds Vbucket 668 2016-04-07T01:43:46.941-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5910 Milliseconds Vbucket 668 2016-04-07T01:43:46.971-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5940 Milliseconds Vbucket 668 2016-04-07T01:43:47.001-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5970 Milliseconds Vbucket 668 2016-04-07T01:43:47.031-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6000 Milliseconds Vbucket 668 2016-04-07T01:43:47.061-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6030 Milliseconds Vbucket 668 2016-04-07T01:43:47.091-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6060 Milliseconds Vbucket 668 2016-04-07T01:43:47.121-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6090 Milliseconds Vbucket 668 2016-04-07T01:43:47.151-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6120 Milliseconds Vbucket 668 2016-04-07T01:43:47.181-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6150 Milliseconds Vbucket 668 2016-04-07T01:43:47.211-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6180 Milliseconds Vbucket 668 2016-04-07T01:43:47.241-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6210 Milliseconds Vbucket 668 2016-04-07T01:43:47.271-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6240 Milliseconds Vbucket 668 2016-04-07T01:43:47.301-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6270 Milliseconds Vbucket 668 2016-04-07T01:43:47.331-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6300 Milliseconds Vbucket 668 2016-04-07T01:43:47.361-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6330 Milliseconds Vbucket 668 2016-04-07T01:43:47.391-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6360 Milliseconds Vbucket 668 2016-04-07T01:43:47.415-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 35.291µs CommitTime 483.556887ms TotalFlushTime 3m2.144174027s TotalCommitTime 8.901364791s 2016-04-07T01:43:47.415-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 4970953, 4970953, 28 committed:true 2016-04-07T01:43:47.415-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 4970953, 4970953, 28 committed:true) SnapCreateDur 491.819437ms SnapOpenDur 67.598µs 2016-04-07T01:43:47.423-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6390 Milliseconds Vbucket 668 2016-04-07T01:43:47.577-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12800000 2016-04-07T01:43:47.81-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4980000 QueuedCount 50000 2016-04-07T01:43:48.003-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12810000 2016-04-07T01:43:48.258-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 4990000 QueuedCount 50000 2016-04-07T01:43:48.37-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12820000 2016-04-07T01:43:48.681-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5000000 QueuedCount 50000 2016-04-07T01:43:48.886-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12830000 2016-04-07T01:43:49.125-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5010000 QueuedCount 50000 2016-04-07T01:43:49.222-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12840000 2016-04-07T01:43:49.497-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5020000 QueuedCount 50000 2016-04-07T01:43:49.499-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12850000 2016-04-07T01:43:49.775-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12860000 2016-04-07T01:43:49.935-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5030000 QueuedCount 50000 2016-04-07T01:43:50.05-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12870000 2016-04-07T01:43:50.328-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12880000 2016-04-07T01:43:50.402-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5040000 QueuedCount 50000 2016-04-07T01:43:50.663-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12890000 2016-04-07T01:43:50.849-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5050000 QueuedCount 50000 2016-04-07T01:43:50.849-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12900000 2016-04-07T01:43:50.859-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12910000 2016-04-07T01:43:50.87-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12920000 2016-04-07T01:43:51.276-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5060000 QueuedCount 44518 2016-04-07T01:43:51.667-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5070000 QueuedCount 34518 2016-04-07T01:43:52.056-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5080000 QueuedCount 24518 2016-04-07T01:43:52.468-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5090000 QueuedCount 14518 2016-04-07T01:43:52.824-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5100000 QueuedCount 4518 2016-04-07T01:43:53.442-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 16.743µs CommitTime 395.143899ms TotalFlushTime 3m7.63856423s TotalCommitTime 9.29650869s 2016-04-07T01:43:53.442-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 5104518, 5104518, 29 committed:true 2016-04-07T01:43:53.442-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 5104518, 5104518, 29 committed:true) SnapCreateDur 403.330097ms SnapOpenDur 73.288µs 2016-04-07T01:43:53.481-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12930000 2016-04-07T01:43:53.492-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12940000 2016-04-07T01:43:53.501-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12950000 2016-04-07T01:43:53.596-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12960000 2016-04-07T01:43:53.606-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12970000 2016-04-07T01:43:53.675-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5110000 QueuedCount 50000 2016-04-07T01:43:53.875-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12980000 2016-04-07T01:43:54.062-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 12990000 2016-04-07T01:43:54.14-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5120000 QueuedCount 50000 2016-04-07T01:43:54.337-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13000000 2016-04-07T01:43:54.586-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5130000 QueuedCount 50000 2016-04-07T01:43:54.614-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13010000 2016-04-07T01:43:54.713-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13020000 2016-04-07T01:43:54.985-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13030000 2016-04-07T01:43:55.051-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5140000 QueuedCount 50000 2016-04-07T01:43:55.504-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5150000 QueuedCount 45157 2016-04-07T01:43:55.935-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5160000 QueuedCount 35157 2016-04-07T01:43:56.287-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5170000 QueuedCount 25157 2016-04-07T01:43:56.722-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5180000 QueuedCount 15157 2016-04-07T01:43:57.128-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5190000 QueuedCount 5157 2016-04-07T01:43:57.838-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.191µs CommitTime 459.010742ms TotalFlushTime 3m11.470954969s TotalCommitTime 9.755519432s 2016-04-07T01:43:57.838-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 5195157, 5195157, 30 committed:true 2016-04-07T01:43:57.838-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 5195157, 5195157, 30 committed:true) SnapCreateDur 467.28628ms SnapOpenDur 50.987µs 2016-04-07T01:43:57.88-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13040000 2016-04-07T01:43:57.892-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13050000 2016-04-07T01:43:58.073-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5200000 QueuedCount 50000 2016-04-07T01:43:58.102-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13060000 2016-04-07T01:43:58.523-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5210000 QueuedCount 50000 2016-04-07T01:43:58.528-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13070000 2016-04-07T01:43:58.716-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13080000 2016-04-07T01:43:58.727-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13090000 2016-04-07T01:43:58.738-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13100000 2016-04-07T01:43:58.75-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13110000 2016-04-07T01:43:58.759-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13120000 2016-04-07T01:43:58.96-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5220000 QueuedCount 50000 2016-04-07T01:43:59.057-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13130000 2016-04-07T01:43:59.406-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5230000 QueuedCount 50000 2016-04-07T01:43:59.544-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13140000 2016-04-07T01:43:59.787-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5240000 QueuedCount 50000 2016-04-07T01:43:59.791-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13150000 2016-04-07T01:43:59.992-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:44:00.005-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:44:00.005-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:44:00.156-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13160000 2016-04-07T01:44:00.234-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5250000 QueuedCount 50000 2016-04-07T01:44:00.434-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13170000 2016-04-07T01:44:00.691-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5260000 QueuedCount 50000 2016-04-07T01:44:01.15-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5270000 QueuedCount 50000 2016-04-07T01:44:01.599-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5280000 QueuedCount 50000 2016-04-07T01:44:01.875-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:44:01.878-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13180000 2016-04-07T01:44:02.049-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5290000 QueuedCount 50000 2016-04-07T01:44:02.58-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13190000 2016-04-07T01:44:02.645-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5300000 QueuedCount 49783 2016-04-07T01:44:02.701-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13200000 2016-04-07T01:44:03.561-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5310000 QueuedCount 50000 2016-04-07T01:44:03.787-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13210000 2016-04-07T01:44:03.984-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5320000 QueuedCount 50000 2016-04-07T01:44:04.471-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5330000 QueuedCount 50000 2016-04-07T01:44:04.576-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13220000 2016-04-07T01:44:04.961-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5340000 QueuedCount 50000 2016-04-07T01:44:05.011-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:44:05.025-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:44:05.025-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:44:05.424-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13230000 2016-04-07T01:44:05.435-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5350000 QueuedCount 50000 2016-04-07T01:44:05.911-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13240000 2016-04-07T01:44:05.92-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5360000 QueuedCount 50000 2016-04-07T01:44:06.309-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13250000 2016-04-07T01:44:06.401-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5370000 QueuedCount 50000 2016-04-07T01:44:06.886-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5380000 QueuedCount 50000 2016-04-07T01:44:07.006-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13260000 2016-04-07T01:44:07.389-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5390000 QueuedCount 50000 2016-04-07T01:44:07.613-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13270000 2016-04-07T01:44:07.811-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5400000 QueuedCount 50000 2016-04-07T01:44:08.011-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13280000 2016-04-07T01:44:08.302-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5410000 QueuedCount 50000 2016-04-07T01:44:08.318-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13290000 2016-04-07T01:44:08.782-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5420000 QueuedCount 50000 2016-04-07T01:44:08.864-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13300000 2016-04-07T01:44:09.266-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5430000 QueuedCount 50000 2016-04-07T01:44:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:44:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T01:44:09.374-07:00 [Info] client load stats {} 2016-04-07T01:44:09.633-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":7810159,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":5791419583,"bucket-1:myindex:avg_ts_items_count":124790,"bucket-1:myindex:build_progress":26,"bucket-1:myindex:data_size":1399552323,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":3453136896,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":50058,"bucket-1:myindex:frag_percent":59,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":568007895,"bucket-1:myindex:items_count":5195157,"bucket-1:myindex:num_commits":30,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":5436976,"bucket-1:myindex:num_docs_pending":6712690,"bucket-1:myindex:num_docs_processed":5336622,"bucket-1:myindex:num_docs_queued":7950688,"bucket-1:myindex:num_flush_queued":5487035,"bucket-1:myindex:num_items_flushed":5436977,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":30,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":4396390679,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"30 9755519432 3552622531402878882","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"5436980 31006675579 151381297531237773","bucket-1:myindex:timings/storage_info":"121 1011164 15279637338","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"29 9868221 3762393108865","bucket-1:myindex:timings/storage_meta_set":"30 10220970 3792670131098","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"30 1463936 76387211148","bucket-1:myindex:timings/storage_set":"10873960 169016256768 7694133082042931310","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":13297138,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"47 95518479 2033530052088015","bucket-1:ts_queue_size":600,"indexer_state":"Active","memory_quota":31457280000,"memory_used":11867654144,"memory_used_storage":3698733056,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"1338 154501471 27598952992371","uptime":"15m1.326447326s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T01:44:09.671-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:1406764146, Disk:3453136896, ExtraSnapshotData:0, Fragmentation:59.261269148363354%, mode : circular 2016-04-07T01:44:09.671-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:44:09.676-07:00 [Info] memstats {"Alloc":6890138704, "TotalAlloc":12652431808, "Sys":9853772104, "Lookups":1835, "Mallocs":325098823,"Frees":174167221, "HeapAlloc":6890138704, "HeapSys":9385082880, "HeapIdle":1512497152, "HeapInuse":7872585728,"HeapReleased":0, "HeapObjects":150931602,"GCSys":296335360, "LastGC":1460018522138384238,"PauseTotalNs":22534959, "PauseNs":[], "NumGC":17} 2016-04-07T01:44:09.745-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5440000 QueuedCount 50000 2016-04-07T01:44:10.243-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5450000 QueuedCount 50000 2016-04-07T01:44:10.709-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5460000 QueuedCount 45764 2016-04-07T01:44:11.143-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5470000 QueuedCount 35764 2016-04-07T01:44:11.58-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5480000 QueuedCount 25764 2016-04-07T01:44:11.95-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5490000 QueuedCount 15764 2016-04-07T01:44:12.383-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5500000 QueuedCount 5764 2016-04-07T01:44:13.225-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 36.342µs CommitTime 573.783942ms TotalFlushTime 3m25.956420501s TotalCommitTime 10.329303374s 2016-04-07T01:44:13.225-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 5505764, 5505764, 31 committed:true 2016-04-07T01:44:13.226-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 5505764, 5505764, 31 committed:true) SnapCreateDur 582.066066ms SnapOpenDur 77.462µs 2016-04-07T01:44:13.372-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13310000 2016-04-07T01:44:13.436-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5510000 QueuedCount 50000 2016-04-07T01:44:13.836-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5520000 QueuedCount 50000 2016-04-07T01:44:14.102-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13320000 2016-04-07T01:44:14.287-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5530000 QueuedCount 50000 2016-04-07T01:44:14.649-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13330000 2016-04-07T01:44:14.75-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5540000 QueuedCount 50000 2016-04-07T01:44:15.05-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:44:15.063-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:44:15.063-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:44:15.223-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5550000 QueuedCount 50000 2016-04-07T01:44:15.256-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13340000 2016-04-07T01:44:15.674-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5560000 QueuedCount 50000 2016-04-07T01:44:15.774-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13350000 2016-04-07T01:44:16.151-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5570000 QueuedCount 50000 2016-04-07T01:44:16.171-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13360000 2016-04-07T01:44:16.449-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13370000 2016-04-07T01:44:16.609-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5580000 QueuedCount 50000 2016-04-07T01:44:17.078-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5590000 QueuedCount 50000 2016-04-07T01:44:17.085-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13380000 2016-04-07T01:44:17.534-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5600000 QueuedCount 50000 2016-04-07T01:44:17.574-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13390000 2016-04-07T01:44:17.822-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13400000 2016-04-07T01:44:17.92-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5610000 QueuedCount 50000 2016-04-07T01:44:18.188-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13410000 2016-04-07T01:44:18.382-07:00 [Info] Indexer::ReadMemstats Time Taken 16.606669ms 2016-04-07T01:44:18.403-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5620000 QueuedCount 50000 2016-04-07T01:44:18.506-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13420000 2016-04-07T01:44:18.813-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13430000 2016-04-07T01:44:18.89-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5630000 QueuedCount 50000 2016-04-07T01:44:18.999-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13440000 2016-04-07T01:44:19.1-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13450000 2016-04-07T01:44:19.11-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13460000 2016-04-07T01:44:19.12-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13470000 2016-04-07T01:44:19.129-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13480000 2016-04-07T01:44:19.139-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13490000 2016-04-07T01:44:19.148-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13500000 2016-04-07T01:44:19.372-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5640000 QueuedCount 50000 2016-04-07T01:44:19.823-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5650000 QueuedCount 43850 2016-04-07T01:44:20.271-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5660000 QueuedCount 33850 2016-04-07T01:44:20.716-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5670000 QueuedCount 23850 2016-04-07T01:44:21.172-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5680000 QueuedCount 13850 2016-04-07T01:44:21.547-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5690000 QueuedCount 3850 2016-04-07T01:44:22.302-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.504µs CommitTime 575.444322ms TotalFlushTime 3m34.270643009s TotalCommitTime 10.904747696s 2016-04-07T01:44:22.302-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 5693850, 5693850, 32 committed:true 2016-04-07T01:44:22.303-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 5693850, 5693850, 32 committed:true) SnapCreateDur 583.719118ms SnapOpenDur 68.272µs 2016-04-07T01:44:22.361-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13510000 2016-04-07T01:44:22.62-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5700000 QueuedCount 50000 2016-04-07T01:44:22.998-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13520000 2016-04-07T01:44:22.999-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5710000 QueuedCount 50000 2016-04-07T01:44:23.453-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5720000 QueuedCount 50000 2016-04-07T01:44:23.846-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13530000 2016-04-07T01:44:23.921-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5730000 QueuedCount 50000 2016-04-07T01:44:24.391-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5740000 QueuedCount 50000 2016-04-07T01:44:24.513-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13540000 2016-04-07T01:44:24.848-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5750000 QueuedCount 50000 2016-04-07T01:44:25.062-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13550000 2016-04-07T01:44:25.084-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:44:25.098-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:44:25.098-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:44:25.333-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5760000 QueuedCount 50000 2016-04-07T01:44:25.637-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13560000 2016-04-07T01:44:25.647-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13570000 2016-04-07T01:44:25.804-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5770000 QueuedCount 50000 2016-04-07T01:44:25.948-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13580000 2016-04-07T01:44:26.298-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5780000 QueuedCount 50000 2016-04-07T01:44:26.438-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13590000 2016-04-07T01:44:26.712-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5790000 QueuedCount 50000 2016-04-07T01:44:26.834-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13600000 2016-04-07T01:44:27.19-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5800000 QueuedCount 50000 2016-04-07T01:44:27.199-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13610000 2016-04-07T01:44:27.656-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5810000 QueuedCount 50000 2016-04-07T01:44:27.658-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13620000 2016-04-07T01:44:27.964-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13630000 2016-04-07T01:44:28.136-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5820000 QueuedCount 50000 2016-04-07T01:44:28.239-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13640000 2016-04-07T01:44:28.339-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13650000 2016-04-07T01:44:28.349-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13660000 2016-04-07T01:44:28.359-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13670000 2016-04-07T01:44:28.37-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13680000 2016-04-07T01:44:28.431-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13690000 2016-04-07T01:44:28.622-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5830000 QueuedCount 50000 2016-04-07T01:44:29.102-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5840000 QueuedCount 50000 2016-04-07T01:44:29.548-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5850000 QueuedCount 42149 2016-04-07T01:44:29.969-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5860000 QueuedCount 32149 2016-04-07T01:44:30.35-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5870000 QueuedCount 22149 2016-04-07T01:44:30.79-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5880000 QueuedCount 12149 2016-04-07T01:44:31.211-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5890000 QueuedCount 2149 2016-04-07T01:44:31.837-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 15.249µs CommitTime 518.511339ms TotalFlushTime 3m43.084260193s TotalCommitTime 11.423259035s 2016-04-07T01:44:31.837-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 5892149, 5892149, 33 committed:true 2016-04-07T01:44:31.837-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 5892149, 5892149, 33 committed:true) SnapCreateDur 527.162289ms SnapOpenDur 51.32µs 2016-04-07T01:44:31.867-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13700000 2016-04-07T01:44:32.143-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5900000 QueuedCount 50000 2016-04-07T01:44:32.235-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13710000 2016-04-07T01:44:32.594-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5910000 QueuedCount 50000 2016-04-07T01:44:33.052-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13720000 2016-04-07T01:44:33.053-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5920000 QueuedCount 50000 2016-04-07T01:44:33.45-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13730000 2016-04-07T01:44:33.54-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5930000 QueuedCount 50000 2016-04-07T01:44:33.846-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13740000 2016-04-07T01:44:34.021-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5940000 QueuedCount 50000 2016-04-07T01:44:34.333-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13750000 2016-04-07T01:44:34.514-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5950000 QueuedCount 50000 2016-04-07T01:44:34.73-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13760000 2016-04-07T01:44:35.022-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5960000 QueuedCount 50000 2016-04-07T01:44:35.127-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:44:35.142-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:44:35.142-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:44:35.247-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13770000 2016-04-07T01:44:35.262-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13780000 2016-04-07T01:44:35.276-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13790000 2016-04-07T01:44:35.288-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13800000 2016-04-07T01:44:35.521-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5970000 QueuedCount 50000 2016-04-07T01:44:35.558-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13810000 2016-04-07T01:44:35.834-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13820000 2016-04-07T01:44:35.939-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5980000 QueuedCount 50000 2016-04-07T01:44:36.141-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13830000 2016-04-07T01:44:36.438-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 5990000 QueuedCount 50000 2016-04-07T01:44:36.449-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13840000 2016-04-07T01:44:36.876-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13850000 2016-04-07T01:44:36.958-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6000000 QueuedCount 50000 2016-04-07T01:44:37.187-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13860000 2016-04-07T01:44:37.459-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13870000 2016-04-07T01:44:37.466-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6010000 QueuedCount 50000 2016-04-07T01:44:37.499-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13880000 2016-04-07T01:44:37.509-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13890000 2016-04-07T01:44:37.52-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13900000 2016-04-07T01:44:37.529-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13910000 2016-04-07T01:44:37.949-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6020000 QueuedCount 50000 2016-04-07T01:44:38.433-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6030000 QueuedCount 48246 2016-04-07T01:44:38.871-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6040000 QueuedCount 38246 2016-04-07T01:44:39.305-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6050000 QueuedCount 28246 2016-04-07T01:44:39.671-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:1563910898, Disk:4223803392, ExtraSnapshotData:0, Fragmentation:62.97387087282305%, mode : circular 2016-04-07T01:44:39.671-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:44:39.679-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6060000 QueuedCount 18246 2016-04-07T01:44:40.117-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6070000 QueuedCount 8246 2016-04-07T01:44:41.119-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.543µs CommitTime 628.95321ms TotalFlushTime 3m51.555369267s TotalCommitTime 12.052212245s 2016-04-07T01:44:41.119-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 6078246, 6078246, 34 committed:true 2016-04-07T01:44:41.119-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 6078246, 6078246, 34 committed:true) SnapCreateDur 638.006538ms SnapOpenDur 70.097µs 2016-04-07T01:44:41.165-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6080000 QueuedCount 50000 2016-04-07T01:44:41.279-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13920000 2016-04-07T01:44:41.439-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13930000 2016-04-07T01:44:41.625-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13940000 2016-04-07T01:44:41.627-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6090000 QueuedCount 50000 2016-04-07T01:44:42.079-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6100000 QueuedCount 47050 2016-04-07T01:44:42.535-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6110000 QueuedCount 37050 2016-04-07T01:44:42.993-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6120000 QueuedCount 27050 2016-04-07T01:44:43.439-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6130000 QueuedCount 17050 2016-04-07T01:44:43.888-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6140000 QueuedCount 7050 2016-04-07T01:44:45.065-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 42.883µs CommitTime 886.000631ms TotalFlushTime 3m54.543861974s TotalCommitTime 12.938212876s 2016-04-07T01:44:45.065-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 6147050, 6147050, 35 committed:true 2016-04-07T01:44:45.065-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 6147050, 6147050, 35 committed:true) SnapCreateDur 894.651645ms SnapOpenDur 60.908µs 2016-04-07T01:44:45.068-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:44:45.07-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:44:45.172-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:44:45.172-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6150000 QueuedCount 50000 2016-04-07T01:44:45.173-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13950000 2016-04-07T01:44:45.188-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:44:45.188-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:44:45.571-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13960000 2016-04-07T01:44:45.575-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6160000 QueuedCount 50000 2016-04-07T01:44:46.016-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6170000 QueuedCount 50000 2016-04-07T01:44:46.028-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13970000 2016-04-07T01:44:46.217-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13980000 2016-04-07T01:44:46.472-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6180000 QueuedCount 50000 2016-04-07T01:44:46.893-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6190000 QueuedCount 50000 2016-04-07T01:44:47.367-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6200000 QueuedCount 50000 2016-04-07T01:44:47.872-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6210000 QueuedCount 50000 2016-04-07T01:44:48.374-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6220000 QueuedCount 50000 2016-04-07T01:44:48.592-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:44:48.715-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 13990000 2016-04-07T01:44:48.9-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6230000 QueuedCount 50000 2016-04-07T01:44:49.438-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6240000 QueuedCount 50000 2016-04-07T01:44:49.654-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14000000 2016-04-07T01:44:49.955-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6250000 QueuedCount 50000 2016-04-07T01:44:50.413-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14010000 2016-04-07T01:44:50.475-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6260000 QueuedCount 50000 2016-04-07T01:44:50.986-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6270000 QueuedCount 50000 2016-04-07T01:44:51.021-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14020000 2016-04-07T01:44:51.415-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6280000 QueuedCount 50000 2016-04-07T01:44:51.748-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14030000 2016-04-07T01:44:51.938-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6290000 QueuedCount 50000 2016-04-07T01:44:52.445-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6300000 QueuedCount 50000 2016-04-07T01:44:52.567-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14040000 2016-04-07T01:44:52.965-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6310000 QueuedCount 50000 2016-04-07T01:44:53.174-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14050000 2016-04-07T01:44:53.473-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6320000 QueuedCount 50000 2016-04-07T01:44:53.813-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14060000 2016-04-07T01:44:53.991-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6330000 QueuedCount 50000 2016-04-07T01:44:54.488-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6340000 QueuedCount 50000 2016-04-07T01:44:54.54-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14070000 2016-04-07T01:44:55.014-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6350000 QueuedCount 50000 2016-04-07T01:44:55.209-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:44:55.227-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:44:55.227-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:44:55.448-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14080000 2016-04-07T01:44:55.452-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6360000 QueuedCount 50000 2016-04-07T01:44:55.666-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14090000 2016-04-07T01:44:55.963-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6370000 QueuedCount 50000 2016-04-07T01:44:56.063-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14100000 2016-04-07T01:44:56.166-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14110000 2016-04-07T01:44:56.177-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14120000 2016-04-07T01:44:56.189-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14130000 2016-04-07T01:44:56.199-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14140000 2016-04-07T01:44:56.465-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6380000 QueuedCount 50000 2016-04-07T01:44:56.468-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14150000 2016-04-07T01:44:56.977-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6390000 QueuedCount 50000 2016-04-07T01:44:57.489-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6400000 QueuedCount 49588 2016-04-07T01:44:57.971-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6410000 QueuedCount 39588 2016-04-07T01:44:58.434-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6420000 QueuedCount 29588 2016-04-07T01:44:58.906-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6430000 QueuedCount 19588 2016-04-07T01:44:59.374-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6440000 QueuedCount 9588 2016-04-07T01:45:00.571-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.88µs CommitTime 798.510692ms TotalFlushTime 4m8.946137034s TotalCommitTime 13.736723568s 2016-04-07T01:45:00.571-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 6449588, 6449588, 36 committed:true 2016-04-07T01:45:00.571-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 6449588, 6449588, 36 committed:true) SnapCreateDur 806.859339ms SnapOpenDur 57.434µs 2016-04-07T01:45:00.596-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6450000 QueuedCount 40229 2016-04-07T01:45:00.614-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14160000 2016-04-07T01:45:00.921-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14170000 2016-04-07T01:45:00.983-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6460000 QueuedCount 50000 2016-04-07T01:45:01.317-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14180000 2016-04-07T01:45:01.323-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6470000 QueuedCount 50000 2016-04-07T01:45:01.656-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14190000 2016-04-07T01:45:01.731-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6480000 QueuedCount 50000 2016-04-07T01:45:02.146-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6490000 QueuedCount 50000 2016-04-07T01:45:02.233-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14200000 2016-04-07T01:45:02.569-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6500000 QueuedCount 50000 2016-04-07T01:45:02.6-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14210000 2016-04-07T01:45:03.047-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6510000 QueuedCount 50000 2016-04-07T01:45:03.058-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14220000 2016-04-07T01:45:03.456-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14230000 2016-04-07T01:45:03.53-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6520000 QueuedCount 50000 2016-04-07T01:45:03.913-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14240000 2016-04-07T01:45:04.01-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6530000 QueuedCount 50000 2016-04-07T01:45:04.133-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14250000 2016-04-07T01:45:04.145-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14260000 2016-04-07T01:45:04.158-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14270000 2016-04-07T01:45:04.169-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14280000 2016-04-07T01:45:04.5-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6540000 QueuedCount 50000 2016-04-07T01:45:04.97-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6550000 QueuedCount 50000 2016-04-07T01:45:05.248-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:45:05.264-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:45:05.264-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:45:05.379-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6560000 QueuedCount 50000 2016-04-07T01:45:05.832-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6570000 QueuedCount 45207 2016-04-07T01:45:06.28-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6580000 QueuedCount 35207 2016-04-07T01:45:06.705-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6590000 QueuedCount 25207 2016-04-07T01:45:07.227-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6600000 QueuedCount 15207 2016-04-07T01:45:07.634-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6610000 QueuedCount 5207 2016-04-07T01:45:08.332-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 35.995µs CommitTime 514.525454ms TotalFlushTime 4m16.019453823s TotalCommitTime 14.251249022s 2016-04-07T01:45:08.332-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 6615207, 6615207, 37 committed:true 2016-04-07T01:45:08.332-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 6615207, 6615207, 37 committed:true) SnapCreateDur 522.529965ms SnapOpenDur 68.221µs 2016-04-07T01:45:08.458-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14290000 2016-04-07T01:45:08.521-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6620000 QueuedCount 50000 2016-04-07T01:45:08.908-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6630000 QueuedCount 50000 2016-04-07T01:45:09.096-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14300000 2016-04-07T01:45:09.321-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6640000 QueuedCount 50000 2016-04-07T01:45:09.373-07:00 [Info] connected with 1 indexers 2016-04-07T01:45:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T01:45:09.374-07:00 [Info] client load stats {} 2016-04-07T01:45:09.634-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":7600022,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":9435656886,"bucket-1:myindex:avg_ts_items_count":191212,"bucket-1:myindex:build_progress":33,"bucket-1:myindex:data_size":1716550565,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":4703744000,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":50095,"bucket-1:myindex:frag_percent":63,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":695672145,"bucket-1:myindex:items_count":6615207,"bucket-1:myindex:num_commits":37,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":6647982,"bucket-1:myindex:num_docs_pending":5706822,"bucket-1:myindex:num_docs_processed":6615207,"bucket-1:myindex:num_docs_queued":7677971,"bucket-1:myindex:num_flush_queued":6698077,"bucket-1:myindex:num_items_flushed":6647982,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":37,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":7761099626,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"37 14251249022 6564775746447478092","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"6647983 38709604115 151437527288492013","bucket-1:myindex:timings/storage_info":"149 1215027 17462657411","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"36 12307787 4658840844665","bucket-1:myindex:timings/storage_meta_set":"37 12709097 4724485198777","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"37 1803500 93424248512","bucket-1:myindex:timings/storage_set":"13295965 214869984602 -8123094199763481488","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":14298005,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"59 105819059 2042416088843217","bucket-1:ts_queue_size":761,"indexer_state":"Active","memory_quota":31457280000,"memory_used":11865225216,"memory_used_storage":4704186368,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"1481 186748139 40614123992155","uptime":"16m1.326822716s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T01:45:09.672-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:1722813758, Disk:4703744000, ExtraSnapshotData:0, Fragmentation:63.373564590249806%, mode : circular 2016-04-07T01:45:09.672-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:45:09.676-07:00 [Info] memstats {"Alloc":5175761640, "TotalAlloc":13358160208, "Sys":9854296392, "Lookups":2271, "Mallocs":346932832,"Frees":234640125, "HeapAlloc":5175761640, "HeapSys":9384689664, "HeapIdle":2521096192, "HeapInuse":6863593472,"HeapReleased":0, "HeapObjects":112292707,"GCSys":297445376, "LastGC":1460018642840087332,"PauseTotalNs":23651282, "PauseNs":[1116323], "NumGC":18} 2016-04-07T01:45:09.683-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6650000 QueuedCount 50000 2016-04-07T01:45:09.763-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14310000 2016-04-07T01:45:10.099-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6660000 QueuedCount 50000 2016-04-07T01:45:10.192-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14320000 2016-04-07T01:45:10.272-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:45:10.286-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:45:10.286-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:45:10.544-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6670000 QueuedCount 50000 2016-04-07T01:45:10.648-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14330000 2016-04-07T01:45:10.982-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6680000 QueuedCount 50000 2016-04-07T01:45:11.106-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14340000 2016-04-07T01:45:11.432-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6690000 QueuedCount 50000 2016-04-07T01:45:11.533-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14350000 2016-04-07T01:45:11.871-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6700000 QueuedCount 50000 2016-04-07T01:45:12.051-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14360000 2016-04-07T01:45:12.267-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14370000 2016-04-07T01:45:12.326-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6710000 QueuedCount 50000 2016-04-07T01:45:12.544-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14380000 2016-04-07T01:45:12.768-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6720000 QueuedCount 50000 2016-04-07T01:45:12.791-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14390000 2016-04-07T01:45:12.889-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14400000 2016-04-07T01:45:12.987-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14410000 2016-04-07T01:45:13.083-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14420000 2016-04-07T01:45:13.223-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6730000 QueuedCount 50000 2016-04-07T01:45:13.566-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6740000 QueuedCount 40486 2016-04-07T01:45:13.934-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6750000 QueuedCount 30486 2016-04-07T01:45:14.304-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6760000 QueuedCount 20486 2016-04-07T01:45:14.664-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6770000 QueuedCount 10486 2016-04-07T01:45:15.033-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6780000 QueuedCount 486 2016-04-07T01:45:15.478-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.768µs CommitTime 430.740942ms TotalFlushTime 4m22.561637694s TotalCommitTime 14.681989964s 2016-04-07T01:45:15.478-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 6780486, 6780486, 38 committed:true 2016-04-07T01:45:15.478-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 6780486, 6780486, 38 committed:true) SnapCreateDur 438.84992ms SnapOpenDur 46.653µs 2016-04-07T01:45:15.522-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14430000 2016-04-07T01:45:15.533-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14440000 2016-04-07T01:45:15.542-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14450000 2016-04-07T01:45:15.81-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6790000 QueuedCount 50000 2016-04-07T01:45:16.051-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14460000 2016-04-07T01:45:16.194-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6800000 QueuedCount 50000 2016-04-07T01:45:16.583-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6810000 QueuedCount 50000 2016-04-07T01:45:16.6-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14470000 2016-04-07T01:45:16.984-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6820000 QueuedCount 50000 2016-04-07T01:45:16.998-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14480000 2016-04-07T01:45:17.414-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6830000 QueuedCount 50000 2016-04-07T01:45:17.454-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14490000 2016-04-07T01:45:17.87-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6840000 QueuedCount 50000 2016-04-07T01:45:17.912-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14500000 2016-04-07T01:45:18.256-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6850000 QueuedCount 50000 2016-04-07T01:45:18.429-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14510000 2016-04-07T01:45:18.707-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14520000 2016-04-07T01:45:18.71-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6860000 QueuedCount 50000 2016-04-07T01:45:18.8-07:00 [Info] Indexer::ReadMemstats Time Taken 14.309927ms 2016-04-07T01:45:19.076-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14530000 2016-04-07T01:45:19.177-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6870000 QueuedCount 50000 2016-04-07T01:45:19.352-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14540000 2016-04-07T01:45:19.627-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6880000 QueuedCount 50000 2016-04-07T01:45:19.629-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14550000 2016-04-07T01:45:19.816-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14560000 2016-04-07T01:45:20.07-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6890000 QueuedCount 50000 2016-04-07T01:45:20.093-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14570000 2016-04-07T01:45:20.19-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14580000 2016-04-07T01:45:20.199-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14590000 2016-04-07T01:45:20.524-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6900000 QueuedCount 50000 2016-04-07T01:45:20.95-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6910000 QueuedCount 43861 2016-04-07T01:45:21.371-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6920000 QueuedCount 33861 2016-04-07T01:45:21.716-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6930000 QueuedCount 23861 2016-04-07T01:45:22.117-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6940000 QueuedCount 13861 2016-04-07T01:45:22.522-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6950000 QueuedCount 3861 2016-04-07T01:45:23.206-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.322µs CommitTime 508.978167ms TotalFlushTime 4m29.59317156s TotalCommitTime 15.190968131s 2016-04-07T01:45:23.206-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 6953861, 6953861, 39 committed:true 2016-04-07T01:45:23.207-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 6953861, 6953861, 39 committed:true) SnapCreateDur 516.872686ms SnapOpenDur 46.956µs 2016-04-07T01:45:23.43-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6960000 QueuedCount 50000 2016-04-07T01:45:23.45-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14600000 2016-04-07T01:45:23.815-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6970000 QueuedCount 50000 2016-04-07T01:45:23.998-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14610000 2016-04-07T01:45:24.233-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6980000 QueuedCount 50000 2016-04-07T01:45:24.678-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 6990000 QueuedCount 50000 2016-04-07T01:45:24.695-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14620000 2016-04-07T01:45:25.099-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7000000 QueuedCount 50000 2016-04-07T01:45:25.122-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14630000 2016-04-07T01:45:25.332-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:45:25.345-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:45:25.345-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:45:25.528-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7010000 QueuedCount 50000 2016-04-07T01:45:25.55-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14640000 2016-04-07T01:45:25.99-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7020000 QueuedCount 50000 2016-04-07T01:45:26.097-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14650000 2016-04-07T01:45:26.459-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7030000 QueuedCount 50000 2016-04-07T01:45:26.495-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14660000 2016-04-07T01:45:26.863-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14670000 2016-04-07T01:45:26.93-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7040000 QueuedCount 50000 2016-04-07T01:45:27.168-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14680000 2016-04-07T01:45:27.323-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7050000 QueuedCount 50000 2016-04-07T01:45:27.421-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14690000 2016-04-07T01:45:27.432-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14700000 2016-04-07T01:45:27.443-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14710000 2016-04-07T01:45:27.455-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14720000 2016-04-07T01:45:27.465-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14730000 2016-04-07T01:45:27.477-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14740000 2016-04-07T01:45:27.488-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14750000 2016-04-07T01:45:27.519-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14760000 2016-04-07T01:45:27.785-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7060000 QueuedCount 50000 2016-04-07T01:45:27.885-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14770000 2016-04-07T01:45:28.254-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7070000 QueuedCount 50000 2016-04-07T01:45:28.692-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7080000 QueuedCount 45756 2016-04-07T01:45:29.097-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7090000 QueuedCount 35756 2016-04-07T01:45:29.495-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7100000 QueuedCount 25756 2016-04-07T01:45:29.914-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7110000 QueuedCount 15756 2016-04-07T01:45:30.3-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7120000 QueuedCount 5756 2016-04-07T01:45:30.936-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.132µs CommitTime 452.07345ms TotalFlushTime 4m36.686484322s TotalCommitTime 15.643041581s 2016-04-07T01:45:30.936-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 7125756, 7125756, 40 committed:true 2016-04-07T01:45:30.936-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 7125756, 7125756, 40 committed:true) SnapCreateDur 461.549674ms SnapOpenDur 50.332µs 2016-04-07T01:45:31.136-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7130000 QueuedCount 50000 2016-04-07T01:45:31.253-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14780000 2016-04-07T01:45:31.491-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7140000 QueuedCount 50000 2016-04-07T01:45:31.859-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14790000 2016-04-07T01:45:31.926-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7150000 QueuedCount 50000 2016-04-07T01:45:32.359-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7160000 QueuedCount 50000 2016-04-07T01:45:32.437-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14800000 2016-04-07T01:45:32.812-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7170000 QueuedCount 50000 2016-04-07T01:45:32.925-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14810000 2016-04-07T01:45:32.941-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14820000 2016-04-07T01:45:32.952-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14830000 2016-04-07T01:45:32.963-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14840000 2016-04-07T01:45:32.973-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14850000 2016-04-07T01:45:32.985-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14860000 2016-04-07T01:45:32.996-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14870000 2016-04-07T01:45:33.008-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14880000 2016-04-07T01:45:33.02-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14890000 2016-04-07T01:45:33.03-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14900000 2016-04-07T01:45:33.04-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14910000 2016-04-07T01:45:33.051-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14920000 2016-04-07T01:45:33.278-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7180000 QueuedCount 50000 2016-04-07T01:45:33.735-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7190000 QueuedCount 50000 2016-04-07T01:45:34.218-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7200000 QueuedCount 50000 2016-04-07T01:45:34.694-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7210000 QueuedCount 50000 2016-04-07T01:45:35.166-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7220000 QueuedCount 50000 2016-04-07T01:45:35.367-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:45:35.38-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:45:35.38-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:45:35.573-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7230000 QueuedCount 50000 2016-04-07T01:45:36.052-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7240000 QueuedCount 50000 2016-04-07T01:45:36.502-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7250000 QueuedCount 41976 2016-04-07T01:45:36.947-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7260000 QueuedCount 31976 2016-04-07T01:45:37.402-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7270000 QueuedCount 21976 2016-04-07T01:45:37.85-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7280000 QueuedCount 11976 2016-04-07T01:45:38.055-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5010 Milliseconds Vbucket 632 2016-04-07T01:45:38.085-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5040 Milliseconds Vbucket 632 2016-04-07T01:45:38.115-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5070 Milliseconds Vbucket 632 2016-04-07T01:45:38.145-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5100 Milliseconds Vbucket 632 2016-04-07T01:45:38.175-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5130 Milliseconds Vbucket 632 2016-04-07T01:45:38.205-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5160 Milliseconds Vbucket 632 2016-04-07T01:45:38.235-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5190 Milliseconds Vbucket 632 2016-04-07T01:45:38.265-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5220 Milliseconds Vbucket 632 2016-04-07T01:45:38.295-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5250 Milliseconds Vbucket 632 2016-04-07T01:45:38.296-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7290000 QueuedCount 1976 2016-04-07T01:45:38.325-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5280 Milliseconds Vbucket 632 2016-04-07T01:45:38.355-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5310 Milliseconds Vbucket 632 2016-04-07T01:45:38.385-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5340 Milliseconds Vbucket 632 2016-04-07T01:45:38.415-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5370 Milliseconds Vbucket 632 2016-04-07T01:45:38.445-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5400 Milliseconds Vbucket 632 2016-04-07T01:45:38.475-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5430 Milliseconds Vbucket 632 2016-04-07T01:45:38.505-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5460 Milliseconds Vbucket 632 2016-04-07T01:45:38.535-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5490 Milliseconds Vbucket 632 2016-04-07T01:45:38.565-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5520 Milliseconds Vbucket 632 2016-04-07T01:45:38.595-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5550 Milliseconds Vbucket 632 2016-04-07T01:45:38.625-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5580 Milliseconds Vbucket 632 2016-04-07T01:45:38.655-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5610 Milliseconds Vbucket 632 2016-04-07T01:45:38.685-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5640 Milliseconds Vbucket 632 2016-04-07T01:45:38.715-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5670 Milliseconds Vbucket 632 2016-04-07T01:45:38.745-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5700 Milliseconds Vbucket 632 2016-04-07T01:45:38.775-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5730 Milliseconds Vbucket 632 2016-04-07T01:45:38.805-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5760 Milliseconds Vbucket 632 2016-04-07T01:45:38.835-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5790 Milliseconds Vbucket 632 2016-04-07T01:45:38.865-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5820 Milliseconds Vbucket 632 2016-04-07T01:45:38.895-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5850 Milliseconds Vbucket 632 2016-04-07T01:45:38.925-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5880 Milliseconds Vbucket 632 2016-04-07T01:45:38.929-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 50.709µs CommitTime 535.515373ms TotalFlushTime 4m43.961889957s TotalCommitTime 16.178556954s 2016-04-07T01:45:38.929-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 7291976, 7291976, 41 committed:true 2016-04-07T01:45:38.929-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 7291976, 7291976, 41 committed:true) SnapCreateDur 543.607604ms SnapOpenDur 46.316µs 2016-04-07T01:45:38.955-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5910 Milliseconds Vbucket 632 2016-04-07T01:45:38.991-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14930000 2016-04-07T01:45:39.207-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7300000 QueuedCount 50000 2016-04-07T01:45:39.386-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14940000 2016-04-07T01:45:39.611-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7310000 QueuedCount 50000 2016-04-07T01:45:39.672-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:1905071489, Disk:4703744000, ExtraSnapshotData:0, Fragmentation:59.49882712579596%, mode : circular 2016-04-07T01:45:39.672-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:45:39.783-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14950000 2016-04-07T01:45:40.051-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7320000 QueuedCount 50000 2016-04-07T01:45:40.24-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14960000 2016-04-07T01:45:40.494-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7330000 QueuedCount 50000 2016-04-07T01:45:40.578-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14970000 2016-04-07T01:45:40.885-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14980000 2016-04-07T01:45:40.968-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7340000 QueuedCount 50000 2016-04-07T01:45:41.373-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 14990000 2016-04-07T01:45:41.456-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7350000 QueuedCount 50000 2016-04-07T01:45:41.649-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15000000 2016-04-07T01:45:41.935-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7360000 QueuedCount 50000 2016-04-07T01:45:41.956-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15010000 2016-04-07T01:45:42.174-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15020000 2016-04-07T01:45:42.428-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7370000 QueuedCount 50000 2016-04-07T01:45:42.451-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15030000 2016-04-07T01:45:42.669-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15040000 2016-04-07T01:45:42.741-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15050000 2016-04-07T01:45:42.752-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15060000 2016-04-07T01:45:42.838-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7380000 QueuedCount 50000 2016-04-07T01:45:43.296-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7390000 QueuedCount 44814 2016-04-07T01:45:43.724-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7400000 QueuedCount 34814 2016-04-07T01:45:44.161-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7410000 QueuedCount 24814 2016-04-07T01:45:44.595-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7420000 QueuedCount 14814 2016-04-07T01:45:45.047-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7430000 QueuedCount 4814 2016-04-07T01:45:45.408-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:45:45.429-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:45:45.429-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:45:45.796-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 41.909µs CommitTime 553.279799ms TotalFlushTime 4m50.137411262s TotalCommitTime 16.731836753s 2016-04-07T01:45:45.796-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 7434814, 7434814, 42 committed:true 2016-04-07T01:45:45.796-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 7434814, 7434814, 42 committed:true) SnapCreateDur 561.385648ms SnapOpenDur 40.354µs 2016-04-07T01:45:46.011-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7440000 QueuedCount 50000 2016-04-07T01:45:46.11-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15070000 2016-04-07T01:45:46.436-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7450000 QueuedCount 50000 2016-04-07T01:45:46.627-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15080000 2016-04-07T01:45:46.876-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7460000 QueuedCount 50000 2016-04-07T01:45:46.907-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15090000 2016-04-07T01:45:47.18-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15100000 2016-04-07T01:45:47.325-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7470000 QueuedCount 50000 2016-04-07T01:45:47.546-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15110000 2016-04-07T01:45:47.761-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7480000 QueuedCount 50000 2016-04-07T01:45:47.884-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15120000 2016-04-07T01:45:48.168-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7490000 QueuedCount 50000 2016-04-07T01:45:48.37-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15130000 2016-04-07T01:45:48.558-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15140000 2016-04-07T01:45:48.649-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7500000 QueuedCount 50000 2016-04-07T01:45:48.984-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15150000 2016-04-07T01:45:49.161-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7510000 QueuedCount 50000 2016-04-07T01:45:49.26-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15160000 2016-04-07T01:45:49.477-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15170000 2016-04-07T01:45:49.544-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15180000 2016-04-07T01:45:49.554-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15190000 2016-04-07T01:45:49.565-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15200000 2016-04-07T01:45:49.573-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15210000 2016-04-07T01:45:49.632-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7520000 QueuedCount 50000 2016-04-07T01:45:50.122-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7530000 QueuedCount 49999 2016-04-07T01:45:50.428-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:45:50.442-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:45:50.442-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:45:50.552-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7540000 QueuedCount 40045 2016-04-07T01:45:50.981-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7550000 QueuedCount 30045 2016-04-07T01:45:51.412-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7560000 QueuedCount 20045 2016-04-07T01:45:51.764-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7570000 QueuedCount 10045 2016-04-07T01:45:52.172-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7580000 QueuedCount 45 2016-04-07T01:45:52.678-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 31.068µs CommitTime 487.906149ms TotalFlushTime 4m56.385683862s TotalCommitTime 17.219742902s 2016-04-07T01:45:52.678-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 7580045, 7580045, 43 committed:true 2016-04-07T01:45:52.679-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 7580045, 7580045, 43 committed:true) SnapCreateDur 495.751446ms SnapOpenDur 46.954µs 2016-04-07T01:45:52.992-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15220000 2016-04-07T01:45:53.052-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7590000 QueuedCount 50000 2016-04-07T01:45:53.472-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7600000 QueuedCount 50000 2016-04-07T01:45:53.569-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15230000 2016-04-07T01:45:53.906-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7610000 QueuedCount 50000 2016-04-07T01:45:53.936-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15240000 2016-04-07T01:45:54.368-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7620000 QueuedCount 50000 2016-04-07T01:45:54.393-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15250000 2016-04-07T01:45:54.728-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15260000 2016-04-07T01:45:54.814-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7630000 QueuedCount 50000 2016-04-07T01:45:55.215-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15270000 2016-04-07T01:45:55.293-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7640000 QueuedCount 50000 2016-04-07T01:45:55.641-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15280000 2016-04-07T01:45:55.805-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7650000 QueuedCount 50000 2016-04-07T01:45:56.037-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15290000 2016-04-07T01:45:56.293-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7660000 QueuedCount 50000 2016-04-07T01:45:56.313-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15300000 2016-04-07T01:45:56.619-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15310000 2016-04-07T01:45:56.629-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15320000 2016-04-07T01:45:56.711-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7670000 QueuedCount 50000 2016-04-07T01:45:56.81-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15330000 2016-04-07T01:45:57.025-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15340000 2016-04-07T01:45:57.212-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7680000 QueuedCount 50000 2016-04-07T01:45:57.241-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15350000 2016-04-07T01:45:57.429-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15360000 2016-04-07T01:45:57.439-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15370000 2016-04-07T01:45:57.449-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15380000 2016-04-07T01:45:57.458-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15390000 2016-04-07T01:45:57.468-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15400000 2016-04-07T01:45:57.478-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15410000 2016-04-07T01:45:57.713-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7690000 QueuedCount 50000 2016-04-07T01:45:58.211-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7700000 QueuedCount 50000 2016-04-07T01:45:58.686-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7710000 QueuedCount 41511 2016-04-07T01:45:59.147-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7720000 QueuedCount 31511 2016-04-07T01:45:59.607-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7730000 QueuedCount 21511 2016-04-07T01:46:00.052-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7740000 QueuedCount 11511 2016-04-07T01:46:00.432-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7750000 QueuedCount 1511 2016-04-07T01:46:01.11-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.131µs CommitTime 577.041544ms TotalFlushTime 5m4.07450179s TotalCommitTime 17.796784446s 2016-04-07T01:46:01.11-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 7751511, 7751511, 44 committed:true 2016-04-07T01:46:01.111-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 7751511, 7751511, 44 committed:true) SnapCreateDur 585.828817ms SnapOpenDur 45.279µs 2016-04-07T01:46:01.137-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15420000 2016-04-07T01:46:01.148-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15430000 2016-04-07T01:46:01.16-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15440000 2016-04-07T01:46:01.169-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15450000 2016-04-07T01:46:01.495-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7760000 QueuedCount 50000 2016-04-07T01:46:01.799-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15460000 2016-04-07T01:46:01.951-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7770000 QueuedCount 50000 2016-04-07T01:46:02.343-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7780000 QueuedCount 50000 2016-04-07T01:46:02.345-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15470000 2016-04-07T01:46:02.791-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7790000 QueuedCount 50000 2016-04-07T01:46:03.373-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15480000 2016-04-07T01:46:03.45-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15490000 2016-04-07T01:46:03.477-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15500000 2016-04-07T01:46:03.492-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7800000 QueuedCount 50000 2016-04-07T01:46:03.507-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15510000 2016-04-07T01:46:03.845-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15520000 2016-04-07T01:46:04.238-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7810000 QueuedCount 50000 2016-04-07T01:46:04.731-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7820000 QueuedCount 50000 2016-04-07T01:46:05.244-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7830000 QueuedCount 50000 2016-04-07T01:46:05.489-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:46:05.506-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:46:05.506-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:46:05.765-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7840000 QueuedCount 50000 2016-04-07T01:46:06.27-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7850000 QueuedCount 50000 2016-04-07T01:46:06.698-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7860000 QueuedCount 50000 2016-04-07T01:46:07.193-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7870000 QueuedCount 46900 2016-04-07T01:46:07.656-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7880000 QueuedCount 36900 2016-04-07T01:46:08.103-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7890000 QueuedCount 26900 2016-04-07T01:46:08.554-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7900000 QueuedCount 16900 2016-04-07T01:46:08.847-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5010 Milliseconds Vbucket 659 2016-04-07T01:46:08.877-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5040 Milliseconds Vbucket 659 2016-04-07T01:46:08.907-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5070 Milliseconds Vbucket 659 2016-04-07T01:46:08.937-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5100 Milliseconds Vbucket 659 2016-04-07T01:46:08.967-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5130 Milliseconds Vbucket 659 2016-04-07T01:46:08.997-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5160 Milliseconds Vbucket 659 2016-04-07T01:46:09.004-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7910000 QueuedCount 6900 2016-04-07T01:46:09.027-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5190 Milliseconds Vbucket 659 2016-04-07T01:46:09.057-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5220 Milliseconds Vbucket 659 2016-04-07T01:46:09.087-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5250 Milliseconds Vbucket 659 2016-04-07T01:46:09.117-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5280 Milliseconds Vbucket 659 2016-04-07T01:46:09.147-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5310 Milliseconds Vbucket 659 2016-04-07T01:46:09.177-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5340 Milliseconds Vbucket 659 2016-04-07T01:46:09.207-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5370 Milliseconds Vbucket 659 2016-04-07T01:46:09.237-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5400 Milliseconds Vbucket 659 2016-04-07T01:46:09.267-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5430 Milliseconds Vbucket 659 2016-04-07T01:46:09.297-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5460 Milliseconds Vbucket 659 2016-04-07T01:46:09.327-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5490 Milliseconds Vbucket 659 2016-04-07T01:46:09.357-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5520 Milliseconds Vbucket 659 2016-04-07T01:46:09.373-07:00 [Info] connected with 1 indexers 2016-04-07T01:46:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T01:46:09.374-07:00 [Info] client load stats {} 2016-04-07T01:46:09.387-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5550 Milliseconds Vbucket 659 2016-04-07T01:46:09.417-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5580 Milliseconds Vbucket 659 2016-04-07T01:46:09.447-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5610 Milliseconds Vbucket 659 2016-04-07T01:46:09.477-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5640 Milliseconds Vbucket 659 2016-04-07T01:46:09.507-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5670 Milliseconds Vbucket 659 2016-04-07T01:46:09.537-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5700 Milliseconds Vbucket 659 2016-04-07T01:46:09.567-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5730 Milliseconds Vbucket 659 2016-04-07T01:46:09.597-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5760 Milliseconds Vbucket 659 2016-04-07T01:46:09.627-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5790 Milliseconds Vbucket 659 2016-04-07T01:46:09.636-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":7595239,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":7786456158,"bucket-1:myindex:avg_ts_items_count":161150,"bucket-1:myindex:build_progress":38,"bucket-1:myindex:data_size":2064899208,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":4703744000,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":0,"bucket-1:myindex:frag_percent":56,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":830900910,"bucket-1:myindex:items_count":7751511,"bucket-1:myindex:num_commits":44,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":7916900,"bucket-1:myindex:num_docs_pending":4477859,"bucket-1:myindex:num_docs_processed":7751511,"bucket-1:myindex:num_docs_queued":7770630,"bucket-1:myindex:num_flush_queued":7916900,"bucket-1:myindex:num_items_flushed":7916900,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":44,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":8431996078,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"44 17796784446 8377667288674831512","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"7916900 46332200675 213347026372044747","bucket-1:myindex:timings/storage_info":"180 1434640 20317073430","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"44 14368975 5219278687079","bucket-1:myindex:timings/storage_meta_set":"45 14724730 5255579431044","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"44 2006427 99496283575","bucket-1:myindex:timings/storage_set":"15833800 261424298973 -5549369191591213393","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":15512139,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"71 115913249 2050968354396105","bucket-1:ts_queue_size":911,"indexer_state":"Active","memory_quota":31457280000,"memory_used":12370196480,"memory_used_storage":4913917952,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"1624 215728061 46802014877651","uptime":"17m1.328266283s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T01:46:09.657-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5820 Milliseconds Vbucket 659 2016-04-07T01:46:09.672-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:2066502689, Disk:4916051968, ExtraSnapshotData:0, Fragmentation:57.96418137050906%, mode : circular 2016-04-07T01:46:09.672-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:46:09.676-07:00 [Info] memstats {"Alloc":5814081312, "TotalAlloc":13996479880, "Sys":9854296392, "Lookups":2703, "Mallocs":366893487,"Frees":238093412, "HeapAlloc":5814081312, "HeapSys":9384886272, "HeapIdle":2226053120, "HeapInuse":7158833152,"HeapReleased":0, "HeapObjects":128800075,"GCSys":297445376, "LastGC":1460018642840087332,"PauseTotalNs":23651282, "PauseNs":[], "NumGC":18} 2016-04-07T01:46:09.687-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5850 Milliseconds Vbucket 659 2016-04-07T01:46:09.717-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5880 Milliseconds Vbucket 659 2016-04-07T01:46:09.747-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5910 Milliseconds Vbucket 659 2016-04-07T01:46:09.777-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5940 Milliseconds Vbucket 659 2016-04-07T01:46:09.807-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5970 Milliseconds Vbucket 659 2016-04-07T01:46:09.837-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6000 Milliseconds Vbucket 659 2016-04-07T01:46:09.867-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6030 Milliseconds Vbucket 659 2016-04-07T01:46:09.897-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6060 Milliseconds Vbucket 659 2016-04-07T01:46:09.92-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 35.818µs CommitTime 616.95478ms TotalFlushTime 5m12.095905072s TotalCommitTime 18.413739226s 2016-04-07T01:46:09.92-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 7916900, 7916900, 45 committed:true 2016-04-07T01:46:09.92-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 7916900, 7916900, 45 committed:true) SnapCreateDur 625.367639ms SnapOpenDur 59.605µs 2016-04-07T01:46:09.947-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15530000 2016-04-07T01:46:10.05-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7920000 QueuedCount 50000 2016-04-07T01:46:10.284-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15540000 2016-04-07T01:46:10.539-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7930000 QueuedCount 50000 2016-04-07T01:46:10.832-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15550000 2016-04-07T01:46:11-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7940000 QueuedCount 50000 2016-04-07T01:46:11.41-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15560000 2016-04-07T01:46:11.472-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7950000 QueuedCount 50000 2016-04-07T01:46:11.968-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7960000 QueuedCount 50000 2016-04-07T01:46:11.988-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15570000 2016-04-07T01:46:12.385-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7970000 QueuedCount 50000 2016-04-07T01:46:12.593-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15580000 2016-04-07T01:46:12.886-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7980000 QueuedCount 50000 2016-04-07T01:46:13.201-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15590000 2016-04-07T01:46:13.402-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 7990000 QueuedCount 50000 2016-04-07T01:46:13.688-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15600000 2016-04-07T01:46:13.905-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8000000 QueuedCount 50000 2016-04-07T01:46:14.237-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15610000 2016-04-07T01:46:14.422-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8010000 QueuedCount 50000 2016-04-07T01:46:14.545-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15620000 2016-04-07T01:46:14.852-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15630000 2016-04-07T01:46:14.934-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8020000 QueuedCount 50000 2016-04-07T01:46:15.443-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8030000 QueuedCount 50000 2016-04-07T01:46:15.52-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:46:15.534-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:46:15.534-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:46:15.957-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8040000 QueuedCount 50000 2016-04-07T01:46:16.37-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8050000 QueuedCount 45884 2016-04-07T01:46:16.851-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8060000 QueuedCount 35884 2016-04-07T01:46:17.304-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8070000 QueuedCount 25884 2016-04-07T01:46:17.765-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8080000 QueuedCount 15884 2016-04-07T01:46:18.218-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8090000 QueuedCount 5884 2016-04-07T01:46:19.178-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.01µs CommitTime 658.023864ms TotalFlushTime 5m20.526215374s TotalCommitTime 19.07176309s 2016-04-07T01:46:19.178-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 8095884, 8095884, 46 committed:true 2016-04-07T01:46:19.178-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 8095884, 8095884, 46 committed:true) SnapCreateDur 666.448644ms SnapOpenDur 72.024µs 2016-04-07T01:46:19.197-07:00 [Info] Indexer::ReadMemstats Time Taken 14.701332ms 2016-04-07T01:46:19.415-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8100000 QueuedCount 50000 2016-04-07T01:46:19.539-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15640000 2016-04-07T01:46:19.827-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8110000 QueuedCount 50000 2016-04-07T01:46:20.318-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8120000 QueuedCount 50000 2016-04-07T01:46:20.506-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15650000 2016-04-07T01:46:20.539-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:46:20.553-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:46:20.553-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:46:20.815-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8130000 QueuedCount 50000 2016-04-07T01:46:21.144-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15660000 2016-04-07T01:46:21.315-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8140000 QueuedCount 50000 2016-04-07T01:46:21.723-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15670000 2016-04-07T01:46:21.805-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8150000 QueuedCount 50000 2016-04-07T01:46:22.209-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15680000 2016-04-07T01:46:22.301-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8160000 QueuedCount 50000 2016-04-07T01:46:22.817-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15690000 2016-04-07T01:46:22.818-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8170000 QueuedCount 50000 2016-04-07T01:46:23.155-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15700000 2016-04-07T01:46:23.34-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8180000 QueuedCount 50000 2016-04-07T01:46:23.641-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15710000 2016-04-07T01:46:23.763-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8190000 QueuedCount 50000 2016-04-07T01:46:23.858-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15720000 2016-04-07T01:46:24.265-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8200000 QueuedCount 50000 2016-04-07T01:46:24.783-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8210000 QueuedCount 50000 2016-04-07T01:46:25.297-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8220000 QueuedCount 42262 2016-04-07T01:46:25.791-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8230000 QueuedCount 32262 2016-04-07T01:46:26.259-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8240000 QueuedCount 22262 2016-04-07T01:46:26.721-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8250000 QueuedCount 12262 2016-04-07T01:46:27.192-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8260000 QueuedCount 2262 2016-04-07T01:46:27.976-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 23.165µs CommitTime 667.104007ms TotalFlushTime 5m28.477515685s TotalCommitTime 19.738867097s 2016-04-07T01:46:27.976-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 8262262, 8262262, 47 committed:true 2016-04-07T01:46:27.976-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 8262262, 8262262, 47 committed:true) SnapCreateDur 675.335077ms SnapOpenDur 60.242µs 2016-04-07T01:46:28.016-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15730000 2016-04-07T01:46:28.031-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15740000 2016-04-07T01:46:28.043-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15750000 2016-04-07T01:46:28.314-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8270000 QueuedCount 46251 2016-04-07T01:46:28.758-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8280000 QueuedCount 36251 2016-04-07T01:46:29.201-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8290000 QueuedCount 26251 2016-04-07T01:46:29.639-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8300000 QueuedCount 16251 2016-04-07T01:46:30.088-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8310000 QueuedCount 6251 2016-04-07T01:46:30.578-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:46:30.592-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:46:30.592-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:46:30.88-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.406µs CommitTime 511.774804ms TotalFlushTime 5m30.821510656s TotalCommitTime 20.250641901s 2016-04-07T01:46:30.88-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 8316251, 8316251, 48 committed:true 2016-04-07T01:46:30.88-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 8316251, 8316251, 48 committed:true) SnapCreateDur 519.765009ms SnapOpenDur 58.882µs 2016-04-07T01:46:30.884-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:30.886-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:30.89-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:30.896-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15760000 2016-04-07T01:46:30.896-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:30.9-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:30.906-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:30.911-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15770000 2016-04-07T01:46:30.924-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15780000 2016-04-07T01:46:30.934-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15790000 2016-04-07T01:46:30.946-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15800000 2016-04-07T01:46:30.956-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15810000 2016-04-07T01:46:31.029-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8320000 QueuedCount 50000 2016-04-07T01:46:31.315-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:31.436-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15820000 2016-04-07T01:46:31.513-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8330000 QueuedCount 50000 2016-04-07T01:46:31.696-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:31.903-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:31.925-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15830000 2016-04-07T01:46:32.011-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8340000 QueuedCount 50000 2016-04-07T01:46:32.111-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:32.293-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:32.501-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8350000 QueuedCount 50000 2016-04-07T01:46:32.801-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15840000 2016-04-07T01:46:33-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8360000 QueuedCount 50000 2016-04-07T01:46:33.501-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8370000 QueuedCount 50000 2016-04-07T01:46:33.62-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15850000 2016-04-07T01:46:33.996-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8380000 QueuedCount 50000 2016-04-07T01:46:34.437-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15860000 2016-04-07T01:46:34.501-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8390000 QueuedCount 50000 2016-04-07T01:46:34.836-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15870000 2016-04-07T01:46:34.94-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8400000 QueuedCount 50000 2016-04-07T01:46:35.461-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8410000 QueuedCount 50000 2016-04-07T01:46:35.991-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8420000 QueuedCount 50000 2016-04-07T01:46:36.013-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15880000 2016-04-07T01:46:36.523-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8430000 QueuedCount 50000 2016-04-07T01:46:36.532-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15890000 2016-04-07T01:46:36.545-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15900000 2016-04-07T01:46:36.756-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15910000 2016-04-07T01:46:37.049-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8440000 QueuedCount 50000 2016-04-07T01:46:37.272-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15920000 2016-04-07T01:46:37.588-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8450000 QueuedCount 50000 2016-04-07T01:46:37.94-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15930000 2016-04-07T01:46:38.119-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8460000 QueuedCount 50000 2016-04-07T01:46:38.646-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8470000 QueuedCount 50000 2016-04-07T01:46:38.667-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15940000 2016-04-07T01:46:39.091-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8480000 QueuedCount 50000 2016-04-07T01:46:39.095-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15950000 2016-04-07T01:46:39.521-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15960000 2016-04-07T01:46:39.615-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8490000 QueuedCount 50000 2016-04-07T01:46:39.673-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:2216413290, Disk:5694390272, ExtraSnapshotData:0, Fragmentation:61.07724999288563%, mode : circular 2016-04-07T01:46:39.673-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:46:39.739-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15970000 2016-04-07T01:46:39.751-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15980000 2016-04-07T01:46:39.762-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 15990000 2016-04-07T01:46:39.853-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16000000 2016-04-07T01:46:39.866-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16010000 2016-04-07T01:46:40.13-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8500000 QueuedCount 50000 2016-04-07T01:46:40.648-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:40.649-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8510000 QueuedCount 50000 2016-04-07T01:46:40.677-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16020000 2016-04-07T01:46:41.156-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8520000 QueuedCount 50000 2016-04-07T01:46:41.463-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16030000 2016-04-07T01:46:41.679-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8530000 QueuedCount 50000 2016-04-07T01:46:41.832-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16040000 2016-04-07T01:46:42.216-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8540000 QueuedCount 50000 2016-04-07T01:46:42.435-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16050000 2016-04-07T01:46:42.74-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8550000 QueuedCount 50000 2016-04-07T01:46:43.174-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8560000 QueuedCount 50000 2016-04-07T01:46:43.372-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16060000 2016-04-07T01:46:43.693-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8570000 QueuedCount 50000 2016-04-07T01:46:44.131-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16070000 2016-04-07T01:46:44.236-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8580000 QueuedCount 50000 2016-04-07T01:46:44.677-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16080000 2016-04-07T01:46:44.769-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8590000 QueuedCount 50000 2016-04-07T01:46:45.224-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16090000 2016-04-07T01:46:45.311-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8600000 QueuedCount 50000 2016-04-07T01:46:45.637-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:46:45.652-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:46:45.652-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:46:45.855-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8610000 QueuedCount 50000 2016-04-07T01:46:45.861-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16100000 2016-04-07T01:46:46.287-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16110000 2016-04-07T01:46:46.386-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8620000 QueuedCount 50000 2016-04-07T01:46:46.505-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16120000 2016-04-07T01:46:46.631-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16130000 2016-04-07T01:46:46.729-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16140000 2016-04-07T01:46:46.907-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8630000 QueuedCount 49016 2016-04-07T01:46:47.41-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8640000 QueuedCount 39016 2016-04-07T01:46:47.809-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8650000 QueuedCount 29016 2016-04-07T01:46:48.286-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8660000 QueuedCount 19016 2016-04-07T01:46:48.756-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8670000 QueuedCount 9016 2016-04-07T01:46:50.019-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 15.183µs CommitTime 779.980167ms TotalFlushTime 5m48.827351697s TotalCommitTime 21.030622068s 2016-04-07T01:46:50.019-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 8679016, 8679016, 49 committed:true 2016-04-07T01:46:50.019-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 8679016, 8679016, 49 committed:true) SnapCreateDur 791.092171ms SnapOpenDur 81.897µs 2016-04-07T01:46:50.033-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8680000 QueuedCount 2324 2016-04-07T01:46:50.264-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.344µs CommitTime 113.121771ms TotalFlushTime 5m48.941387102s TotalCommitTime 21.143743839s 2016-04-07T01:46:50.264-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 8682324, 8682324, 50 committed:true 2016-04-07T01:46:50.264-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 8682324, 8682324, 50 committed:true) SnapCreateDur 122.169203ms SnapOpenDur 64.64µs 2016-04-07T01:46:50.266-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:50.27-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:50.274-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:50.277-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:50.281-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:50.282-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16150000 2016-04-07T01:46:50.285-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:50.289-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:50.293-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:50.297-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16160000 2016-04-07T01:46:50.297-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:50.31-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16170000 2016-04-07T01:46:50.32-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16180000 2016-04-07T01:46:50.33-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16190000 2016-04-07T01:46:50.34-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16200000 2016-04-07T01:46:50.612-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8690000 QueuedCount 50000 2016-04-07T01:46:50.818-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16210000 2016-04-07T01:46:51.109-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8700000 QueuedCount 50000 2016-04-07T01:46:51.547-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16220000 2016-04-07T01:46:51.618-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8710000 QueuedCount 50000 2016-04-07T01:46:52.114-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8720000 QueuedCount 50000 2016-04-07T01:46:52.454-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16230000 2016-04-07T01:46:52.626-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8730000 QueuedCount 50000 2016-04-07T01:46:53.143-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8740000 QueuedCount 50000 2016-04-07T01:46:53.151-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16240000 2016-04-07T01:46:53.694-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8750000 QueuedCount 50000 2016-04-07T01:46:53.939-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16250000 2016-04-07T01:46:54.148-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8760000 QueuedCount 50000 2016-04-07T01:46:54.681-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8770000 QueuedCount 50000 2016-04-07T01:46:54.697-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16260000 2016-04-07T01:46:55.178-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8780000 QueuedCount 50000 2016-04-07T01:46:55.395-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16270000 2016-04-07T01:46:55.675-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:46:55.688-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:46:55.688-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:46:55.718-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8790000 QueuedCount 50000 2016-04-07T01:46:55.732-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16280000 2016-04-07T01:46:55.742-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16290000 2016-04-07T01:46:56.072-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16300000 2016-04-07T01:46:56.245-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8800000 QueuedCount 50000 2016-04-07T01:46:56.679-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16310000 2016-04-07T01:46:56.761-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8810000 QueuedCount 50000 2016-04-07T01:46:56.988-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16320000 2016-04-07T01:46:56.999-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16330000 2016-04-07T01:46:57.284-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8820000 QueuedCount 50000 2016-04-07T01:46:57.733-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:46:57.813-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8830000 QueuedCount 50000 2016-04-07T01:46:58.256-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8840000 QueuedCount 50000 2016-04-07T01:46:58.47-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16340000 2016-04-07T01:46:58.782-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8850000 QueuedCount 50000 2016-04-07T01:46:59.333-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8860000 QueuedCount 50000 2016-04-07T01:46:59.347-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16350000 2016-04-07T01:46:59.867-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8870000 QueuedCount 50000 2016-04-07T01:47:00.414-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8880000 QueuedCount 50000 2016-04-07T01:47:00.435-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16360000 2016-04-07T01:47:00.691-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:47:00.704-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:47:00.704-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:47:00.966-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8890000 QueuedCount 50000 2016-04-07T01:47:01.102-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16370000 2016-04-07T01:47:01.512-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8900000 QueuedCount 50000 2016-04-07T01:47:01.859-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16380000 2016-04-07T01:47:02.062-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8910000 QueuedCount 50000 2016-04-07T01:47:02.407-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16390000 2016-04-07T01:47:02.6-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8920000 QueuedCount 50000 2016-04-07T01:47:03.052-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8930000 QueuedCount 50000 2016-04-07T01:47:03.163-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16400000 2016-04-07T01:47:03.596-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8940000 QueuedCount 50000 2016-04-07T01:47:03.92-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16410000 2016-04-07T01:47:04.142-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8950000 QueuedCount 50000 2016-04-07T01:47:04.497-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16420000 2016-04-07T01:47:04.696-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8960000 QueuedCount 50000 2016-04-07T01:47:04.716-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16430000 2016-04-07T01:47:05.051-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16440000 2016-04-07T01:47:05.24-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8970000 QueuedCount 50000 2016-04-07T01:47:05.476-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16450000 2016-04-07T01:47:05.795-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8980000 QueuedCount 50000 2016-04-07T01:47:06.325-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 8990000 QueuedCount 50000 2016-04-07T01:47:06.833-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9000000 QueuedCount 43249 2016-04-07T01:47:07.251-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9010000 QueuedCount 33249 2016-04-07T01:47:07.749-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9020000 QueuedCount 23249 2016-04-07T01:47:08.231-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9030000 QueuedCount 13249 2016-04-07T01:47:08.713-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9040000 QueuedCount 3249 2016-04-07T01:47:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:47:09.376-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T01:47:09.376-07:00 [Info] client load stats {} 2016-04-07T01:47:09.636-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":7404792,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":6368485292,"bucket-1:myindex:avg_ts_items_count":120187,"bucket-1:myindex:build_progress":44,"bucket-1:myindex:data_size":2363737836,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":6449451008,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":0,"bucket-1:myindex:frag_percent":63,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":949541145,"bucket-1:myindex:items_count":8682324,"bucket-1:myindex:num_commits":50,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":9043249,"bucket-1:myindex:num_docs_pending":3541957,"bucket-1:myindex:num_docs_processed":8879712,"bucket-1:myindex:num_docs_queued":7578331,"bucket-1:myindex:num_flush_queued":9043249,"bucket-1:myindex:num_items_flushed":9043249,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":50,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":244507348,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"50 21143743839 -7927341376724162413","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"9043249 53436163340 213394736696920294","bucket-1:myindex:timings/storage_info":"204 1626133 23202975371","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"50 15926463 5650828652413","bucket-1:myindex:timings/storage_meta_set":"51 16730685 5970699260835","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"50 2280004 112037393694","bucket-1:myindex:timings/storage_set":"18086498 308829520919 -2453901854434491175","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":16448041,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"83 127813477 2066268925341499","bucket-1:ts_queue_size":1088,"indexer_state":"Active","memory_quota":31457280000,"memory_used":12995958784,"memory_used_storage":6449876992,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"1766 244476803 52764725916789","uptime":"18m1.329543665s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T01:47:09.673-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:2363737836, Disk:6449451008, ExtraSnapshotData:0, Fragmentation:63.34978228274031%, mode : circular 2016-04-07T01:47:09.673-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:47:09.676-07:00 [Info] memstats {"Alloc":4270298304, "TotalAlloc":14631314864, "Sys":9854296392, "Lookups":3141, "Mallocs":386807284,"Frees":292700451, "HeapAlloc":4270298304, "HeapSys":9385181184, "HeapIdle":3136544768, "HeapInuse":6248636416,"HeapReleased":0, "HeapObjects":94106833,"GCSys":297445376, "LastGC":1460018763516325336,"PauseTotalNs":24663451, "PauseNs":[1012169], "NumGC":19} 2016-04-07T01:47:09.719-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 15.559µs CommitTime 800.641891ms TotalFlushTime 6m7.231351101s TotalCommitTime 21.94438573s 2016-04-07T01:47:09.719-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 9043249, 9043249, 51 committed:true 2016-04-07T01:47:09.719-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 9043249, 9043249, 51 committed:true) SnapCreateDur 809.43882ms SnapOpenDur 74.328µs 2016-04-07T01:47:09.774-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16460000 2016-04-07T01:47:10.051-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9050000 QueuedCount 50000 2016-04-07T01:47:10.081-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16470000 2016-04-07T01:47:10.564-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9060000 QueuedCount 50000 2016-04-07T01:47:10.687-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16480000 2016-04-07T01:47:11.078-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9070000 QueuedCount 50000 2016-04-07T01:47:11.115-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16490000 2016-04-07T01:47:11.515-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9080000 QueuedCount 50000 2016-04-07T01:47:11.632-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16500000 2016-04-07T01:47:12.03-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16510000 2016-04-07T01:47:12.04-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9090000 QueuedCount 50000 2016-04-07T01:47:12.457-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16520000 2016-04-07T01:47:12.559-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9100000 QueuedCount 50000 2016-04-07T01:47:12.884-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16530000 2016-04-07T01:47:13.09-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9110000 QueuedCount 50000 2016-04-07T01:47:13.22-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16540000 2016-04-07T01:47:13.624-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9120000 QueuedCount 50000 2016-04-07T01:47:14.165-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9130000 QueuedCount 50000 2016-04-07T01:47:14.698-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9140000 QueuedCount 50000 2016-04-07T01:47:15.235-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9150000 QueuedCount 50000 2016-04-07T01:47:15.658-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9160000 QueuedCount 41943 2016-04-07T01:47:15.754-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:47:15.767-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:47:15.767-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:47:16.148-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9170000 QueuedCount 31943 2016-04-07T01:47:16.606-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9180000 QueuedCount 21943 2016-04-07T01:47:17.078-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9190000 QueuedCount 11943 2016-04-07T01:47:17.556-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9200000 QueuedCount 1943 2016-04-07T01:47:18.297-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 40.54µs CommitTime 636.777129ms TotalFlushTime 6m14.995967152s TotalCommitTime 22.581162859s 2016-04-07T01:47:18.297-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 9201943, 9201943, 52 committed:true 2016-04-07T01:47:18.297-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 9201943, 9201943, 52 committed:true) SnapCreateDur 645.205424ms SnapOpenDur 71.148µs 2016-04-07T01:47:18.444-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5010 Milliseconds Vbucket 592 2016-04-07T01:47:18.474-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5040 Milliseconds Vbucket 592 2016-04-07T01:47:18.504-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5070 Milliseconds Vbucket 592 2016-04-07T01:47:18.534-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5100 Milliseconds Vbucket 592 2016-04-07T01:47:18.564-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5130 Milliseconds Vbucket 592 2016-04-07T01:47:18.594-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5160 Milliseconds Vbucket 592 2016-04-07T01:47:18.624-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5190 Milliseconds Vbucket 592 2016-04-07T01:47:18.643-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9210000 QueuedCount 50000 2016-04-07T01:47:18.654-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5220 Milliseconds Vbucket 592 2016-04-07T01:47:18.684-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5250 Milliseconds Vbucket 592 2016-04-07T01:47:18.714-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5280 Milliseconds Vbucket 592 2016-04-07T01:47:18.744-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5310 Milliseconds Vbucket 592 2016-04-07T01:47:18.774-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5340 Milliseconds Vbucket 592 2016-04-07T01:47:18.804-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5370 Milliseconds Vbucket 592 2016-04-07T01:47:18.834-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5400 Milliseconds Vbucket 592 2016-04-07T01:47:18.864-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5430 Milliseconds Vbucket 592 2016-04-07T01:47:18.894-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5460 Milliseconds Vbucket 592 2016-04-07T01:47:18.924-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5490 Milliseconds Vbucket 592 2016-04-07T01:47:18.954-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5520 Milliseconds Vbucket 592 2016-04-07T01:47:18.984-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5550 Milliseconds Vbucket 592 2016-04-07T01:47:19.014-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5580 Milliseconds Vbucket 592 2016-04-07T01:47:19.044-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5610 Milliseconds Vbucket 592 2016-04-07T01:47:19.074-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5640 Milliseconds Vbucket 592 2016-04-07T01:47:19.104-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5670 Milliseconds Vbucket 592 2016-04-07T01:47:19.134-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5700 Milliseconds Vbucket 592 2016-04-07T01:47:19.135-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9220000 QueuedCount 50000 2016-04-07T01:47:19.164-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5730 Milliseconds Vbucket 592 2016-04-07T01:47:19.194-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5760 Milliseconds Vbucket 592 2016-04-07T01:47:19.224-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5790 Milliseconds Vbucket 592 2016-04-07T01:47:19.254-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5820 Milliseconds Vbucket 592 2016-04-07T01:47:19.284-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5850 Milliseconds Vbucket 592 2016-04-07T01:47:19.314-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5880 Milliseconds Vbucket 592 2016-04-07T01:47:19.344-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5910 Milliseconds Vbucket 592 2016-04-07T01:47:19.374-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5940 Milliseconds Vbucket 592 2016-04-07T01:47:19.404-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5970 Milliseconds Vbucket 592 2016-04-07T01:47:19.434-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6000 Milliseconds Vbucket 592 2016-04-07T01:47:19.464-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6030 Milliseconds Vbucket 592 2016-04-07T01:47:19.494-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6060 Milliseconds Vbucket 592 2016-04-07T01:47:19.524-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6090 Milliseconds Vbucket 592 2016-04-07T01:47:19.554-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6120 Milliseconds Vbucket 592 2016-04-07T01:47:19.598-07:00 [Info] Indexer::ReadMemstats Time Taken 17.477127ms 2016-04-07T01:47:19.598-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6150 Milliseconds Vbucket 592 2016-04-07T01:47:19.614-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6180 Milliseconds Vbucket 592 2016-04-07T01:47:19.635-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9230000 QueuedCount 49196 2016-04-07T01:47:19.644-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6210 Milliseconds Vbucket 592 2016-04-07T01:47:19.674-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6240 Milliseconds Vbucket 592 2016-04-07T01:47:19.704-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6270 Milliseconds Vbucket 592 2016-04-07T01:47:19.734-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6300 Milliseconds Vbucket 592 2016-04-07T01:47:19.764-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6330 Milliseconds Vbucket 592 2016-04-07T01:47:19.794-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6360 Milliseconds Vbucket 592 2016-04-07T01:47:19.824-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6390 Milliseconds Vbucket 592 2016-04-07T01:47:19.854-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6420 Milliseconds Vbucket 592 2016-04-07T01:47:19.884-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6450 Milliseconds Vbucket 592 2016-04-07T01:47:19.914-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6480 Milliseconds Vbucket 592 2016-04-07T01:47:19.944-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6510 Milliseconds Vbucket 592 2016-04-07T01:47:19.974-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6540 Milliseconds Vbucket 592 2016-04-07T01:47:20.004-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6570 Milliseconds Vbucket 592 2016-04-07T01:47:20.034-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6600 Milliseconds Vbucket 592 2016-04-07T01:47:20.064-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6630 Milliseconds Vbucket 592 2016-04-07T01:47:20.091-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9240000 QueuedCount 39196 2016-04-07T01:47:20.094-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6660 Milliseconds Vbucket 592 2016-04-07T01:47:20.124-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6690 Milliseconds Vbucket 592 2016-04-07T01:47:20.154-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6720 Milliseconds Vbucket 592 2016-04-07T01:47:20.184-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6750 Milliseconds Vbucket 592 2016-04-07T01:47:20.214-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6780 Milliseconds Vbucket 592 2016-04-07T01:47:20.244-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6810 Milliseconds Vbucket 592 2016-04-07T01:47:20.274-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6840 Milliseconds Vbucket 592 2016-04-07T01:47:20.304-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6870 Milliseconds Vbucket 592 2016-04-07T01:47:20.334-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6900 Milliseconds Vbucket 592 2016-04-07T01:47:20.364-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6930 Milliseconds Vbucket 592 2016-04-07T01:47:20.394-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6960 Milliseconds Vbucket 592 2016-04-07T01:47:20.424-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6990 Milliseconds Vbucket 592 2016-04-07T01:47:20.454-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7020 Milliseconds Vbucket 592 2016-04-07T01:47:20.484-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7050 Milliseconds Vbucket 592 2016-04-07T01:47:20.514-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7080 Milliseconds Vbucket 592 2016-04-07T01:47:20.529-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9250000 QueuedCount 29196 2016-04-07T01:47:20.544-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7110 Milliseconds Vbucket 592 2016-04-07T01:47:20.574-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7140 Milliseconds Vbucket 592 2016-04-07T01:47:20.604-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7170 Milliseconds Vbucket 592 2016-04-07T01:47:20.634-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7200 Milliseconds Vbucket 592 2016-04-07T01:47:20.664-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7230 Milliseconds Vbucket 592 2016-04-07T01:47:20.694-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7260 Milliseconds Vbucket 592 2016-04-07T01:47:20.724-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7290 Milliseconds Vbucket 592 2016-04-07T01:47:20.754-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7320 Milliseconds Vbucket 592 2016-04-07T01:47:20.773-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:47:20.784-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7350 Milliseconds Vbucket 592 2016-04-07T01:47:20.786-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:47:20.786-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:47:20.814-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7380 Milliseconds Vbucket 592 2016-04-07T01:47:20.844-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7410 Milliseconds Vbucket 592 2016-04-07T01:47:20.874-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7440 Milliseconds Vbucket 592 2016-04-07T01:47:20.904-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7470 Milliseconds Vbucket 592 2016-04-07T01:47:20.934-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7500 Milliseconds Vbucket 592 2016-04-07T01:47:20.964-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7530 Milliseconds Vbucket 592 2016-04-07T01:47:20.984-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9260000 QueuedCount 19196 2016-04-07T01:47:20.994-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7560 Milliseconds Vbucket 592 2016-04-07T01:47:21.024-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7590 Milliseconds Vbucket 592 2016-04-07T01:47:21.054-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7620 Milliseconds Vbucket 592 2016-04-07T01:47:21.084-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7650 Milliseconds Vbucket 592 2016-04-07T01:47:21.114-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7680 Milliseconds Vbucket 592 2016-04-07T01:47:21.144-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7710 Milliseconds Vbucket 592 2016-04-07T01:47:21.174-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7740 Milliseconds Vbucket 592 2016-04-07T01:47:21.204-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7770 Milliseconds Vbucket 592 2016-04-07T01:47:21.234-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7800 Milliseconds Vbucket 592 2016-04-07T01:47:21.264-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7830 Milliseconds Vbucket 592 2016-04-07T01:47:21.294-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7860 Milliseconds Vbucket 592 2016-04-07T01:47:21.324-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7890 Milliseconds Vbucket 592 2016-04-07T01:47:21.354-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7920 Milliseconds Vbucket 592 2016-04-07T01:47:21.384-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7950 Milliseconds Vbucket 592 2016-04-07T01:47:21.414-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7980 Milliseconds Vbucket 592 2016-04-07T01:47:21.429-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9270000 QueuedCount 9196 2016-04-07T01:47:21.444-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8010 Milliseconds Vbucket 592 2016-04-07T01:47:21.474-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8040 Milliseconds Vbucket 592 2016-04-07T01:47:21.504-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8070 Milliseconds Vbucket 592 2016-04-07T01:47:21.534-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8100 Milliseconds Vbucket 592 2016-04-07T01:47:21.564-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8130 Milliseconds Vbucket 592 2016-04-07T01:47:21.594-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8160 Milliseconds Vbucket 592 2016-04-07T01:47:21.624-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8190 Milliseconds Vbucket 592 2016-04-07T01:47:21.654-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8220 Milliseconds Vbucket 592 2016-04-07T01:47:21.684-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8250 Milliseconds Vbucket 592 2016-04-07T01:47:21.714-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8280 Milliseconds Vbucket 592 2016-04-07T01:47:21.744-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8310 Milliseconds Vbucket 592 2016-04-07T01:47:21.774-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8340 Milliseconds Vbucket 592 2016-04-07T01:47:21.804-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8370 Milliseconds Vbucket 592 2016-04-07T01:47:21.834-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8400 Milliseconds Vbucket 592 2016-04-07T01:47:21.864-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8430 Milliseconds Vbucket 592 2016-04-07T01:47:21.894-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8460 Milliseconds Vbucket 592 2016-04-07T01:47:21.924-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8490 Milliseconds Vbucket 592 2016-04-07T01:47:21.954-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8520 Milliseconds Vbucket 592 2016-04-07T01:47:21.984-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8550 Milliseconds Vbucket 592 2016-04-07T01:47:22.014-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8580 Milliseconds Vbucket 592 2016-04-07T01:47:22.044-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8610 Milliseconds Vbucket 592 2016-04-07T01:47:22.074-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8640 Milliseconds Vbucket 592 2016-04-07T01:47:22.104-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8670 Milliseconds Vbucket 592 2016-04-07T01:47:22.134-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8700 Milliseconds Vbucket 592 2016-04-07T01:47:22.164-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8730 Milliseconds Vbucket 592 2016-04-07T01:47:22.194-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8760 Milliseconds Vbucket 592 2016-04-07T01:47:22.224-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8790 Milliseconds Vbucket 592 2016-04-07T01:47:22.254-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8820 Milliseconds Vbucket 592 2016-04-07T01:47:22.284-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8850 Milliseconds Vbucket 592 2016-04-07T01:47:22.314-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8880 Milliseconds Vbucket 592 2016-04-07T01:47:22.337-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 34.015µs CommitTime 542.114322ms TotalFlushTime 6m18.408681783s TotalCommitTime 23.123277181s 2016-04-07T01:47:22.337-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 9279196, 9279196, 53 committed:true 2016-04-07T01:47:22.337-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 9279196, 9279196, 53 committed:true) SnapCreateDur 550.730275ms SnapOpenDur 81.327µs 2016-04-07T01:47:22.34-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:22.343-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:22.358-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9280000 QueuedCount 30987 2016-04-07T01:47:22.361-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:22.365-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:22.37-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:22.376-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16550000 2016-04-07T01:47:22.387-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16560000 2016-04-07T01:47:22.396-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16570000 2016-04-07T01:47:22.406-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16580000 2016-04-07T01:47:22.415-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16590000 2016-04-07T01:47:22.476-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16600000 2016-04-07T01:47:22.857-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9290000 QueuedCount 50000 2016-04-07T01:47:22.96-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:22.963-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16610000 2016-04-07T01:47:23.24-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16620000 2016-04-07T01:47:23.33-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9300000 QueuedCount 50000 2016-04-07T01:47:23.346-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:23.819-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9310000 QueuedCount 45394 2016-04-07T01:47:24.2-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9320000 QueuedCount 35394 2016-04-07T01:47:24.663-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9330000 QueuedCount 25394 2016-04-07T01:47:25.127-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9340000 QueuedCount 15394 2016-04-07T01:47:25.576-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9350000 QueuedCount 5394 2016-04-07T01:47:27.258-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 31.473µs CommitTime 1.398508769s TotalFlushTime 6m21.854563759s TotalCommitTime 24.52178595s 2016-04-07T01:47:27.258-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 9355394, 9355394, 54 committed:true 2016-04-07T01:47:27.258-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 9355394, 9355394, 54 committed:true) SnapCreateDur 1.406939661s SnapOpenDur 60.008µs 2016-04-07T01:47:27.26-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:27.261-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:27.264-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:27.266-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:27.269-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:27.275-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:27.277-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:27.281-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:27.292-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:27.299-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16630000 2016-04-07T01:47:27.443-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9360000 QueuedCount 50000 2016-04-07T01:47:27.857-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9370000 QueuedCount 50000 2016-04-07T01:47:28.324-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9380000 QueuedCount 50000 2016-04-07T01:47:28.722-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9390000 QueuedCount 50000 2016-04-07T01:47:29.192-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9400000 QueuedCount 50000 2016-04-07T01:47:29.385-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:29.673-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9410000 QueuedCount 45551 2016-04-07T01:47:30.12-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9420000 QueuedCount 35551 2016-04-07T01:47:30.578-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9430000 QueuedCount 25551 2016-04-07T01:47:30.81-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:47:30.823-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:47:30.823-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:47:31.044-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9440000 QueuedCount 15551 2016-04-07T01:47:31.524-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9450000 QueuedCount 5551 2016-04-07T01:47:32.507-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 38.738µs CommitTime 775.635281ms TotalFlushTime 6m26.229516149s TotalCommitTime 25.297421231s 2016-04-07T01:47:32.507-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 9455551, 9455551, 55 committed:true 2016-04-07T01:47:32.507-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 9455551, 9455551, 55 committed:true) SnapCreateDur 784.120274ms SnapOpenDur 63.789µs 2016-04-07T01:47:32.511-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:32.514-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:32.517-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:32.521-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:32.524-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:32.525-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16640000 2016-04-07T01:47:32.528-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:32.53-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:32.534-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:32.536-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:32.54-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16650000 2016-04-07T01:47:32.542-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:32.552-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16660000 2016-04-07T01:47:32.562-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16670000 2016-04-07T01:47:32.689-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9460000 QueuedCount 50000 2016-04-07T01:47:32.967-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:33.134-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9470000 QueuedCount 50000 2016-04-07T01:47:33.236-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:33.441-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:33.532-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9480000 QueuedCount 50000 2016-04-07T01:47:33.787-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:33.881-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:33.963-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9490000 QueuedCount 50000 2016-04-07T01:47:33.972-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16680000 2016-04-07T01:47:34.337-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16690000 2016-04-07T01:47:34.425-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9500000 QueuedCount 50000 2016-04-07T01:47:34.824-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16700000 2016-04-07T01:47:34.895-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9510000 QueuedCount 50000 2016-04-07T01:47:35.079-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:35.383-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9520000 QueuedCount 45632 2016-04-07T01:47:35.84-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9530000 QueuedCount 35632 2016-04-07T01:47:36.289-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9540000 QueuedCount 25632 2016-04-07T01:47:36.754-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9550000 QueuedCount 15632 2016-04-07T01:47:37.135-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9560000 QueuedCount 5632 2016-04-07T01:47:38.079-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 33.186µs CommitTime 642.783557ms TotalFlushTime 6m31.04926378s TotalCommitTime 25.940204788s 2016-04-07T01:47:38.079-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 9565632, 9565632, 56 committed:true 2016-04-07T01:47:38.079-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 9565632, 9565632, 56 committed:true) SnapCreateDur 651.418305ms SnapOpenDur 47.712µs 2016-04-07T01:47:38.08-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:38.081-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:38.084-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:38.085-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:38.088-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:38.09-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:38.092-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:38.094-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:38.095-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:38.097-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:38.099-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:38.102-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:38.104-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16710000 2016-04-07T01:47:38.105-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:38.109-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:38.112-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:38.114-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:38.265-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9570000 QueuedCount 50000 2016-04-07T01:47:38.354-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:38.38-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16720000 2016-04-07T01:47:38.54-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:38.685-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9580000 QueuedCount 50000 2016-04-07T01:47:38.714-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16730000 2016-04-07T01:47:38.777-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:38.946-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:39.047-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:39.05-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16740000 2016-04-07T01:47:39.052-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9590000 QueuedCount 50000 2016-04-07T01:47:39.227-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:39.505-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:39.511-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9600000 QueuedCount 50000 2016-04-07T01:47:39.537-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16750000 2016-04-07T01:47:39.674-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:2518741566, Disk:7278686208, ExtraSnapshotData:0, Fragmentation:65.39565666079062%, mode : circular 2016-04-07T01:47:39.674-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:47:39.784-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:39.98-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9610000 QueuedCount 50000 2016-04-07T01:47:40.073-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:40.454-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:40.457-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9620000 QueuedCount 50000 2016-04-07T01:47:40.854-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:47:40.869-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:47:40.869-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:47:40.944-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9630000 QueuedCount 50000 2016-04-07T01:47:41.435-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9640000 QueuedCount 50000 2016-04-07T01:47:41.923-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9650000 QueuedCount 50000 2016-04-07T01:47:42.407-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9660000 QueuedCount 43011 2016-04-07T01:47:42.782-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9670000 QueuedCount 33011 2016-04-07T01:47:43.295-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9680000 QueuedCount 23011 2016-04-07T01:47:43.786-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9690000 QueuedCount 13011 2016-04-07T01:47:44.251-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9700000 QueuedCount 3011 2016-04-07T01:47:44.702-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5010 Milliseconds Vbucket 632 2016-04-07T01:47:44.732-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5040 Milliseconds Vbucket 632 2016-04-07T01:47:44.762-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5070 Milliseconds Vbucket 632 2016-04-07T01:47:44.792-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5100 Milliseconds Vbucket 632 2016-04-07T01:47:44.822-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5130 Milliseconds Vbucket 632 2016-04-07T01:47:44.852-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5160 Milliseconds Vbucket 632 2016-04-07T01:47:44.882-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5190 Milliseconds Vbucket 632 2016-04-07T01:47:44.912-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5220 Milliseconds Vbucket 632 2016-04-07T01:47:44.942-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5250 Milliseconds Vbucket 632 2016-04-07T01:47:44.972-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5280 Milliseconds Vbucket 632 2016-04-07T01:47:45.002-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5310 Milliseconds Vbucket 632 2016-04-07T01:47:45.032-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5340 Milliseconds Vbucket 632 2016-04-07T01:47:45.062-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 32.448µs CommitTime 623.367874ms TotalFlushTime 6m37.276350253s TotalCommitTime 26.563572662s 2016-04-07T01:47:45.062-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 9703011, 9703011, 57 committed:true 2016-04-07T01:47:45.062-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5370 Milliseconds Vbucket 632 2016-04-07T01:47:45.062-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 9703011, 9703011, 57 committed:true) SnapCreateDur 632.750486ms SnapOpenDur 60.636µs 2016-04-07T01:47:45.067-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:45.072-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:45.075-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:45.079-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:45.08-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:45.083-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:45.087-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:45.089-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:45.093-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:45.094-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16760000 2016-04-07T01:47:45.096-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:45.103-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:45.105-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16770000 2016-04-07T01:47:45.116-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16780000 2016-04-07T01:47:45.127-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16790000 2016-04-07T01:47:45.137-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16800000 2016-04-07T01:47:45.33-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:45.334-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9710000 QueuedCount 50000 2016-04-07T01:47:45.594-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:45.759-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9720000 QueuedCount 48607 2016-04-07T01:47:46.185-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9730000 QueuedCount 38607 2016-04-07T01:47:46.61-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9740000 QueuedCount 28607 2016-04-07T01:47:46.96-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9750000 QueuedCount 18607 2016-04-07T01:47:47.38-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9760000 QueuedCount 8607 2016-04-07T01:47:48.318-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 15.94µs CommitTime 500.129767ms TotalFlushTime 6m39.968110647s TotalCommitTime 27.063702429s 2016-04-07T01:47:48.318-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 9768607, 9768607, 58 committed:true 2016-04-07T01:47:48.318-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 9768607, 9768607, 58 committed:true) SnapCreateDur 508.271615ms SnapOpenDur 46.379µs 2016-04-07T01:47:48.321-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:48.324-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:48.328-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:48.332-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:48.337-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:48.342-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:48.351-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16810000 2016-04-07T01:47:48.353-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9770000 QueuedCount 50000 2016-04-07T01:47:48.363-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16820000 2016-04-07T01:47:48.771-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9780000 QueuedCount 50000 2016-04-07T01:47:49.183-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9790000 QueuedCount 50000 2016-04-07T01:47:49.639-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9800000 QueuedCount 50000 2016-04-07T01:47:50.093-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9810000 QueuedCount 50000 2016-04-07T01:47:50.191-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:50.373-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:50.547-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9820000 QueuedCount 50000 2016-04-07T01:47:50.892-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:47:50.905-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:47:50.905-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:47:50.956-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:50.968-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9830000 QueuedCount 50000 2016-04-07T01:47:51.061-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16830000 2016-04-07T01:47:51.447-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9840000 QueuedCount 46961 2016-04-07T01:47:51.905-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9850000 QueuedCount 36961 2016-04-07T01:47:52.317-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9860000 QueuedCount 26961 2016-04-07T01:47:52.735-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9870000 QueuedCount 16961 2016-04-07T01:47:53.166-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9880000 QueuedCount 6961 2016-04-07T01:47:53.99-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 16.15µs CommitTime 545.582415ms TotalFlushTime 6m44.980822234s TotalCommitTime 27.609284844s 2016-04-07T01:47:53.99-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 9886961, 9886961, 59 committed:true 2016-04-07T01:47:53.99-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 9886961, 9886961, 59 committed:true) SnapCreateDur 554.015143ms SnapOpenDur 64.839µs 2016-04-07T01:47:53.998-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:54-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:54.002-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:54.004-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:54.007-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:54.009-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:54.013-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:54.016-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16840000 2016-04-07T01:47:54.018-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:54.03-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16850000 2016-04-07T01:47:54.087-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:54.099-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9890000 QueuedCount 50000 2016-04-07T01:47:54.191-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:54.443-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:54.53-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9900000 QueuedCount 50000 2016-04-07T01:47:54.615-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:54.793-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:54.951-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9910000 QueuedCount 50000 2016-04-07T01:47:54.971-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:54.99-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16860000 2016-04-07T01:47:55.319-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:55.406-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9920000 QueuedCount 50000 2016-04-07T01:47:55.536-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16870000 2016-04-07T01:47:55.863-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9930000 QueuedCount 50000 2016-04-07T01:47:55.869-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:55.964-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16880000 2016-04-07T01:47:56.26-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:56.279-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9940000 QueuedCount 50000 2016-04-07T01:47:56.48-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16890000 2016-04-07T01:47:56.765-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9950000 QueuedCount 50000 2016-04-07T01:47:57.228-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9960000 QueuedCount 41414 2016-04-07T01:47:57.678-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9970000 QueuedCount 31414 2016-04-07T01:47:58.14-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9980000 QueuedCount 21414 2016-04-07T01:47:58.567-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 9990000 QueuedCount 11414 2016-04-07T01:47:59.002-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10000000 QueuedCount 1414 2016-04-07T01:47:59.596-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.571µs CommitTime 568.58121ms TotalFlushTime 6m49.91857873s TotalCommitTime 28.177866054s 2016-04-07T01:47:59.597-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 10001414, 10001414, 60 committed:true 2016-04-07T01:47:59.597-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 10001414, 10001414, 60 committed:true) SnapCreateDur 577.304038ms SnapOpenDur 43.758µs 2016-04-07T01:47:59.6-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:59.608-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16900000 2016-04-07T01:47:59.609-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:47:59.648-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16910000 2016-04-07T01:47:59.663-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16920000 2016-04-07T01:47:59.975-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10010000 QueuedCount 50000 2016-04-07T01:48:00.061-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:00.072-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:00.25-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:00.415-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10020000 QueuedCount 50000 2016-04-07T01:48:00.422-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:00.512-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:00.697-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:00.816-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10030000 QueuedCount 50000 2016-04-07T01:48:00.928-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:48:00.941-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:48:00.941-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:48:00.995-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:01.011-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:01.289-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10040000 QueuedCount 50000 2016-04-07T01:48:01.364-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:01.743-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10050000 QueuedCount 47142 2016-04-07T01:48:02.202-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10060000 QueuedCount 37142 2016-04-07T01:48:02.653-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10070000 QueuedCount 27142 2016-04-07T01:48:03.106-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10080000 QueuedCount 17142 2016-04-07T01:48:03.61-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10090000 QueuedCount 7142 2016-04-07T01:48:05.094-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5010 Milliseconds Vbucket 672 2016-04-07T01:48:05.124-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5040 Milliseconds Vbucket 672 2016-04-07T01:48:05.154-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5070 Milliseconds Vbucket 672 2016-04-07T01:48:05.184-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5100 Milliseconds Vbucket 672 2016-04-07T01:48:05.214-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5130 Milliseconds Vbucket 672 2016-04-07T01:48:05.244-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5160 Milliseconds Vbucket 672 2016-04-07T01:48:05.274-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5190 Milliseconds Vbucket 672 2016-04-07T01:48:05.304-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5220 Milliseconds Vbucket 672 2016-04-07T01:48:05.334-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5250 Milliseconds Vbucket 672 2016-04-07T01:48:05.339-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 36.218µs CommitTime 604.624338ms TotalFlushTime 6m54.27910119s TotalCommitTime 28.782490392s 2016-04-07T01:48:05.339-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 10097142, 10097142, 61 committed:true 2016-04-07T01:48:05.339-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 10097142, 10097142, 61 committed:true) SnapCreateDur 1.297388143s SnapOpenDur 55.408µs 2016-04-07T01:48:05.343-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:05.35-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:05.353-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:05.36-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:05.361-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:05.366-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:05.371-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:05.372-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16930000 2016-04-07T01:48:05.409-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16940000 2016-04-07T01:48:05.449-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10100000 QueuedCount 50000 2016-04-07T01:48:05.543-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:05.56-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16950000 2016-04-07T01:48:05.789-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:05.876-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10110000 QueuedCount 50000 2016-04-07T01:48:06.155-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:06.306-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10120000 QueuedCount 50000 2016-04-07T01:48:06.497-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:06.587-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16960000 2016-04-07T01:48:06.775-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10130000 QueuedCount 50000 2016-04-07T01:48:07.189-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10140000 QueuedCount 50000 2016-04-07T01:48:07.671-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10150000 QueuedCount 50000 2016-04-07T01:48:08.164-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10160000 QueuedCount 50000 2016-04-07T01:48:08.667-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10170000 QueuedCount 50000 2016-04-07T01:48:09.131-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10180000 QueuedCount 40183 2016-04-07T01:48:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:48:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T01:48:09.374-07:00 [Info] client load stats {} 2016-04-07T01:48:09.568-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10190000 QueuedCount 30183 2016-04-07T01:48:09.637-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":6731792,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":5579743445,"bucket-1:myindex:avg_ts_items_count":103005,"bucket-1:myindex:build_progress":50,"bucket-1:myindex:data_size":2661507237,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":7278686208,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":28811,"bucket-1:myindex:frag_percent":63,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":1060199910,"bucket-1:myindex:items_count":10001414,"bucket-1:myindex:num_commits":61,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":10191372,"bucket-1:myindex:num_docs_pending":3074641,"bucket-1:myindex:num_docs_processed":10090395,"bucket-1:myindex:num_docs_queued":6834964,"bucket-1:myindex:num_flush_queued":10220183,"bucket-1:myindex:num_items_flushed":10191372,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":61,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":5742409001,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"61 28782490392 -1991100677514698902","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"10191373 59979594782 213436247137824128","bucket-1:myindex:timings/storage_info":"245 2014659 29867791899","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"60 20073482 7509884266354","bucket-1:myindex:timings/storage_meta_set":"61 20797963 7711662801733","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"61 2750900 133643225444","bucket-1:myindex:timings/storage_set":"20382745 352880385745 175167454953357373","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":16951975,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"94 137151097 2074251124998501","bucket-1:ts_queue_size":1026,"indexer_state":"Active","memory_quota":31457280000,"memory_used":14109689856,"memory_used_storage":7279116288,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"1908 273072859 58939228987517","uptime":"19m1.329938069s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T01:48:09.674-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:2686779858, Disk:7278686208, ExtraSnapshotData:0, Fragmentation:63.08702173412886%, mode : circular 2016-04-07T01:48:09.674-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:48:09.676-07:00 [Info] memstats {"Alloc":4840739136, "TotalAlloc":15201755696, "Sys":9854296392, "Lookups":3573, "Mallocs":404739772,"Frees":295911957, "HeapAlloc":4840739136, "HeapSys":9384656896, "HeapIdle":2851528704, "HeapInuse":6533128192,"HeapReleased":0, "HeapObjects":108827815,"GCSys":297445376, "LastGC":1460018763516325336,"PauseTotalNs":24663451, "PauseNs":[], "NumGC":19} 2016-04-07T01:48:10.004-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10200000 QueuedCount 20183 2016-04-07T01:48:10.444-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10210000 QueuedCount 10183 2016-04-07T01:48:10.809-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10220000 QueuedCount 183 2016-04-07T01:48:11.435-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.595µs CommitTime 546.370092ms TotalFlushTime 6m59.714502725s TotalCommitTime 29.328860484s 2016-04-07T01:48:11.435-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 10220183, 10220183, 62 committed:true 2016-04-07T01:48:11.435-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 10220183, 10220183, 62 committed:true) SnapCreateDur 554.683497ms SnapOpenDur 45.002µs 2016-04-07T01:48:11.437-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:11.44-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:11.443-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:11.445-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:11.448-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:11.45-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:11.452-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:11.455-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:11.459-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:11.463-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:11.465-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16970000 2016-04-07T01:48:11.466-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:11.479-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16980000 2016-04-07T01:48:11.491-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 16990000 2016-04-07T01:48:11.503-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17000000 2016-04-07T01:48:11.515-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17010000 2016-04-07T01:48:11.527-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17020000 2016-04-07T01:48:11.542-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:11.808-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10230000 QueuedCount 48098 2016-04-07T01:48:12.209-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10240000 QueuedCount 38098 2016-04-07T01:48:12.598-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10250000 QueuedCount 28098 2016-04-07T01:48:12.998-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10260000 QueuedCount 18098 2016-04-07T01:48:13.436-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10270000 QueuedCount 8098 2016-04-07T01:48:14.172-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 34.974µs CommitTime 384.433958ms TotalFlushTime 7m2.014194833s TotalCommitTime 29.713294442s 2016-04-07T01:48:14.172-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 10278098, 10278098, 63 committed:true 2016-04-07T01:48:14.172-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 10278098, 10278098, 63 committed:true) SnapCreateDur 392.528165ms SnapOpenDur 47.542µs 2016-04-07T01:48:14.175-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:14.18-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:14.184-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:14.188-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:14.192-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:14.199-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17030000 2016-04-07T01:48:14.199-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:14.216-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10280000 QueuedCount 50000 2016-04-07T01:48:14.232-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17040000 2016-04-07T01:48:14.242-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17050000 2016-04-07T01:48:14.298-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:14.646-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:14.664-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17060000 2016-04-07T01:48:14.667-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10290000 QueuedCount 50000 2016-04-07T01:48:15.06-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17070000 2016-04-07T01:48:15.135-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10300000 QueuedCount 50000 2016-04-07T01:48:15.217-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:15.613-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10310000 QueuedCount 50000 2016-04-07T01:48:15.719-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:15.985-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:48:15.999-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:48:15.999-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:48:16.111-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10320000 QueuedCount 50000 2016-04-07T01:48:16.609-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10330000 QueuedCount 50000 2016-04-07T01:48:16.819-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:17.129-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10340000 QueuedCount 50000 2016-04-07T01:48:17.603-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10350000 QueuedCount 40133 2016-04-07T01:48:17.989-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10360000 QueuedCount 30133 2016-04-07T01:48:18.444-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10370000 QueuedCount 20133 2016-04-07T01:48:18.898-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10380000 QueuedCount 10133 2016-04-07T01:48:19.36-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10390000 QueuedCount 133 2016-04-07T01:48:20.001-07:00 [Info] Indexer::ReadMemstats Time Taken 17.794856ms 2016-04-07T01:48:20.004-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 26.404µs CommitTime 622.171287ms TotalFlushTime 7m7.106983085s TotalCommitTime 30.335465729s 2016-04-07T01:48:20.004-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 10390133, 10390133, 64 committed:true 2016-04-07T01:48:20.004-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 10390133, 10390133, 64 committed:true) SnapCreateDur 633.179337ms SnapOpenDur 47.309µs 2016-04-07T01:48:20.006-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:20.009-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:20.011-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:20.014-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:20.016-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:20.019-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:20.022-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:20.025-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:20.028-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:20.031-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:20.034-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:20.037-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:20.047-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17080000 2016-04-07T01:48:20.291-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:20.399-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10400000 QueuedCount 50000 2016-04-07T01:48:20.481-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:20.662-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:20.844-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10410000 QueuedCount 50000 2016-04-07T01:48:21.027-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:21.044-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17090000 2016-04-07T01:48:21.226-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:21.315-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10420000 QueuedCount 50000 2016-04-07T01:48:21.529-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17100000 2016-04-07T01:48:21.708-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:21.816-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10430000 QueuedCount 50000 2016-04-07T01:48:21.926-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17110000 2016-04-07T01:48:21.93-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:22.32-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10440000 QueuedCount 50000 2016-04-07T01:48:22.785-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10450000 QueuedCount 40638 2016-04-07T01:48:23.233-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10460000 QueuedCount 30638 2016-04-07T01:48:23.607-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10470000 QueuedCount 20638 2016-04-07T01:48:24.037-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10480000 QueuedCount 10638 2016-04-07T01:48:24.473-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10490000 QueuedCount 638 2016-04-07T01:48:25.056-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 22.566µs CommitTime 498.44919ms TotalFlushTime 7m11.563541226s TotalCommitTime 30.833914919s 2016-04-07T01:48:25.056-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 10490638, 10490638, 65 committed:true 2016-04-07T01:48:25.056-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 10490638, 10490638, 65 committed:true) SnapCreateDur 506.787618ms SnapOpenDur 60.784µs 2016-04-07T01:48:25.082-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:25.454-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10500000 QueuedCount 50000 2016-04-07T01:48:25.643-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:25.923-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10510000 QueuedCount 50000 2016-04-07T01:48:26.024-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:48:26.037-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:48:26.037-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:48:26.389-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10520000 QueuedCount 50000 2016-04-07T01:48:26.593-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:26.785-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:26.89-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10530000 QueuedCount 50000 2016-04-07T01:48:26.987-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:27.38-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10540000 QueuedCount 45752 2016-04-07T01:48:27.797-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10550000 QueuedCount 35752 2016-04-07T01:48:28.262-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10560000 QueuedCount 25752 2016-04-07T01:48:28.766-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10570000 QueuedCount 15752 2016-04-07T01:48:29.226-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10580000 QueuedCount 5752 2016-04-07T01:48:30.092-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5010 Milliseconds Vbucket 716 2016-04-07T01:48:30.122-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5040 Milliseconds Vbucket 716 2016-04-07T01:48:30.141-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 16.538µs CommitTime 613.140269ms TotalFlushTime 7m15.938335022s TotalCommitTime 31.447055188s 2016-04-07T01:48:30.141-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 10585752, 10585752, 66 committed:true 2016-04-07T01:48:30.142-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 10585752, 10585752, 66 committed:true) SnapCreateDur 623.707275ms SnapOpenDur 55.001µs 2016-04-07T01:48:30.144-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:30.146-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:30.149-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:30.152-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5070 Milliseconds Vbucket 716 2016-04-07T01:48:30.152-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:30.154-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:30.158-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:30.16-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:30.163-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:30.167-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:30.17-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:30.174-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:30.182-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5100 Milliseconds Vbucket 716 2016-04-07T01:48:30.212-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5130 Milliseconds Vbucket 716 2016-04-07T01:48:30.242-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5160 Milliseconds Vbucket 716 2016-04-07T01:48:30.272-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5190 Milliseconds Vbucket 716 2016-04-07T01:48:30.302-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5220 Milliseconds Vbucket 716 2016-04-07T01:48:30.328-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10590000 QueuedCount 50000 2016-04-07T01:48:30.332-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5250 Milliseconds Vbucket 716 2016-04-07T01:48:30.342-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:30.362-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5280 Milliseconds Vbucket 716 2016-04-07T01:48:30.392-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5310 Milliseconds Vbucket 716 2016-04-07T01:48:30.422-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5340 Milliseconds Vbucket 716 2016-04-07T01:48:30.452-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5370 Milliseconds Vbucket 716 2016-04-07T01:48:30.482-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5400 Milliseconds Vbucket 716 2016-04-07T01:48:30.512-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5430 Milliseconds Vbucket 716 2016-04-07T01:48:30.522-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:30.542-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5460 Milliseconds Vbucket 716 2016-04-07T01:48:30.572-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5490 Milliseconds Vbucket 716 2016-04-07T01:48:30.602-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5520 Milliseconds Vbucket 716 2016-04-07T01:48:30.632-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5550 Milliseconds Vbucket 716 2016-04-07T01:48:30.662-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5580 Milliseconds Vbucket 716 2016-04-07T01:48:30.692-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5610 Milliseconds Vbucket 716 2016-04-07T01:48:30.718-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10600000 QueuedCount 50000 2016-04-07T01:48:30.722-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5640 Milliseconds Vbucket 716 2016-04-07T01:48:30.752-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5670 Milliseconds Vbucket 716 2016-04-07T01:48:30.782-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5700 Milliseconds Vbucket 716 2016-04-07T01:48:30.809-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:30.812-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17120000 2016-04-07T01:48:31.186-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10610000 QueuedCount 50000 2016-04-07T01:48:31.667-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10620000 QueuedCount 50000 2016-04-07T01:48:31.87-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:32.108-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17130000 2016-04-07T01:48:32.171-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10630000 QueuedCount 50000 2016-04-07T01:48:32.657-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10640000 QueuedCount 50000 2016-04-07T01:48:32.686-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17140000 2016-04-07T01:48:33.152-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10650000 QueuedCount 50000 2016-04-07T01:48:33.347-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:33.383-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17150000 2016-04-07T01:48:33.679-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10660000 QueuedCount 50000 2016-04-07T01:48:34.191-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10670000 QueuedCount 41468 2016-04-07T01:48:34.66-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10680000 QueuedCount 31468 2016-04-07T01:48:35.066-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10690000 QueuedCount 21468 2016-04-07T01:48:35.543-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10700000 QueuedCount 11468 2016-04-07T01:48:36.025-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10710000 QueuedCount 1468 2016-04-07T01:48:36.063-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:48:36.077-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:48:36.077-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:48:36.835-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 14.635µs CommitTime 705.719329ms TotalFlushTime 7m21.806935577s TotalCommitTime 32.152774517s 2016-04-07T01:48:36.835-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 10711468, 10711468, 67 committed:true 2016-04-07T01:48:36.835-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 10711468, 10711468, 67 committed:true) SnapCreateDur 714.113176ms SnapOpenDur 72.363µs 2016-04-07T01:48:36.841-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:36.845-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:36.847-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17160000 2016-04-07T01:48:36.847-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:36.85-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:36.853-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:36.856-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:36.858-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17170000 2016-04-07T01:48:36.86-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:36.864-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:36.868-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:36.869-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17180000 2016-04-07T01:48:36.879-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17190000 2016-04-07T01:48:36.888-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17200000 2016-04-07T01:48:37.052-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:37.218-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10720000 QueuedCount 50000 2016-04-07T01:48:37.235-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:37.247-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17210000 2016-04-07T01:48:37.514-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:37.668-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10730000 QueuedCount 50000 2016-04-07T01:48:37.772-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:37.794-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17220000 2016-04-07T01:48:37.893-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17230000 2016-04-07T01:48:37.904-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17240000 2016-04-07T01:48:37.913-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17250000 2016-04-07T01:48:38.056-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:38.072-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10740000 QueuedCount 50000 2016-04-07T01:48:38.259-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:38.44-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:38.555-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10750000 QueuedCount 50000 2016-04-07T01:48:39.013-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10760000 QueuedCount 42391 2016-04-07T01:48:39.472-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10770000 QueuedCount 32391 2016-04-07T01:48:39.675-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:2844686882, Disk:7278686208, ExtraSnapshotData:0, Fragmentation:60.917577695911575%, mode : circular 2016-04-07T01:48:39.675-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:48:39.939-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10780000 QueuedCount 22391 2016-04-07T01:48:40.409-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10790000 QueuedCount 12391 2016-04-07T01:48:40.88-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10800000 QueuedCount 2391 2016-04-07T01:48:41.626-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.452µs CommitTime 629.844811ms TotalFlushTime 7m25.874239667s TotalCommitTime 32.782619328s 2016-04-07T01:48:41.626-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 10802391, 10802391, 68 committed:true 2016-04-07T01:48:41.626-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 10802391, 10802391, 68 committed:true) SnapCreateDur 638.933473ms SnapOpenDur 51.352µs 2016-04-07T01:48:41.633-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:41.637-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:41.64-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:41.664-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17260000 2016-04-07T01:48:41.675-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17270000 2016-04-07T01:48:41.686-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17280000 2016-04-07T01:48:41.697-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17290000 2016-04-07T01:48:41.706-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17300000 2016-04-07T01:48:41.93-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10810000 QueuedCount 50000 2016-04-07T01:48:42.373-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10820000 QueuedCount 50000 2016-04-07T01:48:42.836-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10830000 QueuedCount 50000 2016-04-07T01:48:42.932-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:43.321-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10840000 QueuedCount 50000 2016-04-07T01:48:43.432-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:43.801-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10850000 QueuedCount 50000 2016-04-07T01:48:43.804-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:44.322-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10860000 QueuedCount 50000 2016-04-07T01:48:44.779-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10870000 QueuedCount 40186 2016-04-07T01:48:45.196-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10880000 QueuedCount 30186 2016-04-07T01:48:45.675-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10890000 QueuedCount 20186 2016-04-07T01:48:46.098-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:48:46.111-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:48:46.111-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:48:46.159-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10900000 QueuedCount 10186 2016-04-07T01:48:46.629-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10910000 QueuedCount 186 2016-04-07T01:48:46.708-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5010 Milliseconds Vbucket 683 2016-04-07T01:48:46.738-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5040 Milliseconds Vbucket 683 2016-04-07T01:48:46.768-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5070 Milliseconds Vbucket 683 2016-04-07T01:48:46.798-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5100 Milliseconds Vbucket 683 2016-04-07T01:48:46.828-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5130 Milliseconds Vbucket 683 2016-04-07T01:48:46.858-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5160 Milliseconds Vbucket 683 2016-04-07T01:48:46.888-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5190 Milliseconds Vbucket 683 2016-04-07T01:48:46.918-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5220 Milliseconds Vbucket 683 2016-04-07T01:48:46.948-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5250 Milliseconds Vbucket 683 2016-04-07T01:48:46.978-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5280 Milliseconds Vbucket 683 2016-04-07T01:48:47.008-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5310 Milliseconds Vbucket 683 2016-04-07T01:48:47.038-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5340 Milliseconds Vbucket 683 2016-04-07T01:48:47.068-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5370 Milliseconds Vbucket 683 2016-04-07T01:48:47.098-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5400 Milliseconds Vbucket 683 2016-04-07T01:48:47.128-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5430 Milliseconds Vbucket 683 2016-04-07T01:48:47.158-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5460 Milliseconds Vbucket 683 2016-04-07T01:48:47.188-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5490 Milliseconds Vbucket 683 2016-04-07T01:48:47.218-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5520 Milliseconds Vbucket 683 2016-04-07T01:48:47.248-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5550 Milliseconds Vbucket 683 2016-04-07T01:48:47.278-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5580 Milliseconds Vbucket 683 2016-04-07T01:48:47.308-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5610 Milliseconds Vbucket 683 2016-04-07T01:48:47.338-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5640 Milliseconds Vbucket 683 2016-04-07T01:48:47.368-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5670 Milliseconds Vbucket 683 2016-04-07T01:48:47.375-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.006µs CommitTime 731.453487ms TotalFlushTime 7m30.794697421s TotalCommitTime 33.514072815s 2016-04-07T01:48:47.375-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 10910186, 10910186, 69 committed:true 2016-04-07T01:48:47.375-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 10910186, 10910186, 69 committed:true) SnapCreateDur 739.46567ms SnapOpenDur 66.153µs 2016-04-07T01:48:47.393-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:47.4-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:47.403-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:47.404-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:47.413-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17310000 2016-04-07T01:48:47.425-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17320000 2016-04-07T01:48:47.434-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17330000 2016-04-07T01:48:47.513-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:47.728-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:47.824-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10920000 QueuedCount 50000 2016-04-07T01:48:47.921-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:48.322-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:48.333-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10930000 QueuedCount 50000 2016-04-07T01:48:48.436-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:48.831-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:48.842-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10940000 QueuedCount 50000 2016-04-07T01:48:49.338-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10950000 QueuedCount 45296 2016-04-07T01:48:49.792-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10960000 QueuedCount 35296 2016-04-07T01:48:50.252-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10970000 QueuedCount 25296 2016-04-07T01:48:50.711-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10980000 QueuedCount 15296 2016-04-07T01:48:51.117-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 10990000 QueuedCount 5296 2016-04-07T01:48:52.008-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 14.409µs CommitTime 607.358952ms TotalFlushTime 7m34.739255257s TotalCommitTime 34.121431767s 2016-04-07T01:48:52.008-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 10995296, 10995296, 70 committed:true 2016-04-07T01:48:52.008-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 10995296, 10995296, 70 committed:true) SnapCreateDur 615.704421ms SnapOpenDur 88.17µs 2016-04-07T01:48:52.012-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:52.017-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:52.021-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:52.024-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:52.027-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:52.029-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:52.031-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:52.038-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:52.057-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17340000 2016-04-07T01:48:52.067-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17350000 2016-04-07T01:48:52.241-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11000000 QueuedCount 50000 2016-04-07T01:48:52.456-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17360000 2016-04-07T01:48:52.723-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11010000 QueuedCount 50000 2016-04-07T01:48:53.226-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11020000 QueuedCount 50000 2016-04-07T01:48:53.66-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11030000 QueuedCount 50000 2016-04-07T01:48:54.177-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11040000 QueuedCount 50000 2016-04-07T01:48:54.268-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:54.474-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17370000 2016-04-07T01:48:54.698-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11050000 QueuedCount 50000 2016-04-07T01:48:54.931-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17380000 2016-04-07T01:48:55.233-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11060000 QueuedCount 50000 2016-04-07T01:48:55.738-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11070000 QueuedCount 46234 2016-04-07T01:48:56.134-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:48:56.147-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:48:56.148-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:48:56.217-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11080000 QueuedCount 36234 2016-04-07T01:48:56.673-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11090000 QueuedCount 26234 2016-04-07T01:48:57.14-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11100000 QueuedCount 16234 2016-04-07T01:48:57.547-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11110000 QueuedCount 6234 2016-04-07T01:48:58.523-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.13µs CommitTime 610.216716ms TotalFlushTime 7m40.529067195s TotalCommitTime 34.731648483s 2016-04-07T01:48:58.523-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 11116234, 11116234, 71 committed:true 2016-04-07T01:48:58.523-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 11116234, 11116234, 71 committed:true) SnapCreateDur 618.762984ms SnapOpenDur 64.004µs 2016-04-07T01:48:58.526-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:58.529-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:58.532-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:58.534-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:58.536-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:58.539-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:58.541-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:58.544-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17390000 2016-04-07T01:48:58.546-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:58.549-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:58.554-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:58.556-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17400000 2016-04-07T01:48:58.558-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:58.568-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17410000 2016-04-07T01:48:58.579-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17420000 2016-04-07T01:48:58.59-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17430000 2016-04-07T01:48:58.601-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17440000 2016-04-07T01:48:58.671-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11120000 QueuedCount 50000 2016-04-07T01:48:58.883-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:59.185-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11130000 QueuedCount 50000 2016-04-07T01:48:59.269-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:48:59.654-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11140000 QueuedCount 43615 2016-04-07T01:49:00.092-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11150000 QueuedCount 33615 2016-04-07T01:49:00.545-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11160000 QueuedCount 23615 2016-04-07T01:49:01.018-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11170000 QueuedCount 13615 2016-04-07T01:49:01.511-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11180000 QueuedCount 3615 2016-04-07T01:49:02.186-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.46µs CommitTime 549.303451ms TotalFlushTime 7m43.574519966s TotalCommitTime 35.280951934s 2016-04-07T01:49:02.186-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 11183615, 11183615, 72 committed:true 2016-04-07T01:49:02.186-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 11183615, 11183615, 72 committed:true) SnapCreateDur 557.732887ms SnapOpenDur 61.689µs 2016-04-07T01:49:02.19-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:02.194-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:02.199-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:02.205-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:02.214-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:02.234-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17450000 2016-04-07T01:49:02.247-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17460000 2016-04-07T01:49:02.53-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11190000 QueuedCount 50000 2016-04-07T01:49:02.533-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:02.984-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11200000 QueuedCount 50000 2016-04-07T01:49:03.488-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11210000 QueuedCount 50000 2016-04-07T01:49:04.003-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11220000 QueuedCount 50000 2016-04-07T01:49:04.54-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:04.55-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11230000 QueuedCount 50000 2016-04-07T01:49:04.558-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17470000 2016-04-07T01:49:05.043-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11240000 QueuedCount 42712 2016-04-07T01:49:05.517-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11250000 QueuedCount 32712 2016-04-07T01:49:05.978-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11260000 QueuedCount 22712 2016-04-07T01:49:06.177-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:49:06.193-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:49:06.193-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:49:06.479-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11270000 QueuedCount 12712 2016-04-07T01:49:06.967-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11280000 QueuedCount 2712 2016-04-07T01:49:07.875-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.717µs CommitTime 785.7996ms TotalFlushTime 7m48.380050754s TotalCommitTime 36.066751534s 2016-04-07T01:49:07.875-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 11282712, 11282712, 73 committed:true 2016-04-07T01:49:07.875-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 11282712, 11282712, 73 committed:true) SnapCreateDur 794.519114ms SnapOpenDur 70.19µs 2016-04-07T01:49:07.877-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:07.879-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:07.881-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:07.883-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:07.885-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:07.888-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:07.89-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:07.892-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:07.898-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:07.9-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17480000 2016-04-07T01:49:07.901-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:07.904-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:07.907-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:07.91-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:07.915-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17490000 2016-04-07T01:49:07.932-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17500000 2016-04-07T01:49:07.948-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17510000 2016-04-07T01:49:07.958-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17520000 2016-04-07T01:49:08.033-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:08.24-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11290000 QueuedCount 50000 2016-04-07T01:49:08.638-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:08.64-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:08.748-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:08.857-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11300000 QueuedCount 50000 2016-04-07T01:49:08.967-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:49:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T01:49:09.374-07:00 [Info] client load stats {} 2016-04-07T01:49:09.383-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11310000 QueuedCount 50000 2016-04-07T01:49:09.596-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:09.638-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":6153643,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":5210618962,"bucket-1:myindex:avg_ts_items_count":93362,"bucket-1:myindex:build_progress":56,"bucket-1:myindex:data_size":2973918380,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":8158302208,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":50076,"bucket-1:myindex:frag_percent":63,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":1180732307,"bucket-1:myindex:items_count":11183615,"bucket-1:myindex:num_commits":73,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":11315487,"bucket-1:myindex:num_docs_pending":2528178,"bucket-1:myindex:num_docs_processed":11279203,"bucket-1:myindex:num_docs_queued":6192619,"bucket-1:myindex:num_flush_queued":11365563,"bucket-1:myindex:num_items_flushed":11315487,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":73,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":5689187284,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"73 36066751534 2556931853442547648","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"11315488 66665822357 213479110743191861","bucket-1:myindex:timings/storage_info":"293 2391210 34732581438","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"72 25121876 9729836023142","bucket-1:myindex:timings/storage_meta_set":"73 25690312 9777361649804","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"73 3245043 156150818623","bucket-1:myindex:timings/storage_set":"22630975 397251658382 2913845566394641832","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":17519133,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"106 147688832 2083552510466770","bucket-1:ts_queue_size":926,"indexer_state":"Active","memory_quota":31457280000,"memory_used":14464550912,"memory_used_storage":8407023616,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"2050 301867389 64981906862135","uptime":"20m1.331003411s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T01:49:09.639-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17530000 2016-04-07T01:49:09.675-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:2992856389, Disk:8456011776, ExtraSnapshotData:0, Fragmentation:64.6067618130053%, mode : circular 2016-04-07T01:49:09.675-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:49:09.676-07:00 [Info] memstats {"Alloc":3536314920, "TotalAlloc":15660416624, "Sys":9854296392, "Lookups":4009, "Mallocs":419651768,"Frees":340527741, "HeapAlloc":3536314920, "HeapSys":9384919040, "HeapIdle":3624837120, "HeapInuse":5760081920,"HeapReleased":0, "HeapObjects":79124027,"GCSys":297445376, "LastGC":1460018884062254022,"PauseTotalNs":25888237, "PauseNs":[1224786], "NumGC":20} 2016-04-07T01:49:09.917-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11320000 QueuedCount 50000 2016-04-07T01:49:10.402-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11330000 QueuedCount 41008 2016-04-07T01:49:10.811-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11340000 QueuedCount 31008 2016-04-07T01:49:11.297-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11350000 QueuedCount 21008 2016-04-07T01:49:11.796-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11360000 QueuedCount 11008 2016-04-07T01:49:12.278-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11370000 QueuedCount 1008 2016-04-07T01:49:13.136-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 38.776µs CommitTime 757.370807ms TotalFlushTime 7m52.79712697s TotalCommitTime 36.824122341s 2016-04-07T01:49:13.136-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 11371008, 11371008, 74 committed:true 2016-04-07T01:49:13.136-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 11371008, 11371008, 74 committed:true) SnapCreateDur 765.508631ms SnapOpenDur 78.254µs 2016-04-07T01:49:13.141-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:13.147-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:13.164-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17540000 2016-04-07T01:49:13.18-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17550000 2016-04-07T01:49:13.191-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17560000 2016-04-07T01:49:13.201-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17570000 2016-04-07T01:49:13.599-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11380000 QueuedCount 50000 2016-04-07T01:49:13.614-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:13.904-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:14.022-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:14.129-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11390000 QueuedCount 50000 2016-04-07T01:49:14.145-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:14.439-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:14.651-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11400000 QueuedCount 50000 2016-04-07T01:49:14.657-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:15.135-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11410000 QueuedCount 43430 2016-04-07T01:49:15.54-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11420000 QueuedCount 33430 2016-04-07T01:49:16.025-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11430000 QueuedCount 23430 2016-04-07T01:49:16.217-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:49:16.231-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:49:16.231-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:49:16.517-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11440000 QueuedCount 13430 2016-04-07T01:49:17.013-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11450000 QueuedCount 3430 2016-04-07T01:49:18.926-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.159µs CommitTime 1.713631789s TotalFlushTime 7m56.790521385s TotalCommitTime 38.53775413s 2016-04-07T01:49:18.926-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 11453430, 11453430, 75 committed:true 2016-04-07T01:49:18.926-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 11453430, 11453430, 75 committed:true) SnapCreateDur 1.721722703s SnapOpenDur 52.87µs 2016-04-07T01:49:18.928-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:18.934-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:18.936-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:18.939-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:18.942-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:18.944-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:18.946-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:18.949-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:18.952-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:18.961-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17580000 2016-04-07T01:49:19.041-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:19.196-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11460000 QueuedCount 50000 2016-04-07T01:49:19.455-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:19.576-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:19.637-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11470000 QueuedCount 50000 2016-04-07T01:49:19.841-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:20.039-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11480000 QueuedCount 48635 2016-04-07T01:49:20.392-07:00 [Info] Indexer::ReadMemstats Time Taken 14.720133ms 2016-04-07T01:49:20.505-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11490000 QueuedCount 38635 2016-04-07T01:49:20.99-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11500000 QueuedCount 28635 2016-04-07T01:49:21.49-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11510000 QueuedCount 18635 2016-04-07T01:49:21.979-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11520000 QueuedCount 8635 2016-04-07T01:49:23.137-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 32.978µs CommitTime 748.198737ms TotalFlushTime 8m0.184748355s TotalCommitTime 39.285952867s 2016-04-07T01:49:23.137-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 11528635, 11528635, 76 committed:true 2016-04-07T01:49:23.137-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 11528635, 11528635, 76 committed:true) SnapCreateDur 756.193867ms SnapOpenDur 48.58µs 2016-04-07T01:49:23.164-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:23.167-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17590000 2016-04-07T01:49:23.181-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17600000 2016-04-07T01:49:23.182-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11530000 QueuedCount 50000 2016-04-07T01:49:23.194-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17610000 2016-04-07T01:49:23.314-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17620000 2016-04-07T01:49:23.659-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11540000 QueuedCount 50000 2016-04-07T01:49:23.772-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17630000 2016-04-07T01:49:24.154-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11550000 QueuedCount 50000 2016-04-07T01:49:24.678-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11560000 QueuedCount 50000 2016-04-07T01:49:25.214-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11570000 QueuedCount 50000 2016-04-07T01:49:25.747-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11580000 QueuedCount 50000 2016-04-07T01:49:26.204-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11590000 QueuedCount 50000 2016-04-07T01:49:26.752-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11600000 QueuedCount 50000 2016-04-07T01:49:27.317-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11610000 QueuedCount 50000 2016-04-07T01:49:27.323-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:27.86-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11620000 QueuedCount 44791 2016-04-07T01:49:28.363-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11630000 QueuedCount 34791 2016-04-07T01:49:28.86-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11640000 QueuedCount 24791 2016-04-07T01:49:28.873-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5010 Milliseconds Vbucket 755 2016-04-07T01:49:28.903-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5040 Milliseconds Vbucket 755 2016-04-07T01:49:28.933-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5070 Milliseconds Vbucket 755 2016-04-07T01:49:28.963-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5100 Milliseconds Vbucket 755 2016-04-07T01:49:28.993-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5130 Milliseconds Vbucket 755 2016-04-07T01:49:29.023-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5160 Milliseconds Vbucket 755 2016-04-07T01:49:29.053-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5190 Milliseconds Vbucket 755 2016-04-07T01:49:29.082-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5220 Milliseconds Vbucket 755 2016-04-07T01:49:29.112-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5250 Milliseconds Vbucket 755 2016-04-07T01:49:29.142-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5280 Milliseconds Vbucket 755 2016-04-07T01:49:29.172-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5310 Milliseconds Vbucket 755 2016-04-07T01:49:29.203-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5340 Milliseconds Vbucket 755 2016-04-07T01:49:29.232-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5370 Milliseconds Vbucket 755 2016-04-07T01:49:29.262-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5400 Milliseconds Vbucket 755 2016-04-07T01:49:29.292-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5430 Milliseconds Vbucket 755 2016-04-07T01:49:29.322-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5460 Milliseconds Vbucket 755 2016-04-07T01:49:29.353-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5490 Milliseconds Vbucket 755 2016-04-07T01:49:29.362-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11650000 QueuedCount 14791 2016-04-07T01:49:29.383-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5520 Milliseconds Vbucket 755 2016-04-07T01:49:29.413-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5550 Milliseconds Vbucket 755 2016-04-07T01:49:29.443-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5580 Milliseconds Vbucket 755 2016-04-07T01:49:29.472-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5610 Milliseconds Vbucket 755 2016-04-07T01:49:29.502-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5640 Milliseconds Vbucket 755 2016-04-07T01:49:29.532-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5670 Milliseconds Vbucket 755 2016-04-07T01:49:29.562-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5700 Milliseconds Vbucket 755 2016-04-07T01:49:29.592-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5730 Milliseconds Vbucket 755 2016-04-07T01:49:29.622-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5760 Milliseconds Vbucket 755 2016-04-07T01:49:29.653-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5790 Milliseconds Vbucket 755 2016-04-07T01:49:29.682-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5820 Milliseconds Vbucket 755 2016-04-07T01:49:29.713-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5850 Milliseconds Vbucket 755 2016-04-07T01:49:29.743-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5880 Milliseconds Vbucket 755 2016-04-07T01:49:29.772-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5910 Milliseconds Vbucket 755 2016-04-07T01:49:29.803-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5940 Milliseconds Vbucket 755 2016-04-07T01:49:29.833-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5970 Milliseconds Vbucket 755 2016-04-07T01:49:29.862-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6000 Milliseconds Vbucket 755 2016-04-07T01:49:29.87-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11660000 QueuedCount 4791 2016-04-07T01:49:29.893-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6030 Milliseconds Vbucket 755 2016-04-07T01:49:29.922-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6060 Milliseconds Vbucket 755 2016-04-07T01:49:29.952-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6090 Milliseconds Vbucket 755 2016-04-07T01:49:29.982-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6120 Milliseconds Vbucket 755 2016-04-07T01:49:30.013-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6150 Milliseconds Vbucket 755 2016-04-07T01:49:30.043-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6180 Milliseconds Vbucket 755 2016-04-07T01:49:30.073-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6210 Milliseconds Vbucket 755 2016-04-07T01:49:30.102-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6240 Milliseconds Vbucket 755 2016-04-07T01:49:30.133-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6270 Milliseconds Vbucket 755 2016-04-07T01:49:30.163-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6300 Milliseconds Vbucket 755 2016-04-07T01:49:30.193-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6330 Milliseconds Vbucket 755 2016-04-07T01:49:30.223-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6360 Milliseconds Vbucket 755 2016-04-07T01:49:30.253-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6390 Milliseconds Vbucket 755 2016-04-07T01:49:30.283-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6420 Milliseconds Vbucket 755 2016-04-07T01:49:30.313-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6450 Milliseconds Vbucket 755 2016-04-07T01:49:30.343-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6480 Milliseconds Vbucket 755 2016-04-07T01:49:30.373-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6510 Milliseconds Vbucket 755 2016-04-07T01:49:30.403-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6540 Milliseconds Vbucket 755 2016-04-07T01:49:30.433-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6570 Milliseconds Vbucket 755 2016-04-07T01:49:30.463-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6600 Milliseconds Vbucket 755 2016-04-07T01:49:30.493-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6630 Milliseconds Vbucket 755 2016-04-07T01:49:30.523-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6660 Milliseconds Vbucket 755 2016-04-07T01:49:30.553-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6690 Milliseconds Vbucket 755 2016-04-07T01:49:30.583-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6720 Milliseconds Vbucket 755 2016-04-07T01:49:30.613-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6750 Milliseconds Vbucket 755 2016-04-07T01:49:30.643-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6780 Milliseconds Vbucket 755 2016-04-07T01:49:30.673-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6810 Milliseconds Vbucket 755 2016-04-07T01:49:30.703-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6840 Milliseconds Vbucket 755 2016-04-07T01:49:30.733-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6870 Milliseconds Vbucket 755 2016-04-07T01:49:30.763-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6900 Milliseconds Vbucket 755 2016-04-07T01:49:30.793-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6930 Milliseconds Vbucket 755 2016-04-07T01:49:30.823-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6960 Milliseconds Vbucket 755 2016-04-07T01:49:30.853-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6990 Milliseconds Vbucket 755 2016-04-07T01:49:30.877-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.383µs CommitTime 769.246572ms TotalFlushTime 8m7.003111754s TotalCommitTime 40.055199439s 2016-04-07T01:49:30.877-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 11664791, 11664791, 77 committed:true 2016-04-07T01:49:30.877-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 11664791, 11664791, 77 committed:true) SnapCreateDur 780.250777ms SnapOpenDur 46.921µs 2016-04-07T01:49:30.88-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:30.882-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:30.882-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7020 Milliseconds Vbucket 755 2016-04-07T01:49:30.885-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:30.888-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:30.891-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:30.894-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:30.896-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:30.899-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:30.904-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:30.907-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:30.91-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:30.916-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:30.922-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17640000 2016-04-07T01:49:30.933-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17650000 2016-04-07T01:49:30.944-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17660000 2016-04-07T01:49:30.954-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17670000 2016-04-07T01:49:30.963-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17680000 2016-04-07T01:49:31.113-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11670000 QueuedCount 50000 2016-04-07T01:49:31.275-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:49:31.287-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:49:31.287-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:49:31.29-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:31.387-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:31.475-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:31.569-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11680000 QueuedCount 50000 2016-04-07T01:49:31.755-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:31.953-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:32.054-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11690000 QueuedCount 50000 2016-04-07T01:49:32.145-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:32.253-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:32.466-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:32.551-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11700000 QueuedCount 50000 2016-04-07T01:49:33.032-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11710000 QueuedCount 44293 2016-04-07T01:49:33.439-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11720000 QueuedCount 34293 2016-04-07T01:49:33.923-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11730000 QueuedCount 24293 2016-04-07T01:49:34.408-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11740000 QueuedCount 14293 2016-04-07T01:49:34.898-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11750000 QueuedCount 4293 2016-04-07T01:49:35.81-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 34.807µs CommitTime 700.387017ms TotalFlushTime 8m11.14665225s TotalCommitTime 40.755586456s 2016-04-07T01:49:35.81-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 11754293, 11754293, 78 committed:true 2016-04-07T01:49:35.81-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 11754293, 11754293, 78 committed:true) SnapCreateDur 708.783233ms SnapOpenDur 49.179µs 2016-04-07T01:49:35.813-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:35.817-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:35.82-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:35.824-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:35.826-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17690000 2016-04-07T01:49:35.828-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:35.831-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:35.836-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:35.837-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17700000 2016-04-07T01:49:35.84-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:35.848-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:35.85-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17710000 2016-04-07T01:49:35.86-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17720000 2016-04-07T01:49:35.872-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17730000 2016-04-07T01:49:35.884-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17740000 2016-04-07T01:49:35.894-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17750000 2016-04-07T01:49:35.955-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17760000 2016-04-07T01:49:36.018-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11760000 QueuedCount 50000 2016-04-07T01:49:36.278-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:36.381-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17770000 2016-04-07T01:49:36.459-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11770000 QueuedCount 50000 2016-04-07T01:49:36.778-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17780000 2016-04-07T01:49:36.947-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11780000 QueuedCount 50000 2016-04-07T01:49:36.957-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:36.966-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17790000 2016-04-07T01:49:37.255-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:37.27-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17800000 2016-04-07T01:49:37.357-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:37.445-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11790000 QueuedCount 50000 2016-04-07T01:49:37.578-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17810000 2016-04-07T01:49:37.647-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:37.769-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17820000 2016-04-07T01:49:37.78-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17830000 2016-04-07T01:49:37.791-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17840000 2016-04-07T01:49:37.802-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17850000 2016-04-07T01:49:37.813-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17860000 2016-04-07T01:49:37.823-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17870000 2016-04-07T01:49:37.833-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17880000 2016-04-07T01:49:37.845-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17890000 2016-04-07T01:49:37.857-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17900000 2016-04-07T01:49:37.869-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17910000 2016-04-07T01:49:37.88-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17920000 2016-04-07T01:49:37.891-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17930000 2016-04-07T01:49:37.902-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17940000 2016-04-07T01:49:37.912-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17950000 2016-04-07T01:49:37.924-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17960000 2016-04-07T01:49:37.934-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17970000 2016-04-07T01:49:37.945-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17980000 2016-04-07T01:49:37.948-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11800000 QueuedCount 48717 2016-04-07T01:49:37.956-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 17990000 2016-04-07T01:49:37.968-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18000000 2016-04-07T01:49:37.979-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18010000 2016-04-07T01:49:37.99-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18020000 2016-04-07T01:49:38.001-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18030000 2016-04-07T01:49:38.011-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18040000 2016-04-07T01:49:38.022-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18050000 2016-04-07T01:49:38.032-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18060000 2016-04-07T01:49:38.042-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18070000 2016-04-07T01:49:38.052-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18080000 2016-04-07T01:49:38.061-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18090000 2016-04-07T01:49:38.071-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18100000 2016-04-07T01:49:38.082-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18110000 2016-04-07T01:49:38.092-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18120000 2016-04-07T01:49:38.103-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18130000 2016-04-07T01:49:38.114-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18140000 2016-04-07T01:49:38.123-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18150000 2016-04-07T01:49:38.133-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18160000 2016-04-07T01:49:38.144-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18170000 2016-04-07T01:49:38.156-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18180000 2016-04-07T01:49:38.166-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18190000 2016-04-07T01:49:38.176-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18200000 2016-04-07T01:49:38.186-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18210000 2016-04-07T01:49:38.197-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18220000 2016-04-07T01:49:38.206-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18230000 2016-04-07T01:49:38.215-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18240000 2016-04-07T01:49:38.225-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18250000 2016-04-07T01:49:38.234-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18260000 2016-04-07T01:49:38.243-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18270000 2016-04-07T01:49:38.254-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18280000 2016-04-07T01:49:38.263-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18290000 2016-04-07T01:49:38.274-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18300000 2016-04-07T01:49:38.284-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18310000 2016-04-07T01:49:38.298-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18320000 2016-04-07T01:49:38.31-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18330000 2016-04-07T01:49:38.319-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18340000 2016-04-07T01:49:38.462-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11810000 QueuedCount 38717 2016-04-07T01:49:38.946-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11820000 QueuedCount 28717 2016-04-07T01:49:39.362-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11830000 QueuedCount 18717 2016-04-07T01:49:39.676-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:3132247998, Disk:9040855040, ExtraSnapshotData:0, Fragmentation:65.35451587110062%, mode : circular 2016-04-07T01:49:39.676-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:49:39.861-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11840000 QueuedCount 8717 2016-04-07T01:49:41.058-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 15.923µs CommitTime 697.701993ms TotalFlushTime 8m15.602421571s TotalCommitTime 41.453288449s 2016-04-07T01:49:41.058-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 11848717, 11848717, 79 committed:true 2016-04-07T01:49:41.058-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 11848717, 11848717, 79 committed:true) SnapCreateDur 706.485077ms SnapOpenDur 48.72µs 2016-04-07T01:49:41.062-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:41.064-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:41.068-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:41.071-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:41.072-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:41.076-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:41.079-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:41.082-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:41.085-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18350000 2016-04-07T01:49:41.086-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:41.092-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11850000 QueuedCount 50000 2016-04-07T01:49:41.098-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18360000 2016-04-07T01:49:41.111-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18370000 2016-04-07T01:49:41.123-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18380000 2016-04-07T01:49:41.135-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18390000 2016-04-07T01:49:41.147-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18400000 2016-04-07T01:49:41.159-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18410000 2016-04-07T01:49:41.171-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18420000 2016-04-07T01:49:41.185-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18430000 2016-04-07T01:49:41.196-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18440000 2016-04-07T01:49:41.206-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18450000 2016-04-07T01:49:41.215-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18460000 2016-04-07T01:49:41.225-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18470000 2016-04-07T01:49:41.234-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18480000 2016-04-07T01:49:41.244-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18490000 2016-04-07T01:49:41.254-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18500000 2016-04-07T01:49:41.267-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:41.268-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18510000 2016-04-07T01:49:41.278-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18520000 2016-04-07T01:49:41.289-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18530000 2016-04-07T01:49:41.301-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18540000 2016-04-07T01:49:41.312-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:49:41.313-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18550000 2016-04-07T01:49:41.325-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18560000 2016-04-07T01:49:41.326-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:49:41.326-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:49:41.337-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18570000 2016-04-07T01:49:41.349-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18580000 2016-04-07T01:49:41.362-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18590000 2016-04-07T01:49:41.373-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18600000 2016-04-07T01:49:41.386-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18610000 2016-04-07T01:49:41.397-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18620000 2016-04-07T01:49:41.407-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18630000 2016-04-07T01:49:41.419-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18640000 2016-04-07T01:49:41.43-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18650000 2016-04-07T01:49:41.442-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18660000 2016-04-07T01:49:41.454-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18670000 2016-04-07T01:49:41.466-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18680000 2016-04-07T01:49:41.477-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18690000 2016-04-07T01:49:41.489-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18700000 2016-04-07T01:49:41.5-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18710000 2016-04-07T01:49:41.51-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18720000 2016-04-07T01:49:41.519-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18730000 2016-04-07T01:49:41.529-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18740000 2016-04-07T01:49:41.541-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18750000 2016-04-07T01:49:41.553-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18760000 2016-04-07T01:49:41.564-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18770000 2016-04-07T01:49:41.567-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:41.573-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11860000 QueuedCount 50000 2016-04-07T01:49:41.574-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18780000 2016-04-07T01:49:41.587-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18790000 2016-04-07T01:49:41.598-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18800000 2016-04-07T01:49:41.61-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18810000 2016-04-07T01:49:41.621-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18820000 2016-04-07T01:49:41.633-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18830000 2016-04-07T01:49:41.644-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18840000 2016-04-07T01:49:41.654-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18850000 2016-04-07T01:49:41.666-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18860000 2016-04-07T01:49:41.679-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18870000 2016-04-07T01:49:41.689-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18880000 2016-04-07T01:49:41.699-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18890000 2016-04-07T01:49:41.708-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18900000 2016-04-07T01:49:41.717-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18910000 2016-04-07T01:49:41.727-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18920000 2016-04-07T01:49:41.736-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18930000 2016-04-07T01:49:41.746-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18940000 2016-04-07T01:49:41.756-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18950000 2016-04-07T01:49:41.765-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18960000 2016-04-07T01:49:41.775-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18970000 2016-04-07T01:49:41.778-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:41.785-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18980000 2016-04-07T01:49:41.795-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 18990000 2016-04-07T01:49:41.804-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19000000 2016-04-07T01:49:41.814-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19010000 2016-04-07T01:49:41.823-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19020000 2016-04-07T01:49:41.832-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19030000 2016-04-07T01:49:41.842-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19040000 2016-04-07T01:49:41.852-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19050000 2016-04-07T01:49:41.862-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19060000 2016-04-07T01:49:41.875-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19070000 2016-04-07T01:49:41.887-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19080000 2016-04-07T01:49:41.899-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19090000 2016-04-07T01:49:41.908-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19100000 2016-04-07T01:49:42.063-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11870000 QueuedCount 50000 2016-04-07T01:49:42.178-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:42.55-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11880000 QueuedCount 50000 2016-04-07T01:49:42.562-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:42.965-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:43.047-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11890000 QueuedCount 50000 2016-04-07T01:49:43.571-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11900000 QueuedCount 50000 2016-04-07T01:49:43.957-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11910000 QueuedCount 40260 2016-04-07T01:49:44.416-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11920000 QueuedCount 30260 2016-04-07T01:49:44.89-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11930000 QueuedCount 20260 2016-04-07T01:49:45.361-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11940000 QueuedCount 10260 2016-04-07T01:49:45.823-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11950000 QueuedCount 260 2016-04-07T01:49:46.481-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 14.753µs CommitTime 637.747011ms TotalFlushTime 8m20.289832874s TotalCommitTime 42.09103546s 2016-04-07T01:49:46.481-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 11950260, 11950260, 80 committed:true 2016-04-07T01:49:46.481-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 11950260, 11950260, 80 committed:true) SnapCreateDur 646.416346ms SnapOpenDur 52.039µs 2016-04-07T01:49:46.486-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:46.506-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:46.881-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11960000 QueuedCount 50000 2016-04-07T01:49:46.911-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5010 Milliseconds Vbucket 840 2016-04-07T01:49:46.941-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5040 Milliseconds Vbucket 840 2016-04-07T01:49:46.971-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5070 Milliseconds Vbucket 840 2016-04-07T01:49:47.001-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5100 Milliseconds Vbucket 840 2016-04-07T01:49:47.031-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5130 Milliseconds Vbucket 840 2016-04-07T01:49:47.061-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5160 Milliseconds Vbucket 840 2016-04-07T01:49:47.091-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5190 Milliseconds Vbucket 840 2016-04-07T01:49:47.121-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5220 Milliseconds Vbucket 840 2016-04-07T01:49:47.151-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5250 Milliseconds Vbucket 840 2016-04-07T01:49:47.181-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5280 Milliseconds Vbucket 840 2016-04-07T01:49:47.211-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5310 Milliseconds Vbucket 840 2016-04-07T01:49:47.241-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5340 Milliseconds Vbucket 840 2016-04-07T01:49:47.271-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5370 Milliseconds Vbucket 840 2016-04-07T01:49:47.301-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5400 Milliseconds Vbucket 840 2016-04-07T01:49:47.331-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5430 Milliseconds Vbucket 840 2016-04-07T01:49:47.361-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5460 Milliseconds Vbucket 840 2016-04-07T01:49:47.367-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:47.377-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11970000 QueuedCount 50000 2016-04-07T01:49:47.391-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5490 Milliseconds Vbucket 840 2016-04-07T01:49:47.421-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5520 Milliseconds Vbucket 840 2016-04-07T01:49:47.451-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5550 Milliseconds Vbucket 840 2016-04-07T01:49:47.481-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5580 Milliseconds Vbucket 840 2016-04-07T01:49:47.511-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5610 Milliseconds Vbucket 840 2016-04-07T01:49:47.541-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5640 Milliseconds Vbucket 840 2016-04-07T01:49:47.571-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5670 Milliseconds Vbucket 840 2016-04-07T01:49:47.601-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5700 Milliseconds Vbucket 840 2016-04-07T01:49:47.631-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5730 Milliseconds Vbucket 840 2016-04-07T01:49:47.661-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5760 Milliseconds Vbucket 840 2016-04-07T01:49:47.691-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5790 Milliseconds Vbucket 840 2016-04-07T01:49:47.721-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5820 Milliseconds Vbucket 840 2016-04-07T01:49:47.751-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5850 Milliseconds Vbucket 840 2016-04-07T01:49:47.781-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5880 Milliseconds Vbucket 840 2016-04-07T01:49:47.811-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5910 Milliseconds Vbucket 840 2016-04-07T01:49:47.841-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5940 Milliseconds Vbucket 840 2016-04-07T01:49:47.871-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5970 Milliseconds Vbucket 840 2016-04-07T01:49:47.883-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11980000 QueuedCount 50000 2016-04-07T01:49:47.901-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6000 Milliseconds Vbucket 840 2016-04-07T01:49:47.931-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6030 Milliseconds Vbucket 840 2016-04-07T01:49:47.961-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6060 Milliseconds Vbucket 840 2016-04-07T01:49:47.991-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6090 Milliseconds Vbucket 840 2016-04-07T01:49:48.021-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6120 Milliseconds Vbucket 840 2016-04-07T01:49:48.051-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6150 Milliseconds Vbucket 840 2016-04-07T01:49:48.081-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6180 Milliseconds Vbucket 840 2016-04-07T01:49:48.111-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6210 Milliseconds Vbucket 840 2016-04-07T01:49:48.141-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6240 Milliseconds Vbucket 840 2016-04-07T01:49:48.171-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6270 Milliseconds Vbucket 840 2016-04-07T01:49:48.201-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6300 Milliseconds Vbucket 840 2016-04-07T01:49:48.231-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6330 Milliseconds Vbucket 840 2016-04-07T01:49:48.261-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6360 Milliseconds Vbucket 840 2016-04-07T01:49:48.291-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6390 Milliseconds Vbucket 840 2016-04-07T01:49:48.321-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6420 Milliseconds Vbucket 840 2016-04-07T01:49:48.351-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6450 Milliseconds Vbucket 840 2016-04-07T01:49:48.381-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6480 Milliseconds Vbucket 840 2016-04-07T01:49:48.397-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 11990000 QueuedCount 50000 2016-04-07T01:49:48.411-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6510 Milliseconds Vbucket 840 2016-04-07T01:49:48.441-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6540 Milliseconds Vbucket 840 2016-04-07T01:49:48.471-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6570 Milliseconds Vbucket 840 2016-04-07T01:49:48.501-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6600 Milliseconds Vbucket 840 2016-04-07T01:49:48.531-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6630 Milliseconds Vbucket 840 2016-04-07T01:49:48.561-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6660 Milliseconds Vbucket 840 2016-04-07T01:49:48.591-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6690 Milliseconds Vbucket 840 2016-04-07T01:49:48.621-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6720 Milliseconds Vbucket 840 2016-04-07T01:49:48.651-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6750 Milliseconds Vbucket 840 2016-04-07T01:49:48.681-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6780 Milliseconds Vbucket 840 2016-04-07T01:49:48.711-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6810 Milliseconds Vbucket 840 2016-04-07T01:49:48.719-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:48.741-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6840 Milliseconds Vbucket 840 2016-04-07T01:49:48.771-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6870 Milliseconds Vbucket 840 2016-04-07T01:49:48.801-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6900 Milliseconds Vbucket 840 2016-04-07T01:49:48.831-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6930 Milliseconds Vbucket 840 2016-04-07T01:49:48.861-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6960 Milliseconds Vbucket 840 2016-04-07T01:49:48.891-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 6990 Milliseconds Vbucket 840 2016-04-07T01:49:48.915-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12000000 QueuedCount 50000 2016-04-07T01:49:48.921-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7020 Milliseconds Vbucket 840 2016-04-07T01:49:48.951-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7050 Milliseconds Vbucket 840 2016-04-07T01:49:48.981-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7080 Milliseconds Vbucket 840 2016-04-07T01:49:49.011-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7110 Milliseconds Vbucket 840 2016-04-07T01:49:49.041-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7140 Milliseconds Vbucket 840 2016-04-07T01:49:49.071-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7170 Milliseconds Vbucket 840 2016-04-07T01:49:49.101-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7200 Milliseconds Vbucket 840 2016-04-07T01:49:49.131-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7230 Milliseconds Vbucket 840 2016-04-07T01:49:49.161-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7260 Milliseconds Vbucket 840 2016-04-07T01:49:49.191-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7290 Milliseconds Vbucket 840 2016-04-07T01:49:49.221-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7320 Milliseconds Vbucket 840 2016-04-07T01:49:49.251-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7350 Milliseconds Vbucket 840 2016-04-07T01:49:49.281-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7380 Milliseconds Vbucket 840 2016-04-07T01:49:49.311-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7410 Milliseconds Vbucket 840 2016-04-07T01:49:49.341-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7440 Milliseconds Vbucket 840 2016-04-07T01:49:49.371-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7470 Milliseconds Vbucket 840 2016-04-07T01:49:49.401-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7500 Milliseconds Vbucket 840 2016-04-07T01:49:49.431-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7530 Milliseconds Vbucket 840 2016-04-07T01:49:49.442-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12010000 QueuedCount 50000 2016-04-07T01:49:49.461-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7560 Milliseconds Vbucket 840 2016-04-07T01:49:49.491-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7590 Milliseconds Vbucket 840 2016-04-07T01:49:49.521-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7620 Milliseconds Vbucket 840 2016-04-07T01:49:49.551-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7650 Milliseconds Vbucket 840 2016-04-07T01:49:49.581-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7680 Milliseconds Vbucket 840 2016-04-07T01:49:49.611-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7710 Milliseconds Vbucket 840 2016-04-07T01:49:49.641-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7740 Milliseconds Vbucket 840 2016-04-07T01:49:49.671-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7770 Milliseconds Vbucket 840 2016-04-07T01:49:49.701-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7800 Milliseconds Vbucket 840 2016-04-07T01:49:49.731-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7830 Milliseconds Vbucket 840 2016-04-07T01:49:49.761-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7860 Milliseconds Vbucket 840 2016-04-07T01:49:49.791-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7890 Milliseconds Vbucket 840 2016-04-07T01:49:49.821-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7920 Milliseconds Vbucket 840 2016-04-07T01:49:49.851-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7950 Milliseconds Vbucket 840 2016-04-07T01:49:49.881-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 7980 Milliseconds Vbucket 840 2016-04-07T01:49:49.911-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8010 Milliseconds Vbucket 840 2016-04-07T01:49:49.941-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8040 Milliseconds Vbucket 840 2016-04-07T01:49:49.969-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12020000 QueuedCount 50000 2016-04-07T01:49:49.971-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8070 Milliseconds Vbucket 840 2016-04-07T01:49:50.001-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8100 Milliseconds Vbucket 840 2016-04-07T01:49:50.031-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8130 Milliseconds Vbucket 840 2016-04-07T01:49:50.061-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8160 Milliseconds Vbucket 840 2016-04-07T01:49:50.091-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8190 Milliseconds Vbucket 840 2016-04-07T01:49:50.121-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8220 Milliseconds Vbucket 840 2016-04-07T01:49:50.151-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8250 Milliseconds Vbucket 840 2016-04-07T01:49:50.181-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8280 Milliseconds Vbucket 840 2016-04-07T01:49:50.211-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8310 Milliseconds Vbucket 840 2016-04-07T01:49:50.241-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8340 Milliseconds Vbucket 840 2016-04-07T01:49:50.271-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8370 Milliseconds Vbucket 840 2016-04-07T01:49:50.301-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8400 Milliseconds Vbucket 840 2016-04-07T01:49:50.331-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8430 Milliseconds Vbucket 840 2016-04-07T01:49:50.361-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8460 Milliseconds Vbucket 840 2016-04-07T01:49:50.391-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8490 Milliseconds Vbucket 840 2016-04-07T01:49:50.412-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12030000 QueuedCount 48338 2016-04-07T01:49:50.421-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8520 Milliseconds Vbucket 840 2016-04-07T01:49:50.451-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8550 Milliseconds Vbucket 840 2016-04-07T01:49:50.481-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8580 Milliseconds Vbucket 840 2016-04-07T01:49:50.511-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8610 Milliseconds Vbucket 840 2016-04-07T01:49:50.541-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8640 Milliseconds Vbucket 840 2016-04-07T01:49:50.571-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8670 Milliseconds Vbucket 840 2016-04-07T01:49:50.601-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8700 Milliseconds Vbucket 840 2016-04-07T01:49:50.631-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8730 Milliseconds Vbucket 840 2016-04-07T01:49:50.661-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8760 Milliseconds Vbucket 840 2016-04-07T01:49:50.691-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8790 Milliseconds Vbucket 840 2016-04-07T01:49:50.721-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8820 Milliseconds Vbucket 840 2016-04-07T01:49:50.751-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8850 Milliseconds Vbucket 840 2016-04-07T01:49:50.781-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8880 Milliseconds Vbucket 840 2016-04-07T01:49:50.811-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8910 Milliseconds Vbucket 840 2016-04-07T01:49:50.841-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8940 Milliseconds Vbucket 840 2016-04-07T01:49:50.871-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 8970 Milliseconds Vbucket 840 2016-04-07T01:49:50.901-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9000 Milliseconds Vbucket 840 2016-04-07T01:49:50.931-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9030 Milliseconds Vbucket 840 2016-04-07T01:49:50.932-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12040000 QueuedCount 38338 2016-04-07T01:49:50.961-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9060 Milliseconds Vbucket 840 2016-04-07T01:49:50.991-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9090 Milliseconds Vbucket 840 2016-04-07T01:49:51.021-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9120 Milliseconds Vbucket 840 2016-04-07T01:49:51.051-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9150 Milliseconds Vbucket 840 2016-04-07T01:49:51.081-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9180 Milliseconds Vbucket 840 2016-04-07T01:49:51.111-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9210 Milliseconds Vbucket 840 2016-04-07T01:49:51.141-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9240 Milliseconds Vbucket 840 2016-04-07T01:49:51.171-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9270 Milliseconds Vbucket 840 2016-04-07T01:49:51.201-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9300 Milliseconds Vbucket 840 2016-04-07T01:49:51.231-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9330 Milliseconds Vbucket 840 2016-04-07T01:49:51.261-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9360 Milliseconds Vbucket 840 2016-04-07T01:49:51.291-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9390 Milliseconds Vbucket 840 2016-04-07T01:49:51.321-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9420 Milliseconds Vbucket 840 2016-04-07T01:49:51.351-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9450 Milliseconds Vbucket 840 2016-04-07T01:49:51.351-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:49:51.364-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:49:51.365-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:49:51.381-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9480 Milliseconds Vbucket 840 2016-04-07T01:49:51.411-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9510 Milliseconds Vbucket 840 2016-04-07T01:49:51.441-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9540 Milliseconds Vbucket 840 2016-04-07T01:49:51.457-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12050000 QueuedCount 28338 2016-04-07T01:49:51.471-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9570 Milliseconds Vbucket 840 2016-04-07T01:49:51.501-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9600 Milliseconds Vbucket 840 2016-04-07T01:49:51.531-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9630 Milliseconds Vbucket 840 2016-04-07T01:49:51.561-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9660 Milliseconds Vbucket 840 2016-04-07T01:49:51.591-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9690 Milliseconds Vbucket 840 2016-04-07T01:49:51.621-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9720 Milliseconds Vbucket 840 2016-04-07T01:49:51.651-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9750 Milliseconds Vbucket 840 2016-04-07T01:49:51.681-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9780 Milliseconds Vbucket 840 2016-04-07T01:49:51.711-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9810 Milliseconds Vbucket 840 2016-04-07T01:49:51.741-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9840 Milliseconds Vbucket 840 2016-04-07T01:49:51.771-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9870 Milliseconds Vbucket 840 2016-04-07T01:49:51.801-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9900 Milliseconds Vbucket 840 2016-04-07T01:49:51.831-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9930 Milliseconds Vbucket 840 2016-04-07T01:49:51.861-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9960 Milliseconds Vbucket 840 2016-04-07T01:49:51.891-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 9990 Milliseconds Vbucket 840 2016-04-07T01:49:51.921-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10020 Milliseconds Vbucket 840 2016-04-07T01:49:51.951-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10050 Milliseconds Vbucket 840 2016-04-07T01:49:51.981-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10080 Milliseconds Vbucket 840 2016-04-07T01:49:51.997-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12060000 QueuedCount 18338 2016-04-07T01:49:52.011-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10110 Milliseconds Vbucket 840 2016-04-07T01:49:52.041-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10140 Milliseconds Vbucket 840 2016-04-07T01:49:52.071-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10170 Milliseconds Vbucket 840 2016-04-07T01:49:52.101-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10200 Milliseconds Vbucket 840 2016-04-07T01:49:52.131-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10230 Milliseconds Vbucket 840 2016-04-07T01:49:52.161-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10260 Milliseconds Vbucket 840 2016-04-07T01:49:52.191-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10290 Milliseconds Vbucket 840 2016-04-07T01:49:52.221-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10320 Milliseconds Vbucket 840 2016-04-07T01:49:52.251-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10350 Milliseconds Vbucket 840 2016-04-07T01:49:52.281-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10380 Milliseconds Vbucket 840 2016-04-07T01:49:52.311-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10410 Milliseconds Vbucket 840 2016-04-07T01:49:52.341-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10440 Milliseconds Vbucket 840 2016-04-07T01:49:52.371-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10470 Milliseconds Vbucket 840 2016-04-07T01:49:52.401-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10500 Milliseconds Vbucket 840 2016-04-07T01:49:52.431-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10530 Milliseconds Vbucket 840 2016-04-07T01:49:52.461-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10560 Milliseconds Vbucket 840 2016-04-07T01:49:52.491-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10590 Milliseconds Vbucket 840 2016-04-07T01:49:52.494-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12070000 QueuedCount 8338 2016-04-07T01:49:52.521-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10620 Milliseconds Vbucket 840 2016-04-07T01:49:52.551-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10650 Milliseconds Vbucket 840 2016-04-07T01:49:52.581-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10680 Milliseconds Vbucket 840 2016-04-07T01:49:52.611-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10710 Milliseconds Vbucket 840 2016-04-07T01:49:52.641-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10740 Milliseconds Vbucket 840 2016-04-07T01:49:52.671-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10770 Milliseconds Vbucket 840 2016-04-07T01:49:52.701-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10800 Milliseconds Vbucket 840 2016-04-07T01:49:52.731-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10830 Milliseconds Vbucket 840 2016-04-07T01:49:52.761-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10860 Milliseconds Vbucket 840 2016-04-07T01:49:52.791-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10890 Milliseconds Vbucket 840 2016-04-07T01:49:52.821-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10920 Milliseconds Vbucket 840 2016-04-07T01:49:52.851-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10950 Milliseconds Vbucket 840 2016-04-07T01:49:52.881-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 10980 Milliseconds Vbucket 840 2016-04-07T01:49:52.911-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11010 Milliseconds Vbucket 840 2016-04-07T01:49:52.941-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11040 Milliseconds Vbucket 840 2016-04-07T01:49:52.971-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11070 Milliseconds Vbucket 840 2016-04-07T01:49:53.001-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11100 Milliseconds Vbucket 840 2016-04-07T01:49:53.031-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11130 Milliseconds Vbucket 840 2016-04-07T01:49:53.061-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11160 Milliseconds Vbucket 840 2016-04-07T01:49:53.091-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11190 Milliseconds Vbucket 840 2016-04-07T01:49:53.121-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11220 Milliseconds Vbucket 840 2016-04-07T01:49:53.151-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11250 Milliseconds Vbucket 840 2016-04-07T01:49:53.181-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11280 Milliseconds Vbucket 840 2016-04-07T01:49:53.211-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11310 Milliseconds Vbucket 840 2016-04-07T01:49:53.241-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11340 Milliseconds Vbucket 840 2016-04-07T01:49:53.271-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11370 Milliseconds Vbucket 840 2016-04-07T01:49:53.301-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11400 Milliseconds Vbucket 840 2016-04-07T01:49:53.331-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11430 Milliseconds Vbucket 840 2016-04-07T01:49:53.361-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11460 Milliseconds Vbucket 840 2016-04-07T01:49:53.391-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11490 Milliseconds Vbucket 840 2016-04-07T01:49:53.421-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11520 Milliseconds Vbucket 840 2016-04-07T01:49:53.451-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11550 Milliseconds Vbucket 840 2016-04-07T01:49:53.481-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11580 Milliseconds Vbucket 840 2016-04-07T01:49:53.511-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11610 Milliseconds Vbucket 840 2016-04-07T01:49:53.541-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11640 Milliseconds Vbucket 840 2016-04-07T01:49:53.571-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11670 Milliseconds Vbucket 840 2016-04-07T01:49:53.601-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11700 Milliseconds Vbucket 840 2016-04-07T01:49:53.631-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11730 Milliseconds Vbucket 840 2016-04-07T01:49:53.661-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11760 Milliseconds Vbucket 840 2016-04-07T01:49:53.691-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11790 Milliseconds Vbucket 840 2016-04-07T01:49:53.719-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.513µs CommitTime 797.353197ms TotalFlushTime 8m26.60307691s TotalCommitTime 42.888388657s 2016-04-07T01:49:53.719-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 12078338, 12078338, 81 committed:true 2016-04-07T01:49:53.719-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 12078338, 12078338, 81 committed:true) SnapCreateDur 805.36016ms SnapOpenDur 53.461µs 2016-04-07T01:49:53.721-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11820 Milliseconds Vbucket 840 2016-04-07T01:49:53.728-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 26.17µs CommitTime 236.945µs TotalFlushTime 8m26.603134331s TotalCommitTime 42.888625602s 2016-04-07T01:49:53.728-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 12078339, 12078339, 82 committed:true 2016-04-07T01:49:53.728-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 12078339, 12078339, 82 committed:true) SnapCreateDur 8.358142ms SnapOpenDur 43.144µs 2016-04-07T01:49:53.73-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:53.734-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:53.735-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:53.739-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:53.741-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:53.744-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:53.748-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:53.75-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:53.751-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:53.751-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11850 Milliseconds Vbucket 840 2016-04-07T01:49:53.752-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:53.753-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:53.761-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12080000 QueuedCount 50000 2016-04-07T01:49:53.781-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11880 Milliseconds Vbucket 840 2016-04-07T01:49:53.811-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11910 Milliseconds Vbucket 840 2016-04-07T01:49:53.834-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:53.841-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11940 Milliseconds Vbucket 840 2016-04-07T01:49:53.871-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 11970 Milliseconds Vbucket 840 2016-04-07T01:49:53.901-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12000 Milliseconds Vbucket 840 2016-04-07T01:49:53.928-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:53.931-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12030 Milliseconds Vbucket 840 2016-04-07T01:49:53.961-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12060 Milliseconds Vbucket 840 2016-04-07T01:49:53.991-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12090 Milliseconds Vbucket 840 2016-04-07T01:49:54.021-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12120 Milliseconds Vbucket 840 2016-04-07T01:49:54.051-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12150 Milliseconds Vbucket 840 2016-04-07T01:49:54.081-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12180 Milliseconds Vbucket 840 2016-04-07T01:49:54.111-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12210 Milliseconds Vbucket 840 2016-04-07T01:49:54.124-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:54.141-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12240 Milliseconds Vbucket 840 2016-04-07T01:49:54.171-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12270 Milliseconds Vbucket 840 2016-04-07T01:49:54.201-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12300 Milliseconds Vbucket 840 2016-04-07T01:49:54.218-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12090000 QueuedCount 50000 2016-04-07T01:49:54.231-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12330 Milliseconds Vbucket 840 2016-04-07T01:49:54.261-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12360 Milliseconds Vbucket 840 2016-04-07T01:49:54.291-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12390 Milliseconds Vbucket 840 2016-04-07T01:49:54.316-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:54.321-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12420 Milliseconds Vbucket 840 2016-04-07T01:49:54.351-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12450 Milliseconds Vbucket 840 2016-04-07T01:49:54.381-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12480 Milliseconds Vbucket 840 2016-04-07T01:49:54.411-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12510 Milliseconds Vbucket 840 2016-04-07T01:49:54.441-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12540 Milliseconds Vbucket 840 2016-04-07T01:49:54.471-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12570 Milliseconds Vbucket 840 2016-04-07T01:49:54.493-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:54.501-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12600 Milliseconds Vbucket 840 2016-04-07T01:49:54.531-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12630 Milliseconds Vbucket 840 2016-04-07T01:49:54.561-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12660 Milliseconds Vbucket 840 2016-04-07T01:49:54.584-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:54.591-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12690 Milliseconds Vbucket 840 2016-04-07T01:49:54.621-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12720 Milliseconds Vbucket 840 2016-04-07T01:49:54.651-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12750 Milliseconds Vbucket 840 2016-04-07T01:49:54.681-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12780 Milliseconds Vbucket 840 2016-04-07T01:49:54.698-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12100000 QueuedCount 50000 2016-04-07T01:49:54.711-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12810 Milliseconds Vbucket 840 2016-04-07T01:49:54.741-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12840 Milliseconds Vbucket 840 2016-04-07T01:49:54.771-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12870 Milliseconds Vbucket 840 2016-04-07T01:49:54.801-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12900 Milliseconds Vbucket 840 2016-04-07T01:49:54.831-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12930 Milliseconds Vbucket 840 2016-04-07T01:49:54.861-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12960 Milliseconds Vbucket 840 2016-04-07T01:49:54.891-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 12990 Milliseconds Vbucket 840 2016-04-07T01:49:54.921-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13020 Milliseconds Vbucket 840 2016-04-07T01:49:54.951-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13050 Milliseconds Vbucket 840 2016-04-07T01:49:54.981-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13080 Milliseconds Vbucket 840 2016-04-07T01:49:55.011-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13110 Milliseconds Vbucket 840 2016-04-07T01:49:55.041-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13140 Milliseconds Vbucket 840 2016-04-07T01:49:55.071-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13170 Milliseconds Vbucket 840 2016-04-07T01:49:55.101-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13200 Milliseconds Vbucket 840 2016-04-07T01:49:55.131-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13230 Milliseconds Vbucket 840 2016-04-07T01:49:55.161-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13260 Milliseconds Vbucket 840 2016-04-07T01:49:55.181-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12110000 QueuedCount 50000 2016-04-07T01:49:55.191-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13290 Milliseconds Vbucket 840 2016-04-07T01:49:55.221-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13320 Milliseconds Vbucket 840 2016-04-07T01:49:55.251-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13350 Milliseconds Vbucket 840 2016-04-07T01:49:55.281-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13380 Milliseconds Vbucket 840 2016-04-07T01:49:55.311-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13410 Milliseconds Vbucket 840 2016-04-07T01:49:55.341-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13440 Milliseconds Vbucket 840 2016-04-07T01:49:55.371-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13470 Milliseconds Vbucket 840 2016-04-07T01:49:55.401-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13500 Milliseconds Vbucket 840 2016-04-07T01:49:55.431-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13530 Milliseconds Vbucket 840 2016-04-07T01:49:55.461-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13560 Milliseconds Vbucket 840 2016-04-07T01:49:55.491-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13590 Milliseconds Vbucket 840 2016-04-07T01:49:55.521-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13620 Milliseconds Vbucket 840 2016-04-07T01:49:55.551-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13650 Milliseconds Vbucket 840 2016-04-07T01:49:55.581-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13680 Milliseconds Vbucket 840 2016-04-07T01:49:55.611-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13710 Milliseconds Vbucket 840 2016-04-07T01:49:55.641-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13740 Milliseconds Vbucket 840 2016-04-07T01:49:55.671-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13770 Milliseconds Vbucket 840 2016-04-07T01:49:55.701-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13800 Milliseconds Vbucket 840 2016-04-07T01:49:55.708-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12120000 QueuedCount 50000 2016-04-07T01:49:55.731-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13830 Milliseconds Vbucket 840 2016-04-07T01:49:55.761-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13860 Milliseconds Vbucket 840 2016-04-07T01:49:55.791-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13890 Milliseconds Vbucket 840 2016-04-07T01:49:55.821-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13920 Milliseconds Vbucket 840 2016-04-07T01:49:55.851-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13950 Milliseconds Vbucket 840 2016-04-07T01:49:55.881-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 13980 Milliseconds Vbucket 840 2016-04-07T01:49:55.911-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14010 Milliseconds Vbucket 840 2016-04-07T01:49:55.911-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:55.941-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14040 Milliseconds Vbucket 840 2016-04-07T01:49:55.971-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14070 Milliseconds Vbucket 840 2016-04-07T01:49:56.001-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14100 Milliseconds Vbucket 840 2016-04-07T01:49:56.024-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:56.031-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14130 Milliseconds Vbucket 840 2016-04-07T01:49:56.061-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14160 Milliseconds Vbucket 840 2016-04-07T01:49:56.091-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14190 Milliseconds Vbucket 840 2016-04-07T01:49:56.121-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14220 Milliseconds Vbucket 840 2016-04-07T01:49:56.151-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14250 Milliseconds Vbucket 840 2016-04-07T01:49:56.181-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14280 Milliseconds Vbucket 840 2016-04-07T01:49:56.208-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12130000 QueuedCount 50000 2016-04-07T01:49:56.211-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14310 Milliseconds Vbucket 840 2016-04-07T01:49:56.241-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14340 Milliseconds Vbucket 840 2016-04-07T01:49:56.271-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14370 Milliseconds Vbucket 840 2016-04-07T01:49:56.301-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14400 Milliseconds Vbucket 840 2016-04-07T01:49:56.305-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:56.331-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14430 Milliseconds Vbucket 840 2016-04-07T01:49:56.361-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14460 Milliseconds Vbucket 840 2016-04-07T01:49:56.391-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14490 Milliseconds Vbucket 840 2016-04-07T01:49:56.408-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:56.421-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14520 Milliseconds Vbucket 840 2016-04-07T01:49:56.451-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14550 Milliseconds Vbucket 840 2016-04-07T01:49:56.481-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14580 Milliseconds Vbucket 840 2016-04-07T01:49:56.511-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14610 Milliseconds Vbucket 840 2016-04-07T01:49:56.527-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:56.541-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14640 Milliseconds Vbucket 840 2016-04-07T01:49:56.571-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14670 Milliseconds Vbucket 840 2016-04-07T01:49:56.601-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14700 Milliseconds Vbucket 840 2016-04-07T01:49:56.631-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14730 Milliseconds Vbucket 840 2016-04-07T01:49:56.661-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14760 Milliseconds Vbucket 840 2016-04-07T01:49:56.691-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14790 Milliseconds Vbucket 840 2016-04-07T01:49:56.721-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14820 Milliseconds Vbucket 840 2016-04-07T01:49:56.728-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12140000 QueuedCount 50000 2016-04-07T01:49:56.735-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:56.751-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14850 Milliseconds Vbucket 840 2016-04-07T01:49:56.781-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14880 Milliseconds Vbucket 840 2016-04-07T01:49:56.811-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14910 Milliseconds Vbucket 840 2016-04-07T01:49:56.841-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14940 Milliseconds Vbucket 840 2016-04-07T01:49:56.849-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:56.871-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 14970 Milliseconds Vbucket 840 2016-04-07T01:49:56.901-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15000 Milliseconds Vbucket 840 2016-04-07T01:49:56.931-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15030 Milliseconds Vbucket 840 2016-04-07T01:49:56.961-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15060 Milliseconds Vbucket 840 2016-04-07T01:49:56.991-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15090 Milliseconds Vbucket 840 2016-04-07T01:49:57.021-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15120 Milliseconds Vbucket 840 2016-04-07T01:49:57.051-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15150 Milliseconds Vbucket 840 2016-04-07T01:49:57.081-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15180 Milliseconds Vbucket 840 2016-04-07T01:49:57.111-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15210 Milliseconds Vbucket 840 2016-04-07T01:49:57.141-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15240 Milliseconds Vbucket 840 2016-04-07T01:49:57.148-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:57.169-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12150000 QueuedCount 50000 2016-04-07T01:49:57.171-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15270 Milliseconds Vbucket 840 2016-04-07T01:49:57.201-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15300 Milliseconds Vbucket 840 2016-04-07T01:49:57.231-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15330 Milliseconds Vbucket 840 2016-04-07T01:49:57.261-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15360 Milliseconds Vbucket 840 2016-04-07T01:49:57.269-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:57.291-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15390 Milliseconds Vbucket 840 2016-04-07T01:49:57.321-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15420 Milliseconds Vbucket 840 2016-04-07T01:49:57.351-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15450 Milliseconds Vbucket 840 2016-04-07T01:49:57.381-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15480 Milliseconds Vbucket 840 2016-04-07T01:49:57.411-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15510 Milliseconds Vbucket 840 2016-04-07T01:49:57.441-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15540 Milliseconds Vbucket 840 2016-04-07T01:49:57.471-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15570 Milliseconds Vbucket 840 2016-04-07T01:49:57.476-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:57.501-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15600 Milliseconds Vbucket 840 2016-04-07T01:49:57.531-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15630 Milliseconds Vbucket 840 2016-04-07T01:49:57.561-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15660 Milliseconds Vbucket 840 2016-04-07T01:49:57.591-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15690 Milliseconds Vbucket 840 2016-04-07T01:49:57.621-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15720 Milliseconds Vbucket 840 2016-04-07T01:49:57.651-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15750 Milliseconds Vbucket 840 2016-04-07T01:49:57.681-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15780 Milliseconds Vbucket 840 2016-04-07T01:49:57.693-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:57.711-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15810 Milliseconds Vbucket 840 2016-04-07T01:49:57.712-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12160000 QueuedCount 50000 2016-04-07T01:49:57.741-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15840 Milliseconds Vbucket 840 2016-04-07T01:49:57.771-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15870 Milliseconds Vbucket 840 2016-04-07T01:49:57.801-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15900 Milliseconds Vbucket 840 2016-04-07T01:49:57.815-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:57.831-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15930 Milliseconds Vbucket 840 2016-04-07T01:49:57.861-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15960 Milliseconds Vbucket 840 2016-04-07T01:49:57.891-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 15990 Milliseconds Vbucket 840 2016-04-07T01:49:57.921-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16020 Milliseconds Vbucket 840 2016-04-07T01:49:57.929-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:57.951-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16050 Milliseconds Vbucket 840 2016-04-07T01:49:57.981-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16080 Milliseconds Vbucket 840 2016-04-07T01:49:58.011-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16110 Milliseconds Vbucket 840 2016-04-07T01:49:58.041-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16140 Milliseconds Vbucket 840 2016-04-07T01:49:58.071-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16170 Milliseconds Vbucket 840 2016-04-07T01:49:58.101-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16200 Milliseconds Vbucket 840 2016-04-07T01:49:58.131-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16230 Milliseconds Vbucket 840 2016-04-07T01:49:58.143-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:58.161-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16260 Milliseconds Vbucket 840 2016-04-07T01:49:58.191-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16290 Milliseconds Vbucket 840 2016-04-07T01:49:58.221-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16320 Milliseconds Vbucket 840 2016-04-07T01:49:58.251-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16350 Milliseconds Vbucket 840 2016-04-07T01:49:58.27-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12170000 QueuedCount 50000 2016-04-07T01:49:58.281-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16380 Milliseconds Vbucket 840 2016-04-07T01:49:58.311-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16410 Milliseconds Vbucket 840 2016-04-07T01:49:58.341-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16440 Milliseconds Vbucket 840 2016-04-07T01:49:58.371-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:58.371-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16470 Milliseconds Vbucket 840 2016-04-07T01:49:58.401-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16500 Milliseconds Vbucket 840 2016-04-07T01:49:58.431-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16530 Milliseconds Vbucket 840 2016-04-07T01:49:58.461-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16560 Milliseconds Vbucket 840 2016-04-07T01:49:58.491-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16590 Milliseconds Vbucket 840 2016-04-07T01:49:58.521-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16620 Milliseconds Vbucket 840 2016-04-07T01:49:58.551-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16650 Milliseconds Vbucket 840 2016-04-07T01:49:58.581-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16680 Milliseconds Vbucket 840 2016-04-07T01:49:58.598-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:58.611-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16710 Milliseconds Vbucket 840 2016-04-07T01:49:58.641-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16740 Milliseconds Vbucket 840 2016-04-07T01:49:58.671-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16770 Milliseconds Vbucket 840 2016-04-07T01:49:58.701-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16800 Milliseconds Vbucket 840 2016-04-07T01:49:58.731-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16830 Milliseconds Vbucket 840 2016-04-07T01:49:58.761-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16860 Milliseconds Vbucket 840 2016-04-07T01:49:58.791-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16890 Milliseconds Vbucket 840 2016-04-07T01:49:58.82-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12180000 QueuedCount 50000 2016-04-07T01:49:58.821-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16920 Milliseconds Vbucket 840 2016-04-07T01:49:58.851-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16950 Milliseconds Vbucket 840 2016-04-07T01:49:58.881-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 16980 Milliseconds Vbucket 840 2016-04-07T01:49:58.911-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17010 Milliseconds Vbucket 840 2016-04-07T01:49:58.937-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:49:58.941-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17040 Milliseconds Vbucket 840 2016-04-07T01:49:58.971-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17070 Milliseconds Vbucket 840 2016-04-07T01:49:59.001-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17100 Milliseconds Vbucket 840 2016-04-07T01:49:59.031-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17130 Milliseconds Vbucket 840 2016-04-07T01:49:59.061-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17160 Milliseconds Vbucket 840 2016-04-07T01:49:59.091-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17190 Milliseconds Vbucket 840 2016-04-07T01:49:59.121-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17220 Milliseconds Vbucket 840 2016-04-07T01:49:59.151-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17250 Milliseconds Vbucket 840 2016-04-07T01:49:59.181-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17280 Milliseconds Vbucket 840 2016-04-07T01:49:59.211-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17310 Milliseconds Vbucket 840 2016-04-07T01:49:59.241-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17340 Milliseconds Vbucket 840 2016-04-07T01:49:59.271-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17370 Milliseconds Vbucket 840 2016-04-07T01:49:59.301-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17400 Milliseconds Vbucket 840 2016-04-07T01:49:59.331-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17430 Milliseconds Vbucket 840 2016-04-07T01:49:59.35-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12190000 QueuedCount 46806 2016-04-07T01:49:59.361-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17460 Milliseconds Vbucket 840 2016-04-07T01:49:59.391-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17490 Milliseconds Vbucket 840 2016-04-07T01:49:59.421-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17520 Milliseconds Vbucket 840 2016-04-07T01:49:59.451-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17550 Milliseconds Vbucket 840 2016-04-07T01:49:59.481-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17580 Milliseconds Vbucket 840 2016-04-07T01:49:59.511-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17610 Milliseconds Vbucket 840 2016-04-07T01:49:59.541-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17640 Milliseconds Vbucket 840 2016-04-07T01:49:59.571-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17670 Milliseconds Vbucket 840 2016-04-07T01:49:59.601-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17700 Milliseconds Vbucket 840 2016-04-07T01:49:59.631-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17730 Milliseconds Vbucket 840 2016-04-07T01:49:59.661-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17760 Milliseconds Vbucket 840 2016-04-07T01:49:59.691-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17790 Milliseconds Vbucket 840 2016-04-07T01:49:59.721-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17820 Milliseconds Vbucket 840 2016-04-07T01:49:59.751-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17850 Milliseconds Vbucket 840 2016-04-07T01:49:59.781-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17880 Milliseconds Vbucket 840 2016-04-07T01:49:59.811-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17910 Milliseconds Vbucket 840 2016-04-07T01:49:59.841-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17940 Milliseconds Vbucket 840 2016-04-07T01:49:59.856-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12200000 QueuedCount 36806 2016-04-07T01:49:59.871-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 17970 Milliseconds Vbucket 840 2016-04-07T01:49:59.901-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18000 Milliseconds Vbucket 840 2016-04-07T01:49:59.931-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18030 Milliseconds Vbucket 840 2016-04-07T01:49:59.961-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18060 Milliseconds Vbucket 840 2016-04-07T01:49:59.991-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18090 Milliseconds Vbucket 840 2016-04-07T01:50:00.021-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18120 Milliseconds Vbucket 840 2016-04-07T01:50:00.051-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18150 Milliseconds Vbucket 840 2016-04-07T01:50:00.081-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18180 Milliseconds Vbucket 840 2016-04-07T01:50:00.111-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18210 Milliseconds Vbucket 840 2016-04-07T01:50:00.141-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18240 Milliseconds Vbucket 840 2016-04-07T01:50:00.171-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18270 Milliseconds Vbucket 840 2016-04-07T01:50:00.201-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18300 Milliseconds Vbucket 840 2016-04-07T01:50:00.231-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18330 Milliseconds Vbucket 840 2016-04-07T01:50:00.261-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18360 Milliseconds Vbucket 840 2016-04-07T01:50:00.291-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18390 Milliseconds Vbucket 840 2016-04-07T01:50:00.321-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18420 Milliseconds Vbucket 840 2016-04-07T01:50:00.33-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12210000 QueuedCount 26806 2016-04-07T01:50:00.351-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18450 Milliseconds Vbucket 840 2016-04-07T01:50:00.381-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18480 Milliseconds Vbucket 840 2016-04-07T01:50:00.411-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18510 Milliseconds Vbucket 840 2016-04-07T01:50:00.441-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18540 Milliseconds Vbucket 840 2016-04-07T01:50:00.471-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18570 Milliseconds Vbucket 840 2016-04-07T01:50:00.501-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18600 Milliseconds Vbucket 840 2016-04-07T01:50:00.531-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18630 Milliseconds Vbucket 840 2016-04-07T01:50:00.561-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18660 Milliseconds Vbucket 840 2016-04-07T01:50:00.591-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18690 Milliseconds Vbucket 840 2016-04-07T01:50:00.621-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18720 Milliseconds Vbucket 840 2016-04-07T01:50:00.651-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18750 Milliseconds Vbucket 840 2016-04-07T01:50:00.681-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18780 Milliseconds Vbucket 840 2016-04-07T01:50:00.711-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18810 Milliseconds Vbucket 840 2016-04-07T01:50:00.741-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18840 Milliseconds Vbucket 840 2016-04-07T01:50:00.771-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18870 Milliseconds Vbucket 840 2016-04-07T01:50:00.801-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18900 Milliseconds Vbucket 840 2016-04-07T01:50:00.806-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12220000 QueuedCount 16806 2016-04-07T01:50:00.831-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18930 Milliseconds Vbucket 840 2016-04-07T01:50:00.861-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18960 Milliseconds Vbucket 840 2016-04-07T01:50:00.891-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 18990 Milliseconds Vbucket 840 2016-04-07T01:50:00.921-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19020 Milliseconds Vbucket 840 2016-04-07T01:50:00.951-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19050 Milliseconds Vbucket 840 2016-04-07T01:50:00.981-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19080 Milliseconds Vbucket 840 2016-04-07T01:50:01.011-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19110 Milliseconds Vbucket 840 2016-04-07T01:50:01.041-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19140 Milliseconds Vbucket 840 2016-04-07T01:50:01.071-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19170 Milliseconds Vbucket 840 2016-04-07T01:50:01.101-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19200 Milliseconds Vbucket 840 2016-04-07T01:50:01.131-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19230 Milliseconds Vbucket 840 2016-04-07T01:50:01.161-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19260 Milliseconds Vbucket 840 2016-04-07T01:50:01.191-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19290 Milliseconds Vbucket 840 2016-04-07T01:50:01.221-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19320 Milliseconds Vbucket 840 2016-04-07T01:50:01.251-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19350 Milliseconds Vbucket 840 2016-04-07T01:50:01.281-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19380 Milliseconds Vbucket 840 2016-04-07T01:50:01.284-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12230000 QueuedCount 6806 2016-04-07T01:50:01.311-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19410 Milliseconds Vbucket 840 2016-04-07T01:50:01.341-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19440 Milliseconds Vbucket 840 2016-04-07T01:50:01.371-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19470 Milliseconds Vbucket 840 2016-04-07T01:50:01.399-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:50:01.401-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19500 Milliseconds Vbucket 840 2016-04-07T01:50:01.413-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:50:01.413-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:50:01.431-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19530 Milliseconds Vbucket 840 2016-04-07T01:50:01.461-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19560 Milliseconds Vbucket 840 2016-04-07T01:50:01.491-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19590 Milliseconds Vbucket 840 2016-04-07T01:50:01.521-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19620 Milliseconds Vbucket 840 2016-04-07T01:50:01.551-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19650 Milliseconds Vbucket 840 2016-04-07T01:50:01.581-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19680 Milliseconds Vbucket 840 2016-04-07T01:50:01.611-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19710 Milliseconds Vbucket 840 2016-04-07T01:50:01.641-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19740 Milliseconds Vbucket 840 2016-04-07T01:50:01.671-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19770 Milliseconds Vbucket 840 2016-04-07T01:50:01.701-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19800 Milliseconds Vbucket 840 2016-04-07T01:50:01.731-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19830 Milliseconds Vbucket 840 2016-04-07T01:50:01.761-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19860 Milliseconds Vbucket 840 2016-04-07T01:50:01.791-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19890 Milliseconds Vbucket 840 2016-04-07T01:50:01.821-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19920 Milliseconds Vbucket 840 2016-04-07T01:50:01.851-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19950 Milliseconds Vbucket 840 2016-04-07T01:50:01.881-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 19980 Milliseconds Vbucket 840 2016-04-07T01:50:01.911-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20010 Milliseconds Vbucket 840 2016-04-07T01:50:01.941-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20040 Milliseconds Vbucket 840 2016-04-07T01:50:01.971-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20070 Milliseconds Vbucket 840 2016-04-07T01:50:02.001-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20100 Milliseconds Vbucket 840 2016-04-07T01:50:02.031-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20130 Milliseconds Vbucket 840 2016-04-07T01:50:02.061-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20160 Milliseconds Vbucket 840 2016-04-07T01:50:02.091-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20190 Milliseconds Vbucket 840 2016-04-07T01:50:02.121-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20220 Milliseconds Vbucket 840 2016-04-07T01:50:02.151-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20250 Milliseconds Vbucket 840 2016-04-07T01:50:02.181-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20280 Milliseconds Vbucket 840 2016-04-07T01:50:02.211-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20310 Milliseconds Vbucket 840 2016-04-07T01:50:02.241-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20340 Milliseconds Vbucket 840 2016-04-07T01:50:02.271-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20370 Milliseconds Vbucket 840 2016-04-07T01:50:02.301-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20400 Milliseconds Vbucket 840 2016-04-07T01:50:02.329-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.38µs CommitTime 709.075853ms TotalFlushTime 8m34.343370298s TotalCommitTime 43.597701455s 2016-04-07T01:50:02.329-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 12236806, 12236806, 83 committed:true 2016-04-07T01:50:02.329-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 12236806, 12236806, 83 committed:true) SnapCreateDur 717.066199ms SnapOpenDur 43.895µs 2016-04-07T01:50:02.331-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20430 Milliseconds Vbucket 840 2016-04-07T01:50:02.332-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:02.336-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:02.339-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:02.342-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:02.345-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:02.348-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:02.361-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20460 Milliseconds Vbucket 840 2016-04-07T01:50:02.391-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20490 Milliseconds Vbucket 840 2016-04-07T01:50:02.421-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20520 Milliseconds Vbucket 840 2016-04-07T01:50:02.451-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20550 Milliseconds Vbucket 840 2016-04-07T01:50:02.453-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12240000 QueuedCount 50000 2016-04-07T01:50:02.481-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20580 Milliseconds Vbucket 840 2016-04-07T01:50:02.511-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20610 Milliseconds Vbucket 840 2016-04-07T01:50:02.541-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20640 Milliseconds Vbucket 840 2016-04-07T01:50:02.571-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20670 Milliseconds Vbucket 840 2016-04-07T01:50:02.601-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20700 Milliseconds Vbucket 840 2016-04-07T01:50:02.631-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20730 Milliseconds Vbucket 840 2016-04-07T01:50:02.661-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20760 Milliseconds Vbucket 840 2016-04-07T01:50:02.691-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20790 Milliseconds Vbucket 840 2016-04-07T01:50:02.721-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20820 Milliseconds Vbucket 840 2016-04-07T01:50:02.751-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20850 Milliseconds Vbucket 840 2016-04-07T01:50:02.781-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20880 Milliseconds Vbucket 840 2016-04-07T01:50:02.811-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20910 Milliseconds Vbucket 840 2016-04-07T01:50:02.841-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20940 Milliseconds Vbucket 840 2016-04-07T01:50:02.871-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 20970 Milliseconds Vbucket 840 2016-04-07T01:50:02.901-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21000 Milliseconds Vbucket 840 2016-04-07T01:50:02.913-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12250000 QueuedCount 50000 2016-04-07T01:50:02.931-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21030 Milliseconds Vbucket 840 2016-04-07T01:50:02.961-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21060 Milliseconds Vbucket 840 2016-04-07T01:50:02.991-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21090 Milliseconds Vbucket 840 2016-04-07T01:50:03.021-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21120 Milliseconds Vbucket 840 2016-04-07T01:50:03.051-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21150 Milliseconds Vbucket 840 2016-04-07T01:50:03.081-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21180 Milliseconds Vbucket 840 2016-04-07T01:50:03.111-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21210 Milliseconds Vbucket 840 2016-04-07T01:50:03.141-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21240 Milliseconds Vbucket 840 2016-04-07T01:50:03.171-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21270 Milliseconds Vbucket 840 2016-04-07T01:50:03.201-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21300 Milliseconds Vbucket 840 2016-04-07T01:50:03.231-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21330 Milliseconds Vbucket 840 2016-04-07T01:50:03.261-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21360 Milliseconds Vbucket 840 2016-04-07T01:50:03.291-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21390 Milliseconds Vbucket 840 2016-04-07T01:50:03.321-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21420 Milliseconds Vbucket 840 2016-04-07T01:50:03.351-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21450 Milliseconds Vbucket 840 2016-04-07T01:50:03.371-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12260000 QueuedCount 50000 2016-04-07T01:50:03.381-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21480 Milliseconds Vbucket 840 2016-04-07T01:50:03.411-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21510 Milliseconds Vbucket 840 2016-04-07T01:50:03.441-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21540 Milliseconds Vbucket 840 2016-04-07T01:50:03.471-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21570 Milliseconds Vbucket 840 2016-04-07T01:50:03.501-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21600 Milliseconds Vbucket 840 2016-04-07T01:50:03.531-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21630 Milliseconds Vbucket 840 2016-04-07T01:50:03.561-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21660 Milliseconds Vbucket 840 2016-04-07T01:50:03.591-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21690 Milliseconds Vbucket 840 2016-04-07T01:50:03.621-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21720 Milliseconds Vbucket 840 2016-04-07T01:50:03.651-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21750 Milliseconds Vbucket 840 2016-04-07T01:50:03.681-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21780 Milliseconds Vbucket 840 2016-04-07T01:50:03.711-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21810 Milliseconds Vbucket 840 2016-04-07T01:50:03.741-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21840 Milliseconds Vbucket 840 2016-04-07T01:50:03.771-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21870 Milliseconds Vbucket 840 2016-04-07T01:50:03.801-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21900 Milliseconds Vbucket 840 2016-04-07T01:50:03.831-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21930 Milliseconds Vbucket 840 2016-04-07T01:50:03.848-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12270000 QueuedCount 50000 2016-04-07T01:50:03.861-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21960 Milliseconds Vbucket 840 2016-04-07T01:50:03.891-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 21990 Milliseconds Vbucket 840 2016-04-07T01:50:03.921-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22020 Milliseconds Vbucket 840 2016-04-07T01:50:03.951-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22050 Milliseconds Vbucket 840 2016-04-07T01:50:03.981-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22080 Milliseconds Vbucket 840 2016-04-07T01:50:04.011-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22110 Milliseconds Vbucket 840 2016-04-07T01:50:04.041-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22140 Milliseconds Vbucket 840 2016-04-07T01:50:04.062-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:04.073-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22170 Milliseconds Vbucket 840 2016-04-07T01:50:04.11-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22200 Milliseconds Vbucket 840 2016-04-07T01:50:04.15-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22230 Milliseconds Vbucket 840 2016-04-07T01:50:04.17-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22260 Milliseconds Vbucket 840 2016-04-07T01:50:04.191-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22290 Milliseconds Vbucket 840 2016-04-07T01:50:04.235-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22320 Milliseconds Vbucket 840 2016-04-07T01:50:04.255-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22350 Milliseconds Vbucket 840 2016-04-07T01:50:04.295-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22380 Milliseconds Vbucket 840 2016-04-07T01:50:04.315-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22410 Milliseconds Vbucket 840 2016-04-07T01:50:04.341-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22440 Milliseconds Vbucket 840 2016-04-07T01:50:04.376-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22470 Milliseconds Vbucket 840 2016-04-07T01:50:04.41-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22500 Milliseconds Vbucket 840 2016-04-07T01:50:04.451-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22530 Milliseconds Vbucket 840 2016-04-07T01:50:04.471-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22560 Milliseconds Vbucket 840 2016-04-07T01:50:04.491-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22590 Milliseconds Vbucket 840 2016-04-07T01:50:04.495-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12280000 QueuedCount 49969 2016-04-07T01:50:04.531-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22620 Milliseconds Vbucket 840 2016-04-07T01:50:05.027-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22650 Milliseconds Vbucket 840 2016-04-07T01:50:05.027-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22680 Milliseconds Vbucket 840 2016-04-07T01:50:05.031-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22710 Milliseconds Vbucket 840 2016-04-07T01:50:05.061-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22740 Milliseconds Vbucket 840 2016-04-07T01:50:05.091-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22770 Milliseconds Vbucket 840 2016-04-07T01:50:05.121-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22800 Milliseconds Vbucket 840 2016-04-07T01:50:05.151-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22830 Milliseconds Vbucket 840 2016-04-07T01:50:05.181-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22860 Milliseconds Vbucket 840 2016-04-07T01:50:05.211-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22890 Milliseconds Vbucket 840 2016-04-07T01:50:05.241-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22920 Milliseconds Vbucket 840 2016-04-07T01:50:05.271-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22950 Milliseconds Vbucket 840 2016-04-07T01:50:05.301-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 22980 Milliseconds Vbucket 840 2016-04-07T01:50:05.331-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23010 Milliseconds Vbucket 840 2016-04-07T01:50:05.34-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12290000 QueuedCount 41180 2016-04-07T01:50:05.361-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23040 Milliseconds Vbucket 840 2016-04-07T01:50:05.391-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23070 Milliseconds Vbucket 840 2016-04-07T01:50:05.421-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23100 Milliseconds Vbucket 840 2016-04-07T01:50:05.451-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23130 Milliseconds Vbucket 840 2016-04-07T01:50:05.481-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23160 Milliseconds Vbucket 840 2016-04-07T01:50:05.511-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23190 Milliseconds Vbucket 840 2016-04-07T01:50:05.541-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23220 Milliseconds Vbucket 840 2016-04-07T01:50:05.571-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23250 Milliseconds Vbucket 840 2016-04-07T01:50:05.601-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23280 Milliseconds Vbucket 840 2016-04-07T01:50:05.631-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23310 Milliseconds Vbucket 840 2016-04-07T01:50:05.661-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23340 Milliseconds Vbucket 840 2016-04-07T01:50:05.691-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23370 Milliseconds Vbucket 840 2016-04-07T01:50:05.721-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23400 Milliseconds Vbucket 840 2016-04-07T01:50:05.751-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23430 Milliseconds Vbucket 840 2016-04-07T01:50:05.781-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23460 Milliseconds Vbucket 840 2016-04-07T01:50:05.811-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23490 Milliseconds Vbucket 840 2016-04-07T01:50:05.818-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12300000 QueuedCount 31180 2016-04-07T01:50:05.841-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23520 Milliseconds Vbucket 840 2016-04-07T01:50:05.871-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23550 Milliseconds Vbucket 840 2016-04-07T01:50:05.901-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23580 Milliseconds Vbucket 840 2016-04-07T01:50:05.931-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23610 Milliseconds Vbucket 840 2016-04-07T01:50:05.961-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23640 Milliseconds Vbucket 840 2016-04-07T01:50:05.991-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23670 Milliseconds Vbucket 840 2016-04-07T01:50:06.021-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23700 Milliseconds Vbucket 840 2016-04-07T01:50:06.051-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23730 Milliseconds Vbucket 840 2016-04-07T01:50:06.081-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23760 Milliseconds Vbucket 840 2016-04-07T01:50:06.111-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23790 Milliseconds Vbucket 840 2016-04-07T01:50:06.141-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23820 Milliseconds Vbucket 840 2016-04-07T01:50:06.171-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23850 Milliseconds Vbucket 840 2016-04-07T01:50:06.201-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23880 Milliseconds Vbucket 840 2016-04-07T01:50:06.231-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23910 Milliseconds Vbucket 840 2016-04-07T01:50:06.261-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23940 Milliseconds Vbucket 840 2016-04-07T01:50:06.291-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 23970 Milliseconds Vbucket 840 2016-04-07T01:50:06.31-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12310000 QueuedCount 21180 2016-04-07T01:50:06.321-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24000 Milliseconds Vbucket 840 2016-04-07T01:50:06.351-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24030 Milliseconds Vbucket 840 2016-04-07T01:50:06.381-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24060 Milliseconds Vbucket 840 2016-04-07T01:50:06.411-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24090 Milliseconds Vbucket 840 2016-04-07T01:50:06.441-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24120 Milliseconds Vbucket 840 2016-04-07T01:50:06.471-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24150 Milliseconds Vbucket 840 2016-04-07T01:50:06.501-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24180 Milliseconds Vbucket 840 2016-04-07T01:50:06.531-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24210 Milliseconds Vbucket 840 2016-04-07T01:50:06.561-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24240 Milliseconds Vbucket 840 2016-04-07T01:50:06.591-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24270 Milliseconds Vbucket 840 2016-04-07T01:50:06.621-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24300 Milliseconds Vbucket 840 2016-04-07T01:50:06.651-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24330 Milliseconds Vbucket 840 2016-04-07T01:50:06.681-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24360 Milliseconds Vbucket 840 2016-04-07T01:50:06.711-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24390 Milliseconds Vbucket 840 2016-04-07T01:50:06.741-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24420 Milliseconds Vbucket 840 2016-04-07T01:50:06.771-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24450 Milliseconds Vbucket 840 2016-04-07T01:50:06.794-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12320000 QueuedCount 11180 2016-04-07T01:50:06.801-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24480 Milliseconds Vbucket 840 2016-04-07T01:50:06.831-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24510 Milliseconds Vbucket 840 2016-04-07T01:50:06.861-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24540 Milliseconds Vbucket 840 2016-04-07T01:50:06.891-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24570 Milliseconds Vbucket 840 2016-04-07T01:50:06.921-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24600 Milliseconds Vbucket 840 2016-04-07T01:50:06.951-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24630 Milliseconds Vbucket 840 2016-04-07T01:50:06.981-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24660 Milliseconds Vbucket 840 2016-04-07T01:50:07.011-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24690 Milliseconds Vbucket 840 2016-04-07T01:50:07.041-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24720 Milliseconds Vbucket 840 2016-04-07T01:50:07.071-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24750 Milliseconds Vbucket 840 2016-04-07T01:50:07.101-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24780 Milliseconds Vbucket 840 2016-04-07T01:50:07.131-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24810 Milliseconds Vbucket 840 2016-04-07T01:50:07.161-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24840 Milliseconds Vbucket 840 2016-04-07T01:50:07.191-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24870 Milliseconds Vbucket 840 2016-04-07T01:50:07.221-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24900 Milliseconds Vbucket 840 2016-04-07T01:50:07.251-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24930 Milliseconds Vbucket 840 2016-04-07T01:50:07.281-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24960 Milliseconds Vbucket 840 2016-04-07T01:50:07.288-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12330000 QueuedCount 1180 2016-04-07T01:50:07.311-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 24990 Milliseconds Vbucket 840 2016-04-07T01:50:07.341-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25020 Milliseconds Vbucket 840 2016-04-07T01:50:07.371-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25050 Milliseconds Vbucket 840 2016-04-07T01:50:07.401-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25080 Milliseconds Vbucket 840 2016-04-07T01:50:07.431-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25110 Milliseconds Vbucket 840 2016-04-07T01:50:07.461-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25140 Milliseconds Vbucket 840 2016-04-07T01:50:07.491-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25170 Milliseconds Vbucket 840 2016-04-07T01:50:07.521-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25200 Milliseconds Vbucket 840 2016-04-07T01:50:07.551-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25230 Milliseconds Vbucket 840 2016-04-07T01:50:07.581-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25260 Milliseconds Vbucket 840 2016-04-07T01:50:07.611-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25290 Milliseconds Vbucket 840 2016-04-07T01:50:07.641-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25320 Milliseconds Vbucket 840 2016-04-07T01:50:07.671-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25350 Milliseconds Vbucket 840 2016-04-07T01:50:07.701-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25380 Milliseconds Vbucket 840 2016-04-07T01:50:07.731-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25410 Milliseconds Vbucket 840 2016-04-07T01:50:07.761-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25440 Milliseconds Vbucket 840 2016-04-07T01:50:07.791-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25470 Milliseconds Vbucket 840 2016-04-07T01:50:07.821-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25500 Milliseconds Vbucket 840 2016-04-07T01:50:07.851-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25530 Milliseconds Vbucket 840 2016-04-07T01:50:07.881-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25560 Milliseconds Vbucket 840 2016-04-07T01:50:07.911-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25590 Milliseconds Vbucket 840 2016-04-07T01:50:07.941-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25620 Milliseconds Vbucket 840 2016-04-07T01:50:07.971-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25650 Milliseconds Vbucket 840 2016-04-07T01:50:08.001-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25680 Milliseconds Vbucket 840 2016-04-07T01:50:08.031-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25710 Milliseconds Vbucket 840 2016-04-07T01:50:08.061-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 25740 Milliseconds Vbucket 840 2016-04-07T01:50:08.062-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.763µs CommitTime 672.484219ms TotalFlushTime 8m39.3044029s TotalCommitTime 44.270185674s 2016-04-07T01:50:08.062-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 12331180, 12331180, 84 committed:true 2016-04-07T01:50:08.062-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 12331180, 12331180, 84 committed:true) SnapCreateDur 680.474938ms SnapOpenDur 50.4µs 2016-04-07T01:50:08.066-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:08.07-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:08.073-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:08.075-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:08.076-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:08.079-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:08.083-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:08.084-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:08.086-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:08.088-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:08.091-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:08.094-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:08.105-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:08.106-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19110000 2016-04-07T01:50:08.118-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19120000 2016-04-07T01:50:08.268-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19130000 2016-04-07T01:50:08.428-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12340000 QueuedCount 50000 2016-04-07T01:50:08.723-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:08.724-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19140000 2016-04-07T01:50:08.886-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12350000 QueuedCount 50000 2016-04-07T01:50:08.904-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:08.912-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19150000 2016-04-07T01:50:09.168-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:09.321-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12360000 QueuedCount 48632 2016-04-07T01:50:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:50:09.375-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T01:50:09.375-07:00 [Info] client load stats {} 2016-04-07T01:50:09.639-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":6739130,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":5809625283,"bucket-1:myindex:avg_ts_items_count":101004,"bucket-1:myindex:build_progress":61,"bucket-1:myindex:data_size":3260446897,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":9040855040,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":40580,"bucket-1:myindex:frag_percent":63,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":1291109190,"bucket-1:myindex:items_count":12236806,"bucket-1:myindex:num_commits":84,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":12368052,"bucket-1:myindex:num_docs_pending":898548,"bucket-1:myindex:num_docs_processed":12325160,"bucket-1:myindex:num_docs_queued":6776292,"bucket-1:myindex:num_flush_queued":12408632,"bucket-1:myindex:num_items_flushed":12368052,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":84,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":5733477367,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"84 44270185674 -8253279072439149982","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"12368053 73528864001 380169636468131563","bucket-1:myindex:timings/storage_info":"337 2792139 41648294333","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"83 28056852 10607614136770","bucket-1:myindex:timings/storage_meta_set":"84 28612686 10606676268054","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"84 3622399 169971526111","bucket-1:myindex:timings/storage_set":"24736105 440523527683 5704330867883837683","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":19147762,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"118 158106510 2092623772998964","bucket-1:ts_queue_size":820,"indexer_state":"Active","memory_quota":31457280000,"memory_used":15332239360,"memory_used_storage":9041264640,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"2193 338953290 149138638348826","uptime":"21m1.33235072s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T01:50:09.674-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12370000 QueuedCount 38632 2016-04-07T01:50:09.676-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:3280727582, Disk:9040855040, ExtraSnapshotData:0, Fragmentation:63.712197933880375%, mode : circular 2016-04-07T01:50:09.676-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:50:09.677-07:00 [Info] memstats {"Alloc":3988620064, "TotalAlloc":16112721768, "Sys":9854296392, "Lookups":4441, "Mallocs":434284538,"Frees":343550308, "HeapAlloc":3988620064, "HeapSys":9384427520, "HeapIdle":3390898176, "HeapInuse":5993529344,"HeapReleased":1644347392, "HeapObjects":90734230,"GCSys":297445376, "LastGC":1460018884062254022,"PauseTotalNs":25888237, "PauseNs":[], "NumGC":20} 2016-04-07T01:50:10.081-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12380000 QueuedCount 28632 2016-04-07T01:50:10.503-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12390000 QueuedCount 18632 2016-04-07T01:50:10.922-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12400000 QueuedCount 8632 2016-04-07T01:50:11.429-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:50:11.442-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:50:11.442-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:50:11.864-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 43.17µs CommitTime 565.718804ms TotalFlushTime 8m42.459548176s TotalCommitTime 44.835904478s 2016-04-07T01:50:11.864-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 12408632, 12408632, 85 committed:true 2016-04-07T01:50:11.864-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 12408632, 12408632, 85 committed:true) SnapCreateDur 574.099695ms SnapOpenDur 43.794µs 2016-04-07T01:50:11.875-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:11.879-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:11.884-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:11.885-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:11.889-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:11.892-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19160000 2016-04-07T01:50:11.893-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:11.895-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:11.896-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12410000 QueuedCount 44914 2016-04-07T01:50:11.899-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:11.902-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19170000 2016-04-07T01:50:11.914-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19180000 2016-04-07T01:50:11.926-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19190000 2016-04-07T01:50:11.938-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19200000 2016-04-07T01:50:11.95-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19210000 2016-04-07T01:50:11.962-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19220000 2016-04-07T01:50:11.973-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19230000 2016-04-07T01:50:11.984-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19240000 2016-04-07T01:50:11.995-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19250000 2016-04-07T01:50:12.006-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19260000 2016-04-07T01:50:12.018-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19270000 2016-04-07T01:50:12.029-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19280000 2016-04-07T01:50:12.041-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19290000 2016-04-07T01:50:12.055-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19300000 2016-04-07T01:50:12.065-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19310000 2016-04-07T01:50:12.077-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19320000 2016-04-07T01:50:12.088-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19330000 2016-04-07T01:50:12.098-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19340000 2016-04-07T01:50:12.109-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19350000 2016-04-07T01:50:12.121-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19360000 2016-04-07T01:50:12.132-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19370000 2016-04-07T01:50:12.143-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19380000 2016-04-07T01:50:12.153-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19390000 2016-04-07T01:50:12.165-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19400000 2016-04-07T01:50:12.178-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19410000 2016-04-07T01:50:12.19-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19420000 2016-04-07T01:50:12.202-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19430000 2016-04-07T01:50:12.214-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19440000 2016-04-07T01:50:12.224-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19450000 2016-04-07T01:50:12.234-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19460000 2016-04-07T01:50:12.238-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:12.244-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19470000 2016-04-07T01:50:12.254-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19480000 2016-04-07T01:50:12.264-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19490000 2016-04-07T01:50:12.273-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19500000 2016-04-07T01:50:12.283-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19510000 2016-04-07T01:50:12.292-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19520000 2016-04-07T01:50:12.302-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19530000 2016-04-07T01:50:12.311-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19540000 2016-04-07T01:50:12.322-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19550000 2016-04-07T01:50:12.332-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12420000 QueuedCount 50000 2016-04-07T01:50:12.332-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19560000 2016-04-07T01:50:12.343-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19570000 2016-04-07T01:50:12.355-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19580000 2016-04-07T01:50:12.367-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19590000 2016-04-07T01:50:12.38-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19600000 2016-04-07T01:50:12.393-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19610000 2016-04-07T01:50:12.406-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19620000 2016-04-07T01:50:12.418-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19630000 2016-04-07T01:50:12.429-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19640000 2016-04-07T01:50:12.44-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19650000 2016-04-07T01:50:12.593-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:12.62-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19660000 2016-04-07T01:50:12.77-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:12.775-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12430000 QueuedCount 50000 2016-04-07T01:50:13.239-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12440000 QueuedCount 50000 2016-04-07T01:50:13.254-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:13.556-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19670000 2016-04-07T01:50:13.719-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12450000 QueuedCount 50000 2016-04-07T01:50:13.954-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19680000 2016-04-07T01:50:14.214-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12460000 QueuedCount 50000 2016-04-07T01:50:14.35-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:14.351-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19690000 2016-04-07T01:50:14.64-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12470000 QueuedCount 44845 2016-04-07T01:50:15.1-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12480000 QueuedCount 34845 2016-04-07T01:50:15.562-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12490000 QueuedCount 24845 2016-04-07T01:50:16.009-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12500000 QueuedCount 14845 2016-04-07T01:50:16.477-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12510000 QueuedCount 4845 2016-04-07T01:50:17.355-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 15.443µs CommitTime 662.478816ms TotalFlushTime 8m47.185480851s TotalCommitTime 45.498383294s 2016-04-07T01:50:17.355-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 12514845, 12514845, 86 committed:true 2016-04-07T01:50:17.355-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 12514845, 12514845, 86 committed:true) SnapCreateDur 671.013626ms SnapOpenDur 56.754µs 2016-04-07T01:50:17.357-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:17.358-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:17.36-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:17.363-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:17.364-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:17.365-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:17.368-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:17.37-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:17.372-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:17.374-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:17.375-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:17.378-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:17.387-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:17.389-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:17.391-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:17.396-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19700000 2016-04-07T01:50:17.407-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19710000 2016-04-07T01:50:17.417-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19720000 2016-04-07T01:50:17.426-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19730000 2016-04-07T01:50:17.436-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19740000 2016-04-07T01:50:17.446-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19750000 2016-04-07T01:50:17.457-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19760000 2016-04-07T01:50:17.468-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19770000 2016-04-07T01:50:17.479-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19780000 2016-04-07T01:50:17.484-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:17.495-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19790000 2016-04-07T01:50:17.514-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19800000 2016-04-07T01:50:17.53-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19810000 2016-04-07T01:50:17.54-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19820000 2016-04-07T01:50:17.549-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19830000 2016-04-07T01:50:17.559-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19840000 2016-04-07T01:50:17.569-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19850000 2016-04-07T01:50:17.576-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12520000 QueuedCount 50000 2016-04-07T01:50:17.581-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19860000 2016-04-07T01:50:17.592-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19870000 2016-04-07T01:50:17.602-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19880000 2016-04-07T01:50:17.612-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19890000 2016-04-07T01:50:17.75-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:17.853-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:18.013-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12530000 QueuedCount 50000 2016-04-07T01:50:18.1-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:18.197-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:18.452-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:18.456-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12540000 QueuedCount 50000 2016-04-07T01:50:18.564-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:18.742-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:18.931-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:18.935-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12550000 QueuedCount 50000 2016-04-07T01:50:19.035-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:19.242-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:19.437-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:19.437-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12560000 QueuedCount 50000 2016-04-07T01:50:19.858-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12570000 QueuedCount 45129 2016-04-07T01:50:20.305-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12580000 QueuedCount 35129 2016-04-07T01:50:20.788-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12590000 QueuedCount 25129 2016-04-07T01:50:20.815-07:00 [Info] Indexer::ReadMemstats Time Taken 15.271646ms 2016-04-07T01:50:21.251-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12600000 QueuedCount 15129 2016-04-07T01:50:21.466-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:50:21.479-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:50:21.479-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:50:21.75-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12610000 QueuedCount 5129 2016-04-07T01:50:22.617-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5010 Milliseconds Vbucket 72 2016-04-07T01:50:22.647-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5040 Milliseconds Vbucket 72 2016-04-07T01:50:22.677-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5070 Milliseconds Vbucket 72 2016-04-07T01:50:22.707-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5100 Milliseconds Vbucket 72 2016-04-07T01:50:22.713-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.17µs CommitTime 730.185429ms TotalFlushTime 8m51.711887353s TotalCommitTime 46.228568723s 2016-04-07T01:50:22.713-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 12615129, 12615129, 87 committed:true 2016-04-07T01:50:22.713-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 12615129, 12615129, 87 committed:true) SnapCreateDur 738.656073ms SnapOpenDur 39.039µs 2016-04-07T01:50:22.717-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:22.737-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5130 Milliseconds Vbucket 72 2016-04-07T01:50:22.738-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:22.74-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:22.9-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12620000 QueuedCount 50000 2016-04-07T01:50:23.191-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19900000 2016-04-07T01:50:23.346-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12630000 QueuedCount 50000 2016-04-07T01:50:23.803-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12640000 QueuedCount 50000 2016-04-07T01:50:24.274-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12650000 QueuedCount 50000 2016-04-07T01:50:24.678-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12660000 QueuedCount 50000 2016-04-07T01:50:25.167-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12670000 QueuedCount 50000 2016-04-07T01:50:25.679-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12680000 QueuedCount 50000 2016-04-07T01:50:26.221-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12690000 QueuedCount 50000 2016-04-07T01:50:26.762-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12700000 QueuedCount 50000 2016-04-07T01:50:27.407-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12710000 QueuedCount 50000 2016-04-07T01:50:27.945-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12720000 QueuedCount 50000 2016-04-07T01:50:28.491-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12730000 QueuedCount 50000 2016-04-07T01:50:28.654-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5010 Milliseconds Vbucket 72 2016-04-07T01:50:28.684-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5040 Milliseconds Vbucket 72 2016-04-07T01:50:28.714-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5070 Milliseconds Vbucket 72 2016-04-07T01:50:28.744-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5100 Milliseconds Vbucket 72 2016-04-07T01:50:28.774-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5130 Milliseconds Vbucket 72 2016-04-07T01:50:28.804-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5160 Milliseconds Vbucket 72 2016-04-07T01:50:28.834-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5190 Milliseconds Vbucket 72 2016-04-07T01:50:28.864-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5220 Milliseconds Vbucket 72 2016-04-07T01:50:28.894-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5250 Milliseconds Vbucket 72 2016-04-07T01:50:28.924-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5280 Milliseconds Vbucket 72 2016-04-07T01:50:28.954-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5310 Milliseconds Vbucket 72 2016-04-07T01:50:28.961-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12740000 QueuedCount 50000 2016-04-07T01:50:28.984-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5340 Milliseconds Vbucket 72 2016-04-07T01:50:29.014-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5370 Milliseconds Vbucket 72 2016-04-07T01:50:29.044-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5400 Milliseconds Vbucket 72 2016-04-07T01:50:29.074-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5430 Milliseconds Vbucket 72 2016-04-07T01:50:29.104-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5460 Milliseconds Vbucket 72 2016-04-07T01:50:29.134-07:00 [Warn] Indexer::MutationQueue Waiting for Node Alloc for 5490 Milliseconds Vbucket 72 2016-04-07T01:50:29.15-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:29.488-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12750000 QueuedCount 47116 2016-04-07T01:50:29.992-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12760000 QueuedCount 37116 2016-04-07T01:50:30.474-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12770000 QueuedCount 27116 2016-04-07T01:50:30.957-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12780000 QueuedCount 17116 2016-04-07T01:50:31.442-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12790000 QueuedCount 7116 2016-04-07T01:50:32.524-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 24.284µs CommitTime 746.244957ms TotalFlushTime 9m0.588018517s TotalCommitTime 46.97481368s 2016-04-07T01:50:32.524-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 12797116, 12797116, 88 committed:true 2016-04-07T01:50:32.524-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 12797116, 12797116, 88 committed:true) SnapCreateDur 755.208008ms SnapOpenDur 69.569µs 2016-04-07T01:50:32.528-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:32.531-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:32.534-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:32.538-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:32.542-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:32.544-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:32.545-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:32.55-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:32.554-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:32.56-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19910000 2016-04-07T01:50:32.567-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:32.644-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12800000 QueuedCount 50000 2016-04-07T01:50:32.908-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:33.061-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12810000 QueuedCount 50000 2016-04-07T01:50:33.337-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:33.517-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12820000 QueuedCount 50000 2016-04-07T01:50:33.535-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:33.816-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:33.997-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12830000 QueuedCount 50000 2016-04-07T01:50:34.193-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:34.403-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12840000 QueuedCount 50000 2016-04-07T01:50:34.84-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12850000 QueuedCount 40601 2016-04-07T01:50:35.275-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12860000 QueuedCount 30601 2016-04-07T01:50:35.722-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12870000 QueuedCount 20601 2016-04-07T01:50:36.178-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12880000 QueuedCount 10601 2016-04-07T01:50:36.523-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:50:36.537-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:50:36.537-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:50:36.637-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12890000 QueuedCount 601 2016-04-07T01:50:37.265-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.45µs CommitTime 606.491655ms TotalFlushTime 9m4.634178911s TotalCommitTime 47.581305335s 2016-04-07T01:50:37.265-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 12890601, 12890601, 89 committed:true 2016-04-07T01:50:37.265-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 12890601, 12890601, 89 committed:true) SnapCreateDur 615.365815ms SnapOpenDur 48.248µs 2016-04-07T01:50:37.285-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:37.29-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:37.653-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12900000 QueuedCount 50000 2016-04-07T01:50:38.124-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12910000 QueuedCount 50000 2016-04-07T01:50:38.602-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12920000 QueuedCount 50000 2016-04-07T01:50:38.717-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:38.836-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19920000 2016-04-07T01:50:39.093-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12930000 QueuedCount 50000 2016-04-07T01:50:39.301-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:39.593-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12940000 QueuedCount 50000 2016-04-07T01:50:39.678-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:3445499606, Disk:9040855040, ExtraSnapshotData:0, Fragmentation:61.88967093537206%, mode : circular 2016-04-07T01:50:39.678-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:50:40.018-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12950000 QueuedCount 41447 2016-04-07T01:50:40.485-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12960000 QueuedCount 31447 2016-04-07T01:50:40.941-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12970000 QueuedCount 21447 2016-04-07T01:50:41.394-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12980000 QueuedCount 11447 2016-04-07T01:50:41.863-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 12990000 QueuedCount 1447 2016-04-07T01:50:42.539-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 23.451µs CommitTime 568.478548ms TotalFlushTime 9m9.236700871s TotalCommitTime 48.149783883s 2016-04-07T01:50:42.539-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 12991447, 12991447, 90 committed:true 2016-04-07T01:50:42.539-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 12991447, 12991447, 90 committed:true) SnapCreateDur 576.757112ms SnapOpenDur 61.543µs 2016-04-07T01:50:42.544-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:42.547-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:42.553-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:42.561-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:42.564-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:42.569-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19930000 2016-04-07T01:50:42.582-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19940000 2016-04-07T01:50:42.594-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19950000 2016-04-07T01:50:42.606-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19960000 2016-04-07T01:50:42.616-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19970000 2016-04-07T01:50:42.626-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19980000 2016-04-07T01:50:42.635-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 19990000 2016-04-07T01:50:42.645-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20000000 2016-04-07T01:50:42.662-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:42.936-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13000000 QueuedCount 50000 2016-04-07T01:50:43.309-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:43.396-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13010000 QueuedCount 50000 2016-04-07T01:50:43.591-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:43.786-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:43.791-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13020000 QueuedCount 50000 2016-04-07T01:50:44.046-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:44.268-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13030000 QueuedCount 50000 2016-04-07T01:50:44.772-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13040000 QueuedCount 50000 2016-04-07T01:50:45.286-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13050000 QueuedCount 50000 2016-04-07T01:50:45.823-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13060000 QueuedCount 50000 2016-04-07T01:50:46.349-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13070000 QueuedCount 50000 2016-04-07T01:50:46.561-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:50:46.575-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:50:46.575-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:50:46.659-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:46.879-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13080000 QueuedCount 50000 2016-04-07T01:50:47.39-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13090000 QueuedCount 40032 2016-04-07T01:50:47.813-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13100000 QueuedCount 30032 2016-04-07T01:50:48.313-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13110000 QueuedCount 20032 2016-04-07T01:50:48.806-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13120000 QueuedCount 10032 2016-04-07T01:50:49.315-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13130000 QueuedCount 32 2016-04-07T01:50:50.156-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 31.4µs CommitTime 827.872832ms TotalFlushTime 9m15.892202449s TotalCommitTime 48.977656715s 2016-04-07T01:50:50.156-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 13130032, 13130032, 91 committed:true 2016-04-07T01:50:50.156-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 13130032, 13130032, 91 committed:true) SnapCreateDur 836.029684ms SnapOpenDur 51.235µs 2016-04-07T01:50:50.16-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:50.162-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:50.165-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:50.169-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:50.174-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:50.176-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:50.18-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:50.183-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:50.186-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:50.187-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:50.272-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:50.362-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:50.548-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:50.557-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13140000 QueuedCount 50000 2016-04-07T01:50:50.802-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:51.001-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13150000 QueuedCount 50000 2016-04-07T01:50:51.173-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:51.182-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:51.288-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:51.448-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:51.462-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13160000 QueuedCount 50000 2016-04-07T01:50:51.556-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:51.657-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:51.783-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:51.989-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13170000 QueuedCount 48765 2016-04-07T01:50:52.471-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13180000 QueuedCount 38765 2016-04-07T01:50:52.939-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13190000 QueuedCount 28765 2016-04-07T01:50:53.432-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13200000 QueuedCount 18765 2016-04-07T01:50:53.93-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13210000 QueuedCount 8765 2016-04-07T01:50:55.051-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.209µs CommitTime 680.488298ms TotalFlushTime 9m20.027025457s TotalCommitTime 49.658145013s 2016-04-07T01:50:55.051-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 13218765, 13218765, 92 committed:true 2016-04-07T01:50:55.051-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 13218765, 13218765, 92 committed:true) SnapCreateDur 689.110656ms SnapOpenDur 61.791µs 2016-04-07T01:50:55.053-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:55.054-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:55.055-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:55.057-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:55.059-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:55.061-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:55.064-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:55.067-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:55.069-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:55.07-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:55.072-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:55.074-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:55.076-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:55.078-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:55.08-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:55.081-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13220000 QueuedCount 49729 2016-04-07T01:50:55.176-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:55.38-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:55.586-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13230000 QueuedCount 50000 2016-04-07T01:50:55.676-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:55.973-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:56.082-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13240000 QueuedCount 50000 2016-04-07T01:50:56.191-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:56.408-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:56.597-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:50:56.61-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13250000 QueuedCount 50000 2016-04-07T01:50:56.61-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:50:56.61-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:50:56.612-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:56.934-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:50:57.119-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13260000 QueuedCount 49439 2016-04-07T01:50:57.591-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13270000 QueuedCount 39439 2016-04-07T01:50:57.978-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13280000 QueuedCount 29439 2016-04-07T01:50:58.463-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13290000 QueuedCount 19439 2016-04-07T01:50:58.936-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13300000 QueuedCount 9439 2016-04-07T01:51:00.05-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 23.09µs CommitTime 638.492349ms TotalFlushTime 9m24.303899469s TotalCommitTime 50.296637362s 2016-04-07T01:51:00.05-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 13309439, 13309439, 93 committed:true 2016-04-07T01:51:00.05-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 13309439, 13309439, 93 committed:true) SnapCreateDur 647.680526ms SnapOpenDur 71.967µs 2016-04-07T01:51:00.052-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:00.053-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:00.054-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:00.056-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:00.059-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:00.06-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:00.062-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13310000 QueuedCount 19873 2016-04-07T01:51:00.063-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:00.065-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:00.066-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:00.068-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:00.07-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:00.073-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:00.075-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:00.078-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:00.28-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:00.585-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13320000 QueuedCount 50000 2016-04-07T01:51:00.606-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:00.703-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:00.905-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:01.086-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:01.086-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13330000 QueuedCount 49986 2016-04-07T01:51:01.2-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:01.497-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:01.509-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13340000 QueuedCount 50000 2016-04-07T01:51:01.708-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:02.029-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13350000 QueuedCount 50000 2016-04-07T01:51:02.117-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:02.54-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13360000 QueuedCount 46866 2016-04-07T01:51:03.014-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13370000 QueuedCount 36866 2016-04-07T01:51:03.494-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13380000 QueuedCount 26866 2016-04-07T01:51:03.985-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13390000 QueuedCount 16866 2016-04-07T01:51:04.472-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13400000 QueuedCount 6866 2016-04-07T01:51:05.445-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.325µs CommitTime 657.764602ms TotalFlushTime 9m28.955443018s TotalCommitTime 50.954401964s 2016-04-07T01:51:05.445-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 13406866, 13406866, 94 committed:true 2016-04-07T01:51:05.445-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 13406866, 13406866, 94 committed:true) SnapCreateDur 666.41934ms SnapOpenDur 77.129µs 2016-04-07T01:51:05.448-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:05.451-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:05.454-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:05.458-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:05.463-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:05.579-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13410000 QueuedCount 50000 2016-04-07T01:51:06.074-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13420000 QueuedCount 50000 2016-04-07T01:51:06.564-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13430000 QueuedCount 50000 2016-04-07T01:51:06.634-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:51:06.647-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:51:06.647-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:51:07.085-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13440000 QueuedCount 50000 2016-04-07T01:51:07.606-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13450000 QueuedCount 50000 2016-04-07T01:51:07.936-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:08.049-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13460000 QueuedCount 50000 2016-04-07T01:51:08.543-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13470000 QueuedCount 41283 2016-04-07T01:51:09.028-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13480000 QueuedCount 31283 2016-04-07T01:51:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:51:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T01:51:09.374-07:00 [Info] client load stats {} 2016-04-07T01:51:09.521-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13490000 QueuedCount 21283 2016-04-07T01:51:09.64-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":6488717,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":5391515813,"bucket-1:myindex:avg_ts_items_count":97942,"bucket-1:myindex:build_progress":67,"bucket-1:myindex:data_size":3575661103,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":9768939520,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":18381,"bucket-1:myindex:frag_percent":63,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":1409226525,"bucket-1:myindex:items_count":13406866,"bucket-1:myindex:num_commits":94,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":13492902,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":13442853,"bucket-1:myindex:num_docs_queued":6557147,"bucket-1:myindex:num_flush_queued":13511283,"bucket-1:myindex:num_items_flushed":13492902,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":94,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":5394485413,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"94 50954401964 -3724546340844131258","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"13492903 80043955981 380211838674882103","bucket-1:myindex:timings/storage_info":"377 3168008 48404180560","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"93 31469540 11859099567132","bucket-1:myindex:timings/storage_meta_set":"94 31889668 11790858010368","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"94 4026979 187962118063","bucket-1:myindex:timings/storage_set":"26985805 485696353992 8556292610264587688","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":20000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"130 168303985 2101311194490947","bucket-1:ts_queue_size":692,"indexer_state":"Active","memory_quota":31457280000,"memory_used":14247008256,"memory_used_storage":9867317248,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"2336 367293191 154967565967521","uptime":"22m1.333268878s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T01:51:09.677-07:00 [Info] memstats {"Alloc":2908364400, "TotalAlloc":17072714592, "Sys":9854296392, "Lookups":4874, "Mallocs":461338256,"Frees":394006215, "HeapAlloc":2908364400, "HeapSys":9384853504, "HeapIdle":5302607872, "HeapInuse":4082245632,"HeapReleased":1643356160, "HeapObjects":67332041,"GCSys":297445376, "LastGC":1460019004549771477,"PauseTotalNs":27118358, "PauseNs":[1230121], "NumGC":21} 2016-04-07T01:51:09.678-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:3595554590, Disk:9768939520, ExtraSnapshotData:0, Fragmentation:63.19401320236651%, mode : circular 2016-04-07T01:51:09.678-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:51:10.003-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13500000 QueuedCount 11283 2016-04-07T01:51:10.492-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13510000 QueuedCount 1283 2016-04-07T01:51:11.196-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 16.505µs CommitTime 678.740996ms TotalFlushTime 9m33.938278027s TotalCommitTime 51.63314296s 2016-04-07T01:51:11.196-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 13511283, 13511283, 95 committed:true 2016-04-07T01:51:11.196-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 13511283, 13511283, 95 committed:true) SnapCreateDur 687.787929ms SnapOpenDur 85.045µs 2016-04-07T01:51:11.198-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:11.2-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:11.201-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:11.203-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:11.204-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:11.205-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:11.206-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:11.208-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:11.209-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:11.211-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:11.212-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:11.214-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:11.217-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:11.219-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:11.22-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:11.653-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13520000 QueuedCount 50000 2016-04-07T01:51:11.783-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:12.08-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:12.184-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13530000 QueuedCount 50000 2016-04-07T01:51:12.204-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:12.408-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:12.701-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13540000 QueuedCount 50000 2016-04-07T01:51:12.703-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:12.819-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:13.149-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13550000 QueuedCount 50000 2016-04-07T01:51:13.26-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:13.67-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13560000 QueuedCount 47010 2016-04-07T01:51:14.153-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13570000 QueuedCount 37010 2016-04-07T01:51:14.624-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13580000 QueuedCount 27010 2016-04-07T01:51:15.09-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13590000 QueuedCount 17010 2016-04-07T01:51:15.561-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13600000 QueuedCount 7010 2016-04-07T01:51:16.562-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 37.717µs CommitTime 663.527786ms TotalFlushTime 9m38.554079909s TotalCommitTime 52.296670746s 2016-04-07T01:51:16.562-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 13607010, 13607010, 96 committed:true 2016-04-07T01:51:16.562-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 13607010, 13607010, 96 committed:true) SnapCreateDur 672.705712ms SnapOpenDur 85.219µs 2016-04-07T01:51:16.564-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:16.567-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:16.569-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:16.575-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:16.579-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:16.584-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:16.67-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:51:16.684-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:51:16.684-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:51:16.702-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13610000 QueuedCount 50000 2016-04-07T01:51:17.209-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13620000 QueuedCount 50000 2016-04-07T01:51:17.711-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13630000 QueuedCount 50000 2016-04-07T01:51:18.236-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13640000 QueuedCount 50000 2016-04-07T01:51:18.668-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:18.681-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13650000 QueuedCount 50000 2016-04-07T01:51:19.193-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13660000 QueuedCount 44932 2016-04-07T01:51:19.678-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13670000 QueuedCount 34932 2016-04-07T01:51:20.17-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13680000 QueuedCount 24932 2016-04-07T01:51:20.654-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13690000 QueuedCount 14932 2016-04-07T01:51:21.143-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13700000 QueuedCount 4932 2016-04-07T01:51:21.201-07:00 [Info] Indexer::ReadMemstats Time Taken 13.388488ms 2016-04-07T01:51:22.075-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.103µs CommitTime 708.758881ms TotalFlushTime 9m43.269002226s TotalCommitTime 53.005429627s 2016-04-07T01:51:22.075-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 13704932, 13704932, 97 committed:true 2016-04-07T01:51:22.075-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 13704932, 13704932, 97 committed:true) SnapCreateDur 717.517759ms SnapOpenDur 64.615µs 2016-04-07T01:51:22.078-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:22.079-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:22.081-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:22.083-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:22.085-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:22.087-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:22.091-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:22.093-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:22.096-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:22.099-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:22.103-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:22.105-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:22.109-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:22.308-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:22.317-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13710000 QueuedCount 50000 2016-04-07T01:51:22.53-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:22.648-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:22.849-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13720000 QueuedCount 50000 2016-04-07T01:51:23.37-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13730000 QueuedCount 50000 2016-04-07T01:51:23.895-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13740000 QueuedCount 50000 2016-04-07T01:51:24.232-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:24.434-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13750000 QueuedCount 50000 2016-04-07T01:51:24.889-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13760000 QueuedCount 48736 2016-04-07T01:51:25.38-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13770000 QueuedCount 38736 2016-04-07T01:51:25.877-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13780000 QueuedCount 28736 2016-04-07T01:51:26.373-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13790000 QueuedCount 18736 2016-04-07T01:51:26.896-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13800000 QueuedCount 8736 2016-04-07T01:51:29.676-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 30.61µs CommitTime 2.324832121s TotalFlushTime 9m48.443191545s TotalCommitTime 55.330261748s 2016-04-07T01:51:29.678-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 13808736, 13808736, 98 committed:true 2016-04-07T01:51:29.678-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 13808736, 13808736, 98 committed:true) SnapCreateDur 2.334204506s SnapOpenDur 66.799µs 2016-04-07T01:51:29.696-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13810000 QueuedCount 5127 2016-04-07T01:51:30.049-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.712µs CommitTime 107.35167ms TotalFlushTime 9m48.689462054s TotalCommitTime 55.437613418s 2016-04-07T01:51:30.049-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 13815127, 13815127, 99 committed:true 2016-04-07T01:51:30.049-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 13815127, 13815127, 99 committed:true) SnapCreateDur 116.222694ms SnapOpenDur 53.614µs 2016-04-07T01:51:30.054-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:30.058-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:30.062-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:30.065-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:30.07-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:30.072-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:30.073-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:30.175-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:30.273-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13820000 QueuedCount 50000 2016-04-07T01:51:30.367-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:30.749-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13830000 QueuedCount 50000 2016-04-07T01:51:31.243-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13840000 QueuedCount 50000 2016-04-07T01:51:31.731-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:51:31.745-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:51:31.745-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:51:31.752-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13850000 QueuedCount 50000 2016-04-07T01:51:32.202-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13860000 QueuedCount 50000 2016-04-07T01:51:32.287-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:32.727-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13870000 QueuedCount 50000 2016-04-07T01:51:32.826-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:33.262-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13880000 QueuedCount 50000 2016-04-07T01:51:33.472-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:33.798-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13890000 QueuedCount 50000 2016-04-07T01:51:33.81-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:34.127-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:34.358-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13900000 QueuedCount 50000 2016-04-07T01:51:34.568-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:34.795-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:34.905-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13910000 QueuedCount 50000 2016-04-07T01:51:35.422-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13920000 QueuedCount 42781 2016-04-07T01:51:35.941-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13930000 QueuedCount 32781 2016-04-07T01:51:36.4-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13940000 QueuedCount 22781 2016-04-07T01:51:36.926-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13950000 QueuedCount 12781 2016-04-07T01:51:37.453-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13960000 QueuedCount 2781 2016-04-07T01:51:38.375-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.721µs CommitTime 718.712996ms TotalFlushTime 9m56.15491778s TotalCommitTime 56.156326414s 2016-04-07T01:51:38.375-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 13962781, 13962781, 100 committed:true 2016-04-07T01:51:38.375-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 13962781, 13962781, 100 committed:true) SnapCreateDur 727.651575ms SnapOpenDur 50.975µs 2016-04-07T01:51:38.431-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:38.433-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:38.437-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:38.439-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:38.442-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:38.442-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:38.444-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:38.445-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:38.446-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:38.447-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:38.449-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:38.451-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:38.452-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:38.454-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:38.454-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:38.456-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:38.458-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:38.629-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:38.74-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13970000 QueuedCount 50000 2016-04-07T01:51:38.751-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:38.926-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:39.039-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:39.145-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:39.23-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13980000 QueuedCount 50000 2016-04-07T01:51:39.317-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:39.415-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:39.518-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:39.679-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:3732726837, Disk:11049111552, ExtraSnapshotData:0, Fragmentation:66.21695039069147%, mode : circular 2016-04-07T01:51:39.679-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:51:39.697-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:39.703-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 13990000 QueuedCount 50000 2016-04-07T01:51:39.812-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:40.201-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14000000 QueuedCount 45136 2016-04-07T01:51:40.666-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14010000 QueuedCount 35136 2016-04-07T01:51:41.06-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14020000 QueuedCount 25136 2016-04-07T01:51:41.541-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14030000 QueuedCount 15136 2016-04-07T01:51:41.77-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:51:41.783-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:51:41.783-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:51:42.054-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14040000 QueuedCount 5136 2016-04-07T01:51:43.057-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 23.171µs CommitTime 707.963554ms TotalFlushTime 10m0.007118125s TotalCommitTime 56.864289968s 2016-04-07T01:51:43.057-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 14045136, 14045136, 101 committed:true 2016-04-07T01:51:43.057-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 14045136, 14045136, 101 committed:true) SnapCreateDur 716.769108ms SnapOpenDur 45.943µs 2016-04-07T01:51:43.06-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:43.062-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:43.063-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:43.064-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:43.066-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:43.068-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:43.07-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:43.071-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:43.073-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:43.074-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:43.075-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:43.077-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:43.079-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:43.085-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:43.091-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:43.283-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14050000 QueuedCount 50000 2016-04-07T01:51:43.38-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:43.762-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14060000 QueuedCount 50000 2016-04-07T01:51:44.264-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14070000 QueuedCount 50000 2016-04-07T01:51:44.739-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14080000 QueuedCount 50000 2016-04-07T01:51:45.183-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14090000 QueuedCount 50000 2016-04-07T01:51:45.488-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:45.717-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14100000 QueuedCount 50000 2016-04-07T01:51:46.234-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14110000 QueuedCount 41580 2016-04-07T01:51:46.736-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14120000 QueuedCount 31580 2016-04-07T01:51:47.243-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14130000 QueuedCount 21580 2016-04-07T01:51:47.747-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14140000 QueuedCount 11580 2016-04-07T01:51:48.269-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14150000 QueuedCount 1580 2016-04-07T01:51:49.073-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 22.852µs CommitTime 773.683628ms TotalFlushTime 10m5.152863262s TotalCommitTime 57.637973596s 2016-04-07T01:51:49.073-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 14151580, 14151580, 102 committed:true 2016-04-07T01:51:49.073-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 14151580, 14151580, 102 committed:true) SnapCreateDur 782.699586ms SnapOpenDur 44.056µs 2016-04-07T01:51:49.077-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:49.079-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:49.083-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:49.085-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:49.088-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:49.091-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:49.093-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:49.094-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:49.097-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:49.099-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:49.299-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:49.473-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14160000 QueuedCount 50000 2016-04-07T01:51:49.578-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:49.889-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:49.91-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14170000 QueuedCount 50000 2016-04-07T01:51:49.99-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:50.288-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:50.42-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14180000 QueuedCount 50000 2016-04-07T01:51:50.508-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:50.908-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14190000 QueuedCount 45884 2016-04-07T01:51:51.367-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14200000 QueuedCount 35884 2016-04-07T01:51:51.804-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:51:51.817-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:51:51.817-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:51:51.831-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14210000 QueuedCount 25884 2016-04-07T01:51:52.293-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14220000 QueuedCount 15884 2016-04-07T01:51:52.763-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14230000 QueuedCount 5884 2016-04-07T01:51:53.632-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.854µs CommitTime 578.473895ms TotalFlushTime 10m9.065301293s TotalCommitTime 58.216447491s 2016-04-07T01:51:53.632-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 14235884, 14235884, 103 committed:true 2016-04-07T01:51:53.632-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 14235884, 14235884, 103 committed:true) SnapCreateDur 586.520114ms SnapOpenDur 59.51µs 2016-04-07T01:51:53.635-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:53.638-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:53.659-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:53.842-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14240000 QueuedCount 49998 2016-04-07T01:51:53.842-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:53.95-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:54.13-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:54.229-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:54.242-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14250000 QueuedCount 50000 2016-04-07T01:51:54.434-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:54.533-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:54.72-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:54.734-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14260000 QueuedCount 50000 2016-04-07T01:51:54.936-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:55.15-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:55.252-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14270000 QueuedCount 50000 2016-04-07T01:51:55.352-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:55.567-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:55.771-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14280000 QueuedCount 50000 2016-04-07T01:51:56.276-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14290000 QueuedCount 40894 2016-04-07T01:51:56.768-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14300000 QueuedCount 30894 2016-04-07T01:51:57.266-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14310000 QueuedCount 20894 2016-04-07T01:51:57.765-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14320000 QueuedCount 10894 2016-04-07T01:51:58.184-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14330000 QueuedCount 894 2016-04-07T01:51:58.909-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 31.096µs CommitTime 625.710347ms TotalFlushTime 10m13.631590942s TotalCommitTime 58.842157838s 2016-04-07T01:51:58.909-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 14330894, 14330894, 104 committed:true 2016-04-07T01:51:58.909-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 14330894, 14330894, 104 committed:true) SnapCreateDur 634.059831ms SnapOpenDur 68.867µs 2016-04-07T01:51:58.911-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:58.913-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:58.916-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:58.919-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:58.921-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:58.923-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:58.928-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:58.93-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:58.934-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:59.029-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:59.292-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14340000 QueuedCount 50000 2016-04-07T01:51:59.579-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:51:59.768-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14350000 QueuedCount 50000 2016-04-07T01:52:00.285-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14360000 QueuedCount 50000 2016-04-07T01:52:00.801-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14370000 QueuedCount 50000 2016-04-07T01:52:00.83-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:01.214-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14380000 QueuedCount 42959 2016-04-07T01:52:01.707-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14390000 QueuedCount 32959 2016-04-07T01:52:01.841-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:52:01.853-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:52:01.853-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:52:02.204-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14400000 QueuedCount 22959 2016-04-07T01:52:02.696-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14410000 QueuedCount 12959 2016-04-07T01:52:03.219-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14420000 QueuedCount 2959 2016-04-07T01:52:04.141-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 38.029µs CommitTime 790.773236ms TotalFlushTime 10m17.992230072s TotalCommitTime 59.632931074s 2016-04-07T01:52:04.141-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 14422959, 14422959, 105 committed:true 2016-04-07T01:52:04.141-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 14422959, 14422959, 105 committed:true) SnapCreateDur 799.493026ms SnapOpenDur 77.185µs 2016-04-07T01:52:04.157-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:04.158-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:04.16-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:04.161-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:04.164-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:04.166-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:04.254-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:04.427-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14430000 QueuedCount 50000 2016-04-07T01:52:04.518-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:04.776-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:05.296-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14440000 QueuedCount 50000 2016-04-07T01:52:05.313-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:05.562-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:05.748-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:05.75-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14450000 QueuedCount 50000 2016-04-07T01:52:05.853-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:06.201-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14460000 QueuedCount 47395 2016-04-07T01:52:06.564-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14470000 QueuedCount 37395 2016-04-07T01:52:07.033-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14480000 QueuedCount 27395 2016-04-07T01:52:07.495-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14490000 QueuedCount 17395 2016-04-07T01:52:07.95-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14500000 QueuedCount 7395 2016-04-07T01:52:08.894-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 34.843µs CommitTime 566.640003ms TotalFlushTime 10m22.096824648s TotalCommitTime 1m0.199571077s 2016-04-07T01:52:08.894-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 14507395, 14507395, 106 committed:true 2016-04-07T01:52:08.894-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 14507395, 14507395, 106 committed:true) SnapCreateDur 574.918423ms SnapOpenDur 57.596µs 2016-04-07T01:52:08.896-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:08.9-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:08.902-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:08.905-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:08.909-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:08.913-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:08.916-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:08.999-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14510000 QueuedCount 50000 2016-04-07T01:52:09.006-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:09.365-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:09.373-07:00 [Info] connected with 1 indexers 2016-04-07T01:52:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T01:52:09.374-07:00 [Info] client load stats {} 2016-04-07T01:52:09.448-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14520000 QueuedCount 50000 2016-04-07T01:52:09.641-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":5425376,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":4981756992,"bucket-1:myindex:avg_ts_items_count":88565,"bucket-1:myindex:build_progress":72,"bucket-1:myindex:data_size":3859955084,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":11049111552,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":50104,"bucket-1:myindex:frag_percent":65,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":1519142687,"bucket-1:myindex:items_count":14422959,"bucket-1:myindex:num_commits":106,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":14524623,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":14503113,"bucket-1:myindex:num_docs_queued":5496887,"bucket-1:myindex:num_flush_queued":14574730,"bucket-1:myindex:num_items_flushed":14524624,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":106,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":4752635757,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"106 60199571077 6384015854842454711","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"14524627 86507391690 451296686640703708","bucket-1:myindex:timings/storage_info":"425 3617213 55961242861","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"105 35938152 13698804153656","bucket-1:myindex:timings/storage_meta_set":"106 36196517 13495510766593","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"106 4515449 209482655523","bucket-1:myindex:timings/storage_set":"29049252 528451264380 -7077062994965724316","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":20000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"142 178493524 2109977268715240","bucket-1:ts_queue_size":510,"indexer_state":"Active","memory_quota":31457280000,"memory_used":15592753152,"memory_used_storage":11049525248,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"2478 395749948 160923704336168","uptime":"23m1.334173904s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T01:52:09.677-07:00 [Info] memstats {"Alloc":3243321160, "TotalAlloc":17407671352, "Sys":9854296392, "Lookups":5306, "Mallocs":472901099,"Frees":396794666, "HeapAlloc":3243321160, "HeapSys":9384394752, "HeapIdle":5138612224, "HeapInuse":4245782528,"HeapReleased":1641373696, "HeapObjects":76106433,"GCSys":297445376, "LastGC":1460019004549771477,"PauseTotalNs":27118358, "PauseNs":[], "NumGC":21} 2016-04-07T01:52:09.679-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:3875580695, Disk:11049111552, ExtraSnapshotData:0, Fragmentation:64.92405134330932%, mode : circular 2016-04-07T01:52:09.679-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:52:09.94-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14530000 QueuedCount 50000 2016-04-07T01:52:10.361-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14540000 QueuedCount 50000 2016-04-07T01:52:10.735-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:10.764-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:10.862-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14550000 QueuedCount 50000 2016-04-07T01:52:10.943-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:11.342-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14560000 QueuedCount 44022 2016-04-07T01:52:11.801-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14570000 QueuedCount 34022 2016-04-07T01:52:11.879-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:52:11.894-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:52:11.894-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:52:12.298-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14580000 QueuedCount 24022 2016-04-07T01:52:12.759-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14590000 QueuedCount 14022 2016-04-07T01:52:13.262-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14600000 QueuedCount 4022 2016-04-07T01:52:14.084-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.5µs CommitTime 619.419621ms TotalFlushTime 10m26.582943949s TotalCommitTime 1m0.818990698s 2016-04-07T01:52:14.085-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 14604022, 14604022, 107 committed:true 2016-04-07T01:52:14.085-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 14604022, 14604022, 107 committed:true) SnapCreateDur 627.503371ms SnapOpenDur 68.326µs 2016-04-07T01:52:14.087-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:14.09-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:14.091-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:14.093-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:14.095-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:14.098-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:14.101-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:14.105-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:14.107-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:14.11-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:14.112-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:14.115-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:14.118-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:14.289-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:14.302-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14610000 QueuedCount 50000 2016-04-07T01:52:14.56-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:14.751-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14620000 QueuedCount 50000 2016-04-07T01:52:15.02-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:15.13-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:15.238-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14630000 QueuedCount 50000 2016-04-07T01:52:15.339-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:15.736-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14640000 QueuedCount 50000 2016-04-07T01:52:16.26-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14650000 QueuedCount 50000 2016-04-07T01:52:16.797-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14660000 QueuedCount 50000 2016-04-07T01:52:17.345-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14670000 QueuedCount 50000 2016-04-07T01:52:17.892-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14680000 QueuedCount 50000 2016-04-07T01:52:18.374-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14690000 QueuedCount 50000 2016-04-07T01:52:18.817-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:18.947-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14700000 QueuedCount 50000 2016-04-07T01:52:19.505-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14710000 QueuedCount 50000 2016-04-07T01:52:20.079-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14720000 QueuedCount 44457 2016-04-07T01:52:20.626-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14730000 QueuedCount 34457 2016-04-07T01:52:21.188-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14740000 QueuedCount 24457 2016-04-07T01:52:21.6-07:00 [Info] Indexer::ReadMemstats Time Taken 14.572517ms 2016-04-07T01:52:21.752-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14750000 QueuedCount 14457 2016-04-07T01:52:22.332-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14760000 QueuedCount 4457 2016-04-07T01:52:23.536-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.435µs CommitTime 965.149979ms TotalFlushTime 10m34.917581825s TotalCommitTime 1m1.784140677s 2016-04-07T01:52:23.536-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 14764457, 14764457, 108 committed:true 2016-04-07T01:52:23.536-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 14764457, 14764457, 108 committed:true) SnapCreateDur 973.82334ms SnapOpenDur 47.931µs 2016-04-07T01:52:23.54-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:23.541-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:23.544-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:23.546-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:23.549-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:23.55-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:23.553-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:23.555-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:23.557-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:23.56-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:23.562-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:23.564-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:23.751-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14770000 QueuedCount 50000 2016-04-07T01:52:24.198-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14780000 QueuedCount 50000 2016-04-07T01:52:24.659-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14790000 QueuedCount 50000 2016-04-07T01:52:25.156-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14800000 QueuedCount 50000 2016-04-07T01:52:25.654-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14810000 QueuedCount 50000 2016-04-07T01:52:26.175-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14820000 QueuedCount 50000 2016-04-07T01:52:26.635-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14830000 QueuedCount 50000 2016-04-07T01:52:26.93-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:52:26.943-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:52:26.943-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:52:27.179-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14840000 QueuedCount 50000 2016-04-07T01:52:27.713-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14850000 QueuedCount 50000 2016-04-07T01:52:28.27-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14860000 QueuedCount 50000 2016-04-07T01:52:28.844-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14870000 QueuedCount 50000 2016-04-07T01:52:29.07-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:29.41-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14880000 QueuedCount 50000 2016-04-07T01:52:29.931-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14890000 QueuedCount 42883 2016-04-07T01:52:30.466-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14900000 QueuedCount 32883 2016-04-07T01:52:30.96-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14910000 QueuedCount 22883 2016-04-07T01:52:31.401-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14920000 QueuedCount 12883 2016-04-07T01:52:31.902-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14930000 QueuedCount 2883 2016-04-07T01:52:32.87-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 31.481µs CommitTime 757.270417ms TotalFlushTime 10m43.342095737s TotalCommitTime 1m2.541411094s 2016-04-07T01:52:32.87-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 14932883, 14932883, 109 committed:true 2016-04-07T01:52:32.87-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 14932883, 14932883, 109 committed:true) SnapCreateDur 765.755499ms SnapOpenDur 76.905µs 2016-04-07T01:52:32.873-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:32.876-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:32.878-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:32.882-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:32.884-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:32.887-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:32.89-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:32.892-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:32.896-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:32.899-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:33.082-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:33.157-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14940000 QueuedCount 50000 2016-04-07T01:52:33.41-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:33.601-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14950000 QueuedCount 50000 2016-04-07T01:52:33.694-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:33.975-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:34.083-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14960000 QueuedCount 50000 2016-04-07T01:52:34.394-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:34.576-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14970000 QueuedCount 50000 2016-04-07T01:52:35.029-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14980000 QueuedCount 43596 2016-04-07T01:52:35.41-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 14990000 QueuedCount 33596 2016-04-07T01:52:35.863-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15000000 QueuedCount 23596 2016-04-07T01:52:36.328-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15010000 QueuedCount 13596 2016-04-07T01:52:36.792-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15020000 QueuedCount 3596 2016-04-07T01:52:36.972-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:52:36.987-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:52:36.987-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:52:37.612-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 22.075µs CommitTime 597.50576ms TotalFlushTime 10m47.399407348s TotalCommitTime 1m3.138916854s 2016-04-07T01:52:37.612-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 15023596, 15023596, 110 committed:true 2016-04-07T01:52:37.612-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 15023596, 15023596, 110 committed:true) SnapCreateDur 606.734993ms SnapOpenDur 61.617µs 2016-04-07T01:52:37.617-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:37.622-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:37.634-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:37.635-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:37.637-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:37.64-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:37.808-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:37.895-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15030000 QueuedCount 50000 2016-04-07T01:52:37.914-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:38.173-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:38.362-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15040000 QueuedCount 50000 2016-04-07T01:52:38.676-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:38.771-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15050000 QueuedCount 50000 2016-04-07T01:52:38.949-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:39.162-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:39.264-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15060000 QueuedCount 50000 2016-04-07T01:52:39.364-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:39.68-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:4022809505, Disk:11049111552, ExtraSnapshotData:0, Fragmentation:63.59155678655601%, mode : circular 2016-04-07T01:52:39.68-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:52:39.777-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15070000 QueuedCount 50000 2016-04-07T01:52:39.784-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:40.278-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15080000 QueuedCount 44182 2016-04-07T01:52:40.785-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15090000 QueuedCount 34182 2016-04-07T01:52:41.288-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15100000 QueuedCount 24182 2016-04-07T01:52:41.786-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15110000 QueuedCount 14182 2016-04-07T01:52:42.288-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15120000 QueuedCount 4182 2016-04-07T01:52:43.292-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 26.958µs CommitTime 774.809375ms TotalFlushTime 10m52.211101307s TotalCommitTime 1m3.913726229s 2016-04-07T01:52:43.292-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 15124182, 15124182, 111 committed:true 2016-04-07T01:52:43.292-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 15124182, 15124182, 111 committed:true) SnapCreateDur 784.129559ms SnapOpenDur 65.092µs 2016-04-07T01:52:43.295-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:43.299-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:43.303-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:43.307-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:43.311-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:43.316-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:43.324-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:43.519-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15130000 QueuedCount 50000 2016-04-07T01:52:43.972-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15140000 QueuedCount 50000 2016-04-07T01:52:44.415-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15150000 QueuedCount 50000 2016-04-07T01:52:44.891-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15160000 QueuedCount 50000 2016-04-07T01:52:44.898-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:45.386-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15170000 QueuedCount 46435 2016-04-07T01:52:45.865-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15180000 QueuedCount 36435 2016-04-07T01:52:46.352-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15190000 QueuedCount 26435 2016-04-07T01:52:46.854-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15200000 QueuedCount 16435 2016-04-07T01:52:47.009-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:52:47.022-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:52:47.022-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:52:47.271-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15210000 QueuedCount 6435 2016-04-07T01:52:48.347-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.423µs CommitTime 684.392929ms TotalFlushTime 10m56.492322179s TotalCommitTime 1m4.598119158s 2016-04-07T01:52:48.347-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 15216435, 15216435, 112 committed:true 2016-04-07T01:52:48.347-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 15216435, 15216435, 112 committed:true) SnapCreateDur 692.773605ms SnapOpenDur 41.225µs 2016-04-07T01:52:48.35-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:48.353-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:48.355-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:48.358-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:48.361-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:48.363-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:48.367-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:48.37-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:48.374-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:48.376-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:48.379-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:48.476-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15220000 QueuedCount 50000 2016-04-07T01:52:48.561-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:48.751-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:48.937-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15230000 QueuedCount 50000 2016-04-07T01:52:48.945-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:49.211-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:49.398-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15240000 QueuedCount 50000 2016-04-07T01:52:49.49-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:49.802-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:49.888-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15250000 QueuedCount 50000 2016-04-07T01:52:50.181-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:50.388-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15260000 QueuedCount 50000 2016-04-07T01:52:50.907-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15270000 QueuedCount 50000 2016-04-07T01:52:51.388-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15280000 QueuedCount 43794 2016-04-07T01:52:51.803-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15290000 QueuedCount 33794 2016-04-07T01:52:52.3-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15300000 QueuedCount 23794 2016-04-07T01:52:52.776-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15310000 QueuedCount 13794 2016-04-07T01:52:53.254-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15320000 QueuedCount 3794 2016-04-07T01:52:54.125-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.714µs CommitTime 667.532565ms TotalFlushTime 11m1.501510132s TotalCommitTime 1m5.265651723s 2016-04-07T01:52:54.125-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 15323794, 15323794, 113 committed:true 2016-04-07T01:52:54.125-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 15323794, 15323794, 113 committed:true) SnapCreateDur 675.941809ms SnapOpenDur 46.41µs 2016-04-07T01:52:54.13-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:54.137-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:54.142-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:54.399-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15330000 QueuedCount 50000 2016-04-07T01:52:54.791-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15340000 QueuedCount 50000 2016-04-07T01:52:55.257-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15350000 QueuedCount 50000 2016-04-07T01:52:55.756-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15360000 QueuedCount 50000 2016-04-07T01:52:55.958-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:56.244-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15370000 QueuedCount 49188 2016-04-07T01:52:56.683-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15380000 QueuedCount 39188 2016-04-07T01:52:57.051-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:52:57.064-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:52:57.064-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:52:57.161-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15390000 QueuedCount 29188 2016-04-07T01:52:57.612-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15400000 QueuedCount 19188 2016-04-07T01:52:58.083-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15410000 QueuedCount 9188 2016-04-07T01:52:59.079-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.536µs CommitTime 614.385166ms TotalFlushTime 11m5.75649293s TotalCommitTime 1m5.880036889s 2016-04-07T01:52:59.079-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 15419188, 15419188, 114 committed:true 2016-04-07T01:52:59.079-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 15419188, 15419188, 114 committed:true) SnapCreateDur 622.896805ms SnapOpenDur 63.808µs 2016-04-07T01:52:59.099-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:59.101-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:59.102-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:59.103-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:59.107-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:59.109-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:59.111-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15420000 QueuedCount 50000 2016-04-07T01:52:59.216-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:59.381-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:59.403-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:59.48-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:59.495-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:52:59.57-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15430000 QueuedCount 50000 2016-04-07T01:53:00.037-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15440000 QueuedCount 50000 2016-04-07T01:53:00.514-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15450000 QueuedCount 50000 2016-04-07T01:53:00.63-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:00.938-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15460000 QueuedCount 50000 2016-04-07T01:53:01.441-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15470000 QueuedCount 50000 2016-04-07T01:53:01.549-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:01.953-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15480000 QueuedCount 50000 2016-04-07T01:53:02.472-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15490000 QueuedCount 48570 2016-04-07T01:53:02.961-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15500000 QueuedCount 38570 2016-04-07T01:53:03.446-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15510000 QueuedCount 28570 2016-04-07T01:53:03.943-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15520000 QueuedCount 18570 2016-04-07T01:53:04.428-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15530000 QueuedCount 8570 2016-04-07T01:53:05.496-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 35.278µs CommitTime 668.206724ms TotalFlushTime 11m11.389080931s TotalCommitTime 1m6.548243613s 2016-04-07T01:53:05.496-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 15538570, 15538570, 115 committed:true 2016-04-07T01:53:05.497-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 15538570, 15538570, 115 committed:true) SnapCreateDur 676.855398ms SnapOpenDur 84.861µs 2016-04-07T01:53:05.498-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:05.499-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:05.502-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:05.504-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:05.506-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:05.508-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:05.51-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:05.512-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:05.513-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:05.517-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:05.518-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:05.522-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:05.523-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:05.524-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:05.527-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:05.529-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15540000 QueuedCount 50000 2016-04-07T01:53:05.609-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:05.796-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:05.992-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:05.993-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15550000 QueuedCount 50000 2016-04-07T01:53:06.17-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:06.359-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:06.453-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15560000 QueuedCount 50000 2016-04-07T01:53:06.557-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:06.903-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15570000 QueuedCount 49112 2016-04-07T01:53:07.08-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:53:07.093-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:53:07.093-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:53:07.352-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15580000 QueuedCount 39112 2016-04-07T01:53:07.797-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15590000 QueuedCount 29112 2016-04-07T01:53:08.206-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15600000 QueuedCount 19112 2016-04-07T01:53:08.675-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15610000 QueuedCount 9112 2016-04-07T01:53:09.373-07:00 [Info] connected with 1 indexers 2016-04-07T01:53:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T01:53:09.374-07:00 [Info] client load stats {} 2016-04-07T01:53:09.642-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":4380888,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":5866853249,"bucket-1:myindex:avg_ts_items_count":109590,"bucket-1:myindex:build_progress":78,"bucket-1:myindex:data_size":4164110007,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":11049111552,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":0,"bucket-1:myindex:frag_percent":62,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":1634776185,"bucket-1:myindex:items_count":15538570,"bucket-1:myindex:num_commits":115,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":15619112,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":15612928,"bucket-1:myindex:num_docs_queued":4387072,"bucket-1:myindex:num_flush_queued":15619112,"bucket-1:myindex:num_items_flushed":15619112,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":115,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":6417902622,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"115 66548243613 -7478769119938852711","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"15619112 92866094215 451337210022954753","bucket-1:myindex:timings/storage_info":"464 3894743 59565068241","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"115 39562462 15179725973334","bucket-1:myindex:timings/storage_meta_set":"116 39355752 14565674595206","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"115 4921260 229569582896","bucket-1:myindex:timings/storage_set":"31238224 573671203369 -4134978013973577083","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":20000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"154 193287709 2145750168525725","bucket-1:ts_queue_size":373,"indexer_state":"Active","memory_quota":31457280000,"memory_used":14383851520,"memory_used_storage":11049517056,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"2621 430259424 199833290454590","uptime":"24m1.334642477s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T01:53:09.677-07:00 [Info] memstats {"Alloc":1875669360, "TotalAlloc":17695451560, "Sys":9854296392, "Lookups":5742, "Mallocs":483134007,"Frees":437270619, "HeapAlloc":1875669360, "HeapSys":9384755200, "HeapIdle":6347866112, "HeapInuse":3036889088,"HeapReleased":2186379264, "HeapObjects":45863388,"GCSys":297445376, "LastGC":1460019124960802989,"PauseTotalNs":28228488, "PauseNs":[1110130], "NumGC":22} 2016-04-07T01:53:09.68-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:4178132793, Disk:11049111552, ExtraSnapshotData:0, Fragmentation:62.18580314501652%, mode : circular 2016-04-07T01:53:09.68-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:53:09.709-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 26.751µs CommitTime 551.434821ms TotalFlushTime 11m14.978653967s TotalCommitTime 1m7.099678434s 2016-04-07T01:53:09.709-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 15619112, 15619112, 116 committed:true 2016-04-07T01:53:09.709-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 15619112, 15619112, 116 committed:true) SnapCreateDur 559.525079ms SnapOpenDur 68.245µs 2016-04-07T01:53:09.712-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:09.716-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:09.717-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:09.719-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:09.72-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:09.722-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:09.723-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:09.724-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:09.726-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:09.727-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:09.729-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:09.73-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15620000 QueuedCount 34258 2016-04-07T01:53:09.731-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:09.733-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:09.734-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:09.736-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:09.738-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:09.825-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:10.106-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:10.185-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15630000 QueuedCount 50000 2016-04-07T01:53:10.198-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:10.388-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:10.512-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:10.708-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15640000 QueuedCount 50000 2016-04-07T01:53:10.812-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:11.123-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:11.234-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15650000 QueuedCount 50000 2016-04-07T01:53:11.36-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:11.562-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:11.681-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15660000 QueuedCount 50000 2016-04-07T01:53:12.215-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15670000 QueuedCount 43111 2016-04-07T01:53:12.723-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15680000 QueuedCount 33111 2016-04-07T01:53:13.223-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15690000 QueuedCount 23111 2016-04-07T01:53:13.724-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15700000 QueuedCount 13111 2016-04-07T01:53:14.244-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15710000 QueuedCount 3111 2016-04-07T01:53:15.181-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.699µs CommitTime 793.242694ms TotalFlushTime 11m19.573521236s TotalCommitTime 1m7.892921128s 2016-04-07T01:53:15.181-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 15713111, 15713111, 117 committed:true 2016-04-07T01:53:15.182-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 15713111, 15713111, 117 committed:true) SnapCreateDur 801.548299ms SnapOpenDur 81.653µs 2016-04-07T01:53:15.184-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:15.186-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:15.199-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:15.514-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15720000 QueuedCount 50000 2016-04-07T01:53:16.039-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15730000 QueuedCount 50000 2016-04-07T01:53:16.56-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15740000 QueuedCount 50000 2016-04-07T01:53:17.098-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15750000 QueuedCount 50000 2016-04-07T01:53:17.558-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15760000 QueuedCount 50000 2016-04-07T01:53:17.774-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:18.088-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15770000 QueuedCount 50000 2016-04-07T01:53:18.598-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15780000 QueuedCount 40542 2016-04-07T01:53:19.1-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15790000 QueuedCount 30542 2016-04-07T01:53:19.611-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15800000 QueuedCount 20542 2016-04-07T01:53:20.119-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15810000 QueuedCount 10542 2016-04-07T01:53:20.629-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15820000 QueuedCount 542 2016-04-07T01:53:21.45-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 22.429µs CommitTime 803.191718ms TotalFlushTime 11m24.957645756s TotalCommitTime 1m8.696112846s 2016-04-07T01:53:21.45-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 15820542, 15820542, 118 committed:true 2016-04-07T01:53:21.451-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 15820542, 15820542, 118 committed:true) SnapCreateDur 811.536344ms SnapOpenDur 75.79µs 2016-04-07T01:53:21.452-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.453-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.456-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.456-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.458-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.459-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.459-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.46-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.462-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.463-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.464-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.465-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.466-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.468-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.469-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.471-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.473-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.483-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.58-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.682-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.781-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.793-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.899-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15830000 QueuedCount 50000 2016-04-07T01:53:21.911-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:21.988-07:00 [Info] Indexer::ReadMemstats Time Taken 15.448603ms 2016-04-07T01:53:22.108-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:22.136-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:53:22.15-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:53:22.15-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:53:22.229-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:22.438-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15840000 QueuedCount 50000 2016-04-07T01:53:22.977-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15850000 QueuedCount 50000 2016-04-07T01:53:23.425-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:23.517-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15860000 QueuedCount 50000 2016-04-07T01:53:24.052-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15870000 QueuedCount 44958 2016-04-07T01:53:24.568-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15880000 QueuedCount 34958 2016-04-07T01:53:24.984-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15890000 QueuedCount 24958 2016-04-07T01:53:25.483-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15900000 QueuedCount 14958 2016-04-07T01:53:25.986-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15910000 QueuedCount 4958 2016-04-07T01:53:27.013-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 23.193µs CommitTime 729.662167ms TotalFlushTime 11m29.720798567s TotalCommitTime 1m9.425775013s 2016-04-07T01:53:27.013-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 15914958, 15914958, 119 committed:true 2016-04-07T01:53:27.013-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 15914958, 15914958, 119 committed:true) SnapCreateDur 737.666275ms SnapOpenDur 71.539µs 2016-04-07T01:53:27.015-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:27.016-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:27.017-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:27.018-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:27.019-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:27.02-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:27.021-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:27.023-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:27.024-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:27.027-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:27.027-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:27.03-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:27.032-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:27.035-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:27.246-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15920000 QueuedCount 50000 2016-04-07T01:53:27.778-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15930000 QueuedCount 50000 2016-04-07T01:53:28.301-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15940000 QueuedCount 50000 2016-04-07T01:53:28.813-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15950000 QueuedCount 50000 2016-04-07T01:53:29.259-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15960000 QueuedCount 50000 2016-04-07T01:53:29.682-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:29.802-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15970000 QueuedCount 50000 2016-04-07T01:53:30.317-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15980000 QueuedCount 40789 2016-04-07T01:53:30.816-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 15990000 QueuedCount 30789 2016-04-07T01:53:31.319-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16000000 QueuedCount 20789 2016-04-07T01:53:31.811-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16010000 QueuedCount 10789 2016-04-07T01:53:32.174-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:53:32.187-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:53:32.187-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:53:32.316-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16020000 QueuedCount 789 2016-04-07T01:53:33.036-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.323µs CommitTime 700.735508ms TotalFlushTime 11m34.964231109s TotalCommitTime 1m10.126510521s 2016-04-07T01:53:33.036-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 16020789, 16020789, 120 committed:true 2016-04-07T01:53:33.036-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 16020789, 16020789, 120 committed:true) SnapCreateDur 709.17644ms SnapOpenDur 77.225µs 2016-04-07T01:53:33.038-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.04-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.041-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.042-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.043-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.048-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.049-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.05-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.051-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.052-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.053-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.054-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.056-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.057-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.058-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.059-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.06-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.165-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.295-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.494-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16030000 QueuedCount 50000 2016-04-07T01:53:33.585-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.694-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:33.817-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:34.033-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16040000 QueuedCount 50000 2016-04-07T01:53:34.121-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:34.227-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:34.336-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:34.537-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16050000 QueuedCount 50000 2016-04-07T01:53:34.547-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:34.841-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:35.058-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16060000 QueuedCount 49958 2016-04-07T01:53:35.55-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16070000 QueuedCount 39958 2016-04-07T01:53:35.991-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16080000 QueuedCount 29958 2016-04-07T01:53:36.485-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16090000 QueuedCount 19958 2016-04-07T01:53:36.986-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16100000 QueuedCount 9958 2016-04-07T01:53:38.181-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 22.404µs CommitTime 664.11434ms TotalFlushTime 11m39.36455764s TotalCommitTime 1m10.790624861s 2016-04-07T01:53:38.181-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 16109958, 16109958, 121 committed:true 2016-04-07T01:53:38.181-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 16109958, 16109958, 121 committed:true) SnapCreateDur 673.555173ms SnapOpenDur 73.499µs 2016-04-07T01:53:38.184-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16110000 QueuedCount 5330 2016-04-07T01:53:38.185-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:38.192-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:38.195-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:38.196-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:38.2-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:38.621-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16120000 QueuedCount 50000 2016-04-07T01:53:39.139-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16130000 QueuedCount 50000 2016-04-07T01:53:39.673-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16140000 QueuedCount 50000 2016-04-07T01:53:39.681-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:4318464470, Disk:12602109952, ExtraSnapshotData:0, Fragmentation:65.73221082462747%, mode : circular 2016-04-07T01:53:39.681-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:53:40.249-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16150000 QueuedCount 50000 2016-04-07T01:53:40.479-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:40.792-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16160000 QueuedCount 47564 2016-04-07T01:53:41.308-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16170000 QueuedCount 37564 2016-04-07T01:53:41.82-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16180000 QueuedCount 27564 2016-04-07T01:53:42.212-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:53:42.225-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:53:42.225-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:53:42.357-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16190000 QueuedCount 17564 2016-04-07T01:53:42.8-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16200000 QueuedCount 7564 2016-04-07T01:53:46.2-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.316µs CommitTime 2.967651651s TotalFlushTime 11m44.327998525s TotalCommitTime 1m13.758276512s 2016-04-07T01:53:46.2-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 16207564, 16207564, 122 committed:true 2016-04-07T01:53:46.2-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 16207564, 16207564, 122 committed:true) SnapCreateDur 2.97605874s SnapOpenDur 77.949µs 2016-04-07T01:53:46.284-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16210000 QueuedCount 1248 2016-04-07T01:53:46.413-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.482µs CommitTime 101.729242ms TotalFlushTime 11m44.426320388s TotalCommitTime 1m13.860005754s 2016-04-07T01:53:46.413-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 16211248, 16211248, 123 committed:true 2016-04-07T01:53:46.413-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 16211248, 16211248, 123 committed:true) SnapCreateDur 110.870895ms SnapOpenDur 41.928µs 2016-04-07T01:53:46.415-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.417-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.418-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.419-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.42-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.421-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.423-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.425-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.426-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.428-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.43-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.432-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.433-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.434-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.436-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.437-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.439-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.44-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.441-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.454-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.543-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.739-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:46.828-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16220000 QueuedCount 50000 2016-04-07T01:53:46.922-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:47.226-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:47.317-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16230000 QueuedCount 50000 2016-04-07T01:53:47.831-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16240000 QueuedCount 50000 2016-04-07T01:53:48.154-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:48.275-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16250000 QueuedCount 50000 2016-04-07T01:53:48.352-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:48.558-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:48.664-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:48.783-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16260000 QueuedCount 49993 2016-04-07T01:53:48.784-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:49.088-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:49.307-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16270000 QueuedCount 50000 2016-04-07T01:53:49.849-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16280000 QueuedCount 50000 2016-04-07T01:53:49.963-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:50.086-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:50.281-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:50.302-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:50.404-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16290000 QueuedCount 50000 2016-04-07T01:53:50.502-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:50.623-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:50.637-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:50.834-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:50.955-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:50.956-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16300000 QueuedCount 50000 2016-04-07T01:53:50.963-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:51.058-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:51.07-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:51.185-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:51.301-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:51.51-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16310000 QueuedCount 50000 2016-04-07T01:53:52.054-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16320000 QueuedCount 41102 2016-04-07T01:53:52.501-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16330000 QueuedCount 31102 2016-04-07T01:53:53.031-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16340000 QueuedCount 21102 2016-04-07T01:53:53.551-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16350000 QueuedCount 11102 2016-04-07T01:53:54.108-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16360000 QueuedCount 1102 2016-04-07T01:53:55.063-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 32.473µs CommitTime 850.52823ms TotalFlushTime 11m52.09552812s TotalCommitTime 1m14.710533984s 2016-04-07T01:53:55.063-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 16361102, 16361102, 124 committed:true 2016-04-07T01:53:55.063-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 16361102, 16361102, 124 committed:true) SnapCreateDur 859.384335ms SnapOpenDur 51.374µs 2016-04-07T01:53:55.065-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.067-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.068-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.071-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.072-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.074-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.075-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.077-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.078-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.081-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.082-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.083-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.086-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.087-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.088-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.09-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.092-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.184-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.299-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.48-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16370000 QueuedCount 49990 2016-04-07T01:53:55.48-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.758-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:55.962-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16380000 QueuedCount 50000 2016-04-07T01:53:56.078-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:56.473-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16390000 QueuedCount 50000 2016-04-07T01:53:56.481-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:56.892-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16400000 QueuedCount 46594 2016-04-07T01:53:57.266-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:53:57.28-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:53:57.28-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:53:57.359-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16410000 QueuedCount 36594 2016-04-07T01:53:57.813-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16420000 QueuedCount 26594 2016-04-07T01:53:58.301-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16430000 QueuedCount 16594 2016-04-07T01:53:58.767-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16440000 QueuedCount 6594 2016-04-07T01:53:59.696-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 15.438µs CommitTime 622.682739ms TotalFlushTime 11m56.03522878s TotalCommitTime 1m15.333216723s 2016-04-07T01:53:59.696-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 16446594, 16446594, 125 committed:true 2016-04-07T01:53:59.696-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 16446594, 16446594, 125 committed:true) SnapCreateDur 631.711602ms SnapOpenDur 60.783µs 2016-04-07T01:53:59.699-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:59.7-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:59.703-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:59.708-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:59.71-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:59.713-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:59.715-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:59.716-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:59.719-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:59.722-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:59.724-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:59.73-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:53:59.815-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16450000 QueuedCount 50000 2016-04-07T01:53:59.895-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:00.106-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:00.281-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16460000 QueuedCount 50000 2016-04-07T01:54:00.378-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:00.57-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:00.758-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16470000 QueuedCount 50000 2016-04-07T01:54:00.759-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:01.037-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:01.239-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16480000 QueuedCount 50000 2016-04-07T01:54:01.248-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:01.54-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:01.751-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16490000 QueuedCount 50000 2016-04-07T01:54:02.271-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16500000 QueuedCount 41772 2016-04-07T01:54:02.69-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16510000 QueuedCount 31772 2016-04-07T01:54:03.195-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16520000 QueuedCount 21772 2016-04-07T01:54:03.702-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16530000 QueuedCount 11772 2016-04-07T01:54:04.227-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16540000 QueuedCount 1772 2016-04-07T01:54:05.181-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.776µs CommitTime 838.908401ms TotalFlushTime 12m0.592329118s TotalCommitTime 1m16.172125124s 2016-04-07T01:54:05.181-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 16541772, 16541772, 126 committed:true 2016-04-07T01:54:05.181-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 16541772, 16541772, 126 committed:true) SnapCreateDur 847.732473ms SnapOpenDur 79.114µs 2016-04-07T01:54:05.193-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:06.016-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16550000 QueuedCount 50000 2016-04-07T01:54:06.022-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:06.124-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:06.424-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:06.43-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16560000 QueuedCount 50000 2016-04-07T01:54:06.704-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:06.932-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16570000 QueuedCount 50000 2016-04-07T01:54:07.428-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16580000 QueuedCount 42574 2016-04-07T01:54:07.9-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16590000 QueuedCount 32574 2016-04-07T01:54:08.4-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16600000 QueuedCount 22574 2016-04-07T01:54:08.88-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16610000 QueuedCount 12574 2016-04-07T01:54:09.365-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16620000 QueuedCount 2574 2016-04-07T01:54:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:54:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T01:54:09.374-07:00 [Info] client load stats {} 2016-04-07T01:54:09.643-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":3377426,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":5417634766,"bucket-1:myindex:avg_ts_items_count":93957,"bucket-1:myindex:build_progress":83,"bucket-1:myindex:data_size":4439398716,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":12950466560,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":0,"bucket-1:myindex:frag_percent":65,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":1740542580,"bucket-1:myindex:items_count":16541772,"bucket-1:myindex:num_commits":126,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":16622574,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":16614354,"bucket-1:myindex:num_docs_queued":3385646,"bucket-1:myindex:num_flush_queued":16622574,"bucket-1:myindex:num_items_flushed":16622574,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":126,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":5485051165,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"126 76172125124 6196351665142180230","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"16622574 99285696844 631568924480016626","bucket-1:myindex:timings/storage_info":"508 4221935 64206580365","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"126 43607573 16776870412271","bucket-1:myindex:timings/storage_meta_set":"127 43495348 16203591174366","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"126 5496759 261061977089","bucket-1:myindex:timings/storage_set":"33245148 616560030445 -1204605830925747283","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":20000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"166 203592308 2154614021616802","bucket-1:ts_queue_size":167,"indexer_state":"Active","memory_quota":31457280000,"memory_used":16437757952,"memory_used_storage":12950872064,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"2764 459380220 205991263116354","uptime":"25m1.335773801s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T01:54:09.677-07:00 [Info] memstats {"Alloc":2172144200, "TotalAlloc":17991926400, "Sys":9854296392, "Lookups":6174, "Mallocs":493719951,"Frees":440038163, "HeapAlloc":2172144200, "HeapSys":9383936000, "HeapIdle":6194495488, "HeapInuse":3189440512,"HeapReleased":2184511488, "HeapObjects":53681788,"GCSys":297445376, "LastGC":1460019124960802989,"PauseTotalNs":28228488, "PauseNs":[], "NumGC":22} 2016-04-07T01:54:09.681-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:4453048627, Disk:12950466560, ExtraSnapshotData:0, Fragmentation:65.6147629402473%, mode : circular 2016-04-07T01:54:09.681-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:54:10.114-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 37.086µs CommitTime 635.346234ms TotalFlushTime 12m4.80925683s TotalCommitTime 1m16.807471358s 2016-04-07T01:54:10.114-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 16622574, 16622574, 127 committed:true 2016-04-07T01:54:10.114-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 16622574, 16622574, 127 committed:true) SnapCreateDur 643.825106ms SnapOpenDur 52.575µs 2016-04-07T01:54:10.116-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:10.125-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:10.125-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:10.247-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:10.46-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16630000 QueuedCount 50000 2016-04-07T01:54:10.75-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:10.961-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:10.962-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16640000 QueuedCount 50000 2016-04-07T01:54:11.151-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:11.268-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:11.47-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16650000 QueuedCount 50000 2016-04-07T01:54:11.568-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:11.679-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:11.799-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:11.989-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16660000 QueuedCount 50000 2016-04-07T01:54:12.32-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:54:12.333-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:54:12.333-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:54:12.506-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16670000 QueuedCount 44570 2016-04-07T01:54:13-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16680000 QueuedCount 34570 2016-04-07T01:54:13.424-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16690000 QueuedCount 24570 2016-04-07T01:54:13.932-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16700000 QueuedCount 14570 2016-04-07T01:54:14.442-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16710000 QueuedCount 4570 2016-04-07T01:54:15.53-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 16.703µs CommitTime 860.627815ms TotalFlushTime 12m9.276059709s TotalCommitTime 1m17.668099173s 2016-04-07T01:54:15.53-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 16714570, 16714570, 128 committed:true 2016-04-07T01:54:15.53-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 16714570, 16714570, 128 committed:true) SnapCreateDur 869.986731ms SnapOpenDur 33.418µs 2016-04-07T01:54:15.537-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:15.539-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:15.544-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:15.546-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:15.55-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:15.552-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:15.557-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:15.558-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:15.75-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16720000 QueuedCount 50000 2016-04-07T01:54:16.236-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16730000 QueuedCount 50000 2016-04-07T01:54:16.737-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16740000 QueuedCount 50000 2016-04-07T01:54:17.256-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16750000 QueuedCount 50000 2016-04-07T01:54:17.382-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:17.794-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16760000 QueuedCount 50000 2016-04-07T01:54:18.343-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16770000 QueuedCount 50000 2016-04-07T01:54:18.804-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16780000 QueuedCount 42662 2016-04-07T01:54:19.317-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16790000 QueuedCount 32662 2016-04-07T01:54:19.845-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16800000 QueuedCount 22662 2016-04-07T01:54:20.372-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16810000 QueuedCount 12662 2016-04-07T01:54:20.907-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16820000 QueuedCount 2662 2016-04-07T01:54:21.863-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.137µs CommitTime 825.85248ms TotalFlushTime 12m14.672763928s TotalCommitTime 1m18.493951653s 2016-04-07T01:54:21.865-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 16822662, 16822662, 129 committed:true 2016-04-07T01:54:21.865-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 16822662, 16822662, 129 committed:true) SnapCreateDur 836.090594ms SnapOpenDur 53.243µs 2016-04-07T01:54:21.866-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:21.869-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:21.871-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:21.885-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:21.885-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:21.888-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:21.891-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:21.894-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:22.093-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:22.184-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16830000 QueuedCount 50000 2016-04-07T01:54:22.28-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:22.318-07:00 [Info] Indexer::ReadMemstats Time Taken 12.503394ms 2016-04-07T01:54:22.477-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:22.677-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16840000 QueuedCount 50000 2016-04-07T01:54:23.141-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16850000 QueuedCount 41176 2016-04-07T01:54:23.609-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16860000 QueuedCount 31176 2016-04-07T01:54:24.096-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16870000 QueuedCount 21176 2016-04-07T01:54:24.513-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16880000 QueuedCount 11176 2016-04-07T01:54:25.006-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16890000 QueuedCount 1176 2016-04-07T01:54:25.794-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 16.565µs CommitTime 686.888315ms TotalFlushTime 12m17.860882784s TotalCommitTime 1m19.180839968s 2016-04-07T01:54:25.794-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 16891176, 16891176, 130 committed:true 2016-04-07T01:54:25.795-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 16891176, 16891176, 130 committed:true) SnapCreateDur 695.144965ms SnapOpenDur 53.61µs 2016-04-07T01:54:25.798-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:25.798-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:25.809-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:25.81-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:25.81-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:26.175-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:26.179-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16900000 QueuedCount 50000 2016-04-07T01:54:26.268-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:26.358-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:26.465-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:26.64-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16910000 QueuedCount 50000 2016-04-07T01:54:26.657-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:26.747-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:27.143-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16920000 QueuedCount 50000 2016-04-07T01:54:27.375-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:54:27.39-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:54:27.39-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:54:27.584-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16930000 QueuedCount 50000 2016-04-07T01:54:27.903-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:28.134-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16940000 QueuedCount 50000 2016-04-07T01:54:28.681-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16950000 QueuedCount 50000 2016-04-07T01:54:29.235-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16960000 QueuedCount 50000 2016-04-07T01:54:29.79-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16970000 QueuedCount 50000 2016-04-07T01:54:30.334-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16980000 QueuedCount 43457 2016-04-07T01:54:30.85-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 16990000 QueuedCount 33457 2016-04-07T01:54:31.369-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17000000 QueuedCount 23457 2016-04-07T01:54:31.931-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17010000 QueuedCount 13457 2016-04-07T01:54:32.396-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17020000 QueuedCount 3457 2016-04-07T01:54:33.516-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 36.682µs CommitTime 899.077965ms TotalFlushTime 12m24.573864867s TotalCommitTime 1m20.079917933s 2016-04-07T01:54:33.516-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 17023457, 17023457, 131 committed:true 2016-04-07T01:54:33.517-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 17023457, 17023457, 131 committed:true) SnapCreateDur 907.21544ms SnapOpenDur 89.294µs 2016-04-07T01:54:33.792-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17030000 QueuedCount 20852 2016-04-07T01:54:34.24-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17040000 QueuedCount 10852 2016-04-07T01:54:34.598-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17050000 QueuedCount 852 2016-04-07T01:54:35.039-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.406µs CommitTime 350.692849ms TotalFlushTime 12m25.706978526s TotalCommitTime 1m20.430610782s 2016-04-07T01:54:35.039-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 17050852, 17050852, 132 committed:true 2016-04-07T01:54:35.039-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 17050852, 17050852, 132 committed:true) SnapCreateDur 358.890769ms SnapOpenDur 40.668µs 2016-04-07T01:54:35.042-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:35.044-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:35.047-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:35.049-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:35.05-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:35.052-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:35.053-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:35.057-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:35.059-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:35.06-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:35.063-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:35.076-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:35.428-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:35.439-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17060000 QueuedCount 50000 2016-04-07T01:54:35.744-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:35.932-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:35.936-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:35.937-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17070000 QueuedCount 50000 2016-04-07T01:54:36.13-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:36.13-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:36.442-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:36.445-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17080000 QueuedCount 50000 2016-04-07T01:54:36.546-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:36.758-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:36.876-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:36.961-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17090000 QueuedCount 50000 2016-04-07T01:54:37.193-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:37.293-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:37.417-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17100000 QueuedCount 50000 2016-04-07T01:54:37.614-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:37.74-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:37.964-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17110000 QueuedCount 50000 2016-04-07T01:54:38.528-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17120000 QueuedCount 50000 2016-04-07T01:54:39.122-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17130000 QueuedCount 50000 2016-04-07T01:54:39.682-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:4591111391, Disk:12950466560, ExtraSnapshotData:0, Fragmentation:64.54867961915343%, mode : circular 2016-04-07T01:54:39.682-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:54:39.702-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17140000 QueuedCount 50000 2016-04-07T01:54:40.04-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:40.268-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17150000 QueuedCount 50000 2016-04-07T01:54:40.853-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17160000 QueuedCount 48088 2016-04-07T01:54:41.396-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17170000 QueuedCount 38088 2016-04-07T01:54:41.922-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17180000 QueuedCount 28088 2016-04-07T01:54:42.375-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17190000 QueuedCount 18088 2016-04-07T01:54:42.427-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:54:42.44-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:54:42.44-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:54:42.912-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17200000 QueuedCount 8088 2016-04-07T01:54:44.249-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 35.664µs CommitTime 894.703709ms TotalFlushTime 12m33.890404702s TotalCommitTime 1m21.325314491s 2016-04-07T01:54:44.249-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 17208088, 17208088, 133 committed:true 2016-04-07T01:54:44.249-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 17208088, 17208088, 133 committed:true) SnapCreateDur 902.808614ms SnapOpenDur 59.519µs 2016-04-07T01:54:44.25-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:44.251-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:44.253-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:44.256-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:44.27-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:44.287-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17210000 QueuedCount 50000 2016-04-07T01:54:44.766-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17220000 QueuedCount 50000 2016-04-07T01:54:45.281-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:45.29-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17230000 QueuedCount 50000 2016-04-07T01:54:45.82-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17240000 QueuedCount 50000 2016-04-07T01:54:46.353-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17250000 QueuedCount 50000 2016-04-07T01:54:46.853-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17260000 QueuedCount 40552 2016-04-07T01:54:47.371-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17270000 QueuedCount 30552 2016-04-07T01:54:47.886-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17280000 QueuedCount 20552 2016-04-07T01:54:48.341-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17290000 QueuedCount 10552 2016-04-07T01:54:48.853-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17300000 QueuedCount 552 2016-04-07T01:54:49.716-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.23µs CommitTime 766.142701ms TotalFlushTime 12m38.515011778s TotalCommitTime 1m22.091457192s 2016-04-07T01:54:49.716-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 17300552, 17300552, 134 committed:true 2016-04-07T01:54:49.716-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 17300552, 17300552, 134 committed:true) SnapCreateDur 774.098132ms SnapOpenDur 43.053µs 2016-04-07T01:54:49.731-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:49.846-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:50.05-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:50.15-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17310000 QueuedCount 50000 2016-04-07T01:54:50.158-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:50.343-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:50.635-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17320000 QueuedCount 50000 2016-04-07T01:54:51.089-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17330000 QueuedCount 42007 2016-04-07T01:54:51.565-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17340000 QueuedCount 32007 2016-04-07T01:54:52.059-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17350000 QueuedCount 22007 2016-04-07T01:54:52.472-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:54:52.485-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:54:52.485-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:54:52.574-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17360000 QueuedCount 12007 2016-04-07T01:54:52.977-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17370000 QueuedCount 2007 2016-04-07T01:54:53.776-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.622µs CommitTime 676.953508ms TotalFlushTime 12m41.824811747s TotalCommitTime 1m22.7684107s 2016-04-07T01:54:53.776-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 17372007, 17372007, 135 committed:true 2016-04-07T01:54:53.776-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 17372007, 17372007, 135 committed:true) SnapCreateDur 685.974398ms SnapOpenDur 54.609µs 2016-04-07T01:54:53.778-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:53.78-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:53.782-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:53.783-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:54.129-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17380000 QueuedCount 50000 2016-04-07T01:54:54.639-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17390000 QueuedCount 50000 2016-04-07T01:54:55.034-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:55.163-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17400000 QueuedCount 50000 2016-04-07T01:54:55.677-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17410000 QueuedCount 50000 2016-04-07T01:54:56.068-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:56.178-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17420000 QueuedCount 47632 2016-04-07T01:54:56.687-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17430000 QueuedCount 37632 2016-04-07T01:54:57.2-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17440000 QueuedCount 27632 2016-04-07T01:54:57.732-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17450000 QueuedCount 17632 2016-04-07T01:54:58.187-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17460000 QueuedCount 7632 2016-04-07T01:54:59.514-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 31.223µs CommitTime 893.223372ms TotalFlushTime 12m46.58299651s TotalCommitTime 1m23.661634072s 2016-04-07T01:54:59.514-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 17467632, 17467632, 136 committed:true 2016-04-07T01:54:59.514-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 17467632, 17467632, 136 committed:true) SnapCreateDur 901.749997ms SnapOpenDur 53.506µs 2016-04-07T01:54:59.515-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:59.517-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:59.637-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17470000 QueuedCount 50000 2016-04-07T01:54:59.95-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:54:59.959-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:00.154-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17480000 QueuedCount 50000 2016-04-07T01:55:00.16-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:00.362-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:00.593-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17490000 QueuedCount 50000 2016-04-07T01:55:01.125-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17500000 QueuedCount 50000 2016-04-07T01:55:01.675-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17510000 QueuedCount 50000 2016-04-07T01:55:02.233-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17520000 QueuedCount 50000 2016-04-07T01:55:02.359-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:02.505-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:55:02.518-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:55:02.518-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:55:02.788-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17530000 QueuedCount 45543 2016-04-07T01:55:03.284-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17540000 QueuedCount 35543 2016-04-07T01:55:03.794-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17550000 QueuedCount 25543 2016-04-07T01:55:04.315-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17560000 QueuedCount 15543 2016-04-07T01:55:04.75-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17570000 QueuedCount 5543 2016-04-07T01:55:05.849-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.997µs CommitTime 781.359634ms TotalFlushTime 12m52.034965206s TotalCommitTime 1m24.442993706s 2016-04-07T01:55:05.849-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 17575543, 17575543, 137 committed:true 2016-04-07T01:55:05.849-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 17575543, 17575543, 137 committed:true) SnapCreateDur 789.667886ms SnapOpenDur 45.411µs 2016-04-07T01:55:05.85-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:05.857-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:05.874-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:05.876-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:05.996-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:06.071-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17580000 QueuedCount 50000 2016-04-07T01:55:06.18-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:06.575-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17590000 QueuedCount 50000 2016-04-07T01:55:07.012-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17600000 QueuedCount 50000 2016-04-07T01:55:07.527-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17610000 QueuedCount 50000 2016-04-07T01:55:08.066-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17620000 QueuedCount 50000 2016-04-07T01:55:08.609-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17630000 QueuedCount 50000 2016-04-07T01:55:08.928-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:09.153-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17640000 QueuedCount 48630 2016-04-07T01:55:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:55:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T01:55:09.374-07:00 [Info] client load stats {} 2016-04-07T01:55:09.643-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":2311370,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":5861305220,"bucket-1:myindex:avg_ts_items_count":99601,"bucket-1:myindex:build_progress":88,"bucket-1:myindex:data_size":4713182697,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":12950466560,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":39342,"bucket-1:myindex:frag_percent":63,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":1848873452,"bucket-1:myindex:items_count":17575543,"bucket-1:myindex:num_commits":137,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":17649288,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":17632209,"bucket-1:myindex:num_docs_queued":2367791,"bucket-1:myindex:num_flush_queued":17688630,"bucket-1:myindex:num_items_flushed":17649288,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":137,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":6335206752,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"137 84442993706 -5766766510944644268","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"17649289 105526937777 631609381214433655","bucket-1:myindex:timings/storage_info":"549 4581933 70260077337","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"136 47028915 18029985817771","bucket-1:myindex:timings/storage_meta_set":"137 46684195 17312697103207","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"137 5835449 272174318387","bucket-1:myindex:timings/storage_set":"35298577 660647214207 1830753137056576095","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":20000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"178 213640656 2163153363571614","bucket-1:ts_queue_size":55,"indexer_state":"Active","memory_quota":31457280000,"memory_used":15461984256,"memory_used_storage":12950863872,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"2907 489421487 213311926133919","uptime":"26m1.337029919s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T01:55:09.655-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17650000 QueuedCount 38630 2016-04-07T01:55:09.677-07:00 [Info] memstats {"Alloc":1279761720, "TotalAlloc":18271276704, "Sys":9854296392, "Lookups":6614, "Mallocs":503578275,"Frees":470682094, "HeapAlloc":1279761720, "HeapSys":9384329216, "HeapIdle":7170654208, "HeapInuse":2213675008,"HeapReleased":2325905408, "HeapObjects":32896181,"GCSys":297445376, "LastGC":1460019245237204721,"PauseTotalNs":29804527, "PauseNs":[1576039], "NumGC":23} 2016-04-07T01:55:09.682-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:4725052809, Disk:12950466560, ExtraSnapshotData:0, Fragmentation:63.51442021715085%, mode : circular 2016-04-07T01:55:09.682-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:55:10.151-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17660000 QueuedCount 28630 2016-04-07T01:55:10.656-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17670000 QueuedCount 18630 2016-04-07T01:55:11.098-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17680000 QueuedCount 8630 2016-04-07T01:55:12.362-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 22.274µs CommitTime 755.996995ms TotalFlushTime 12m57.692285156s TotalCommitTime 1m25.198990701s 2016-04-07T01:55:12.362-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 17688630, 17688630, 138 committed:true 2016-04-07T01:55:12.363-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 17688630, 17688630, 138 committed:true) SnapCreateDur 763.952738ms SnapOpenDur 48.989µs 2016-04-07T01:55:12.393-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17690000 QueuedCount 50000 2016-04-07T01:55:12.481-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:12.543-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:55:12.557-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:55:12.557-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:55:12.882-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17700000 QueuedCount 50000 2016-04-07T01:55:13.373-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17710000 QueuedCount 50000 2016-04-07T01:55:13.48-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:13.685-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:13.882-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17720000 QueuedCount 50000 2016-04-07T01:55:14.001-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:14.394-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17730000 QueuedCount 47841 2016-04-07T01:55:14.86-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17740000 QueuedCount 37841 2016-04-07T01:55:15.251-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17750000 QueuedCount 27841 2016-04-07T01:55:15.728-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17760000 QueuedCount 17841 2016-04-07T01:55:16.237-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17770000 QueuedCount 7841 2016-04-07T01:55:17.293-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 22.327µs CommitTime 653.919709ms TotalFlushTime 13m1.885809272s TotalCommitTime 1m25.85291041s 2016-04-07T01:55:17.293-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 17777841, 17777841, 139 committed:true 2016-04-07T01:55:17.293-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 17777841, 17777841, 139 committed:true) SnapCreateDur 662.353824ms SnapOpenDur 51.68µs 2016-04-07T01:55:17.299-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:17.306-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:17.309-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:17.311-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:17.314-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:17.408-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17780000 QueuedCount 50000 2016-04-07T01:55:17.826-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17790000 QueuedCount 50000 2016-04-07T01:55:18.32-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17800000 QueuedCount 50000 2016-04-07T01:55:18.824-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17810000 QueuedCount 50000 2016-04-07T01:55:19.366-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17820000 QueuedCount 50000 2016-04-07T01:55:19.903-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17830000 QueuedCount 50000 2016-04-07T01:55:20.459-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17840000 QueuedCount 50000 2016-04-07T01:55:21.013-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17850000 QueuedCount 50000 2016-04-07T01:55:21.569-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17860000 QueuedCount 50000 2016-04-07T01:55:22.038-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17870000 QueuedCount 50000 2016-04-07T01:55:22.359-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:22.541-07:00 [Info] Indexer::ReadMemstats Time Taken 12.422732ms 2016-04-07T01:55:22.58-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:55:22.593-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:55:22.593-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:55:22.623-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17880000 QueuedCount 50000 2016-04-07T01:55:23.208-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17890000 QueuedCount 50000 2016-04-07T01:55:23.797-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17900000 QueuedCount 50000 2016-04-07T01:55:24.403-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17910000 QueuedCount 50000 2016-04-07T01:55:25.012-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17920000 QueuedCount 50000 2016-04-07T01:55:25.616-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17930000 QueuedCount 50000 2016-04-07T01:55:26.239-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17940000 QueuedCount 50000 2016-04-07T01:55:26.844-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17950000 QueuedCount 50000 2016-04-07T01:55:27.367-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17960000 QueuedCount 50000 2016-04-07T01:55:27.99-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17970000 QueuedCount 50000 2016-04-07T01:55:28.602-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17980000 QueuedCount 50000 2016-04-07T01:55:29.219-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 17990000 QueuedCount 50000 2016-04-07T01:55:29.838-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18000000 QueuedCount 50000 2016-04-07T01:55:30.463-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18010000 QueuedCount 50000 2016-04-07T01:55:31.099-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18020000 QueuedCount 50000 2016-04-07T01:55:31.72-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18030000 QueuedCount 50000 2016-04-07T01:55:32.266-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18040000 QueuedCount 50000 2016-04-07T01:55:32.918-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18050000 QueuedCount 50000 2016-04-07T01:55:33.545-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18060000 QueuedCount 50000 2016-04-07T01:55:34.159-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18070000 QueuedCount 43601 2016-04-07T01:55:34.729-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18080000 QueuedCount 33601 2016-04-07T01:55:35.292-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18090000 QueuedCount 23601 2016-04-07T01:55:35.841-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18100000 QueuedCount 13601 2016-04-07T01:55:36.394-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18110000 QueuedCount 3601 2016-04-07T01:55:37.634-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.459µs CommitTime 1.103111014s TotalFlushTime 13m20.811356518s TotalCommitTime 1m26.956021424s 2016-04-07T01:55:37.634-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 18113601, 18113601, 140 committed:true 2016-04-07T01:55:37.634-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 18113601, 18113601, 140 committed:true) SnapCreateDur 1.111140015s SnapOpenDur 45.977µs 2016-04-07T01:55:37.944-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18120000 QueuedCount 50000 2016-04-07T01:55:38.454-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18130000 QueuedCount 50000 2016-04-07T01:55:38.899-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18140000 QueuedCount 50000 2016-04-07T01:55:39.413-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18150000 QueuedCount 50000 2016-04-07T01:55:39.683-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:4858250837, Disk:12950466560, ExtraSnapshotData:0, Fragmentation:62.485901071659924%, mode : circular 2016-04-07T01:55:39.683-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:55:39.957-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18160000 QueuedCount 50000 2016-04-07T01:55:40.512-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18170000 QueuedCount 50000 2016-04-07T01:55:41.067-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18180000 QueuedCount 50000 2016-04-07T01:55:41.62-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18190000 QueuedCount 50000 2016-04-07T01:55:42.187-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18200000 QueuedCount 50000 2016-04-07T01:55:42.774-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18210000 QueuedCount 50000 2016-04-07T01:55:43.274-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18220000 QueuedCount 50000 2016-04-07T01:55:43.853-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18230000 QueuedCount 50000 2016-04-07T01:55:44.454-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18240000 QueuedCount 50000 2016-04-07T01:55:45.055-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18250000 QueuedCount 50000 2016-04-07T01:55:45.653-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18260000 QueuedCount 49999 2016-04-07T01:55:46.216-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18270000 QueuedCount 40397 2016-04-07T01:55:46.754-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18280000 QueuedCount 30397 2016-04-07T01:55:47.273-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18290000 QueuedCount 20397 2016-04-07T01:55:47.67-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:55:47.683-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:55:47.683-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:55:47.735-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18300000 QueuedCount 10397 2016-04-07T01:55:48.268-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18310000 QueuedCount 397 2016-04-07T01:55:49.173-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.534µs CommitTime 805.294606ms TotalFlushTime 13m31.349085042s TotalCommitTime 1m27.76131603s 2016-04-07T01:55:49.173-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 18310397, 18310397, 141 committed:true 2016-04-07T01:55:49.173-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 18310397, 18310397, 141 committed:true) SnapCreateDur 813.875974ms SnapOpenDur 52.372µs 2016-04-07T01:55:49.587-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18320000 QueuedCount 21073 2016-04-07T01:55:50.037-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18330000 QueuedCount 11073 2016-04-07T01:55:50.499-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18340000 QueuedCount 1073 2016-04-07T01:55:50.974-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.286µs CommitTime 447.53594ms TotalFlushTime 13m32.669659083s TotalCommitTime 1m28.20885197s 2016-04-07T01:55:50.975-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 18341073, 18341073, 142 committed:true 2016-04-07T01:55:50.975-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 18341073, 18341073, 142 committed:true) SnapCreateDur 456.739465ms SnapOpenDur 55.782µs 2016-04-07T01:55:50.98-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:51.394-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18350000 QueuedCount 50000 2016-04-07T01:55:51.886-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18360000 QueuedCount 50000 2016-04-07T01:55:52.404-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18370000 QueuedCount 50000 2016-04-07T01:55:52.935-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18380000 QueuedCount 50000 2016-04-07T01:55:53.381-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18390000 QueuedCount 50000 2016-04-07T01:55:53.914-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18400000 QueuedCount 50000 2016-04-07T01:55:54.473-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18410000 QueuedCount 50000 2016-04-07T01:55:55.06-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18420000 QueuedCount 50000 2016-04-07T01:55:55.647-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18430000 QueuedCount 50000 2016-04-07T01:55:56.245-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18440000 QueuedCount 50000 2016-04-07T01:55:56.846-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18450000 QueuedCount 50000 2016-04-07T01:55:57.433-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18460000 QueuedCount 50000 2016-04-07T01:55:57.709-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:55:57.723-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:55:57.723-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:55:57.946-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18470000 QueuedCount 50000 2016-04-07T01:55:58.306-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:55:58.563-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18480000 QueuedCount 50000 2016-04-07T01:55:59.15-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18490000 QueuedCount 50000 2016-04-07T01:55:59.751-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18500000 QueuedCount 50000 2016-04-07T01:56:00.346-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18510000 QueuedCount 50000 2016-04-07T01:56:00.937-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18520000 QueuedCount 50000 2016-04-07T01:56:01.535-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18530000 QueuedCount 50000 2016-04-07T01:56:02.157-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18540000 QueuedCount 50000 2016-04-07T01:56:02.689-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18550000 QueuedCount 50000 2016-04-07T01:56:03.312-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18560000 QueuedCount 50000 2016-04-07T01:56:03.927-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18570000 QueuedCount 50000 2016-04-07T01:56:04.554-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18580000 QueuedCount 50000 2016-04-07T01:56:05.168-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18590000 QueuedCount 50000 2016-04-07T01:56:06.075-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18600000 QueuedCount 50000 2016-04-07T01:56:06.722-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18610000 QueuedCount 50000 2016-04-07T01:56:07.357-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18620000 QueuedCount 50000 2016-04-07T01:56:07.75-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:56:07.763-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:56:07.763-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:56:07.979-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18630000 QueuedCount 50000 2016-04-07T01:56:08.502-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18640000 QueuedCount 50000 2016-04-07T01:56:09.119-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18650000 QueuedCount 46702 2016-04-07T01:56:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:56:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T01:56:09.374-07:00 [Info] client load stats {} 2016-04-07T01:56:09.645-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":1303298,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":7023356207,"bucket-1:myindex:avg_ts_items_count":118729,"bucket-1:myindex:build_progress":92,"bucket-1:myindex:data_size":4984373716,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":12950466560,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":38112,"bucket-1:myindex:frag_percent":61,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":1955710470,"bucket-1:myindex:items_count":18341073,"bucket-1:myindex:num_commits":142,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":18658590,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":18525595,"bucket-1:myindex:num_docs_queued":1474405,"bucket-1:myindex:num_flush_queued":18696702,"bucket-1:myindex:num_items_flushed":18658590,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":142,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":1802091483,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"142 88208851970 -2701982339424482530","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"18658591 112299739712 696653020037908332","bucket-1:myindex:timings/storage_info":"569 4729506 72404192306","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"141 48618761 18631502123719","bucket-1:myindex:timings/storage_meta_set":"142 48222395 17850456681733","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"142 5997986 277568613110","bucket-1:myindex:timings/storage_set":"37317180 708622287450 5526869543269414876","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":20000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"190 223448773 2171180699175961","bucket-1:ts_queue_size":38,"indexer_state":"Active","memory_quota":31457280000,"memory_used":15784982528,"memory_used_storage":13131239424,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"3049 519062151 220317920333463","uptime":"27m1.337430237s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T01:56:09.667-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18660000 QueuedCount 36702 2016-04-07T01:56:09.678-07:00 [Info] memstats {"Alloc":1558747920, "TotalAlloc":18550262904, "Sys":9854296392, "Lookups":7046, "Mallocs":513710070,"Frees":473344628, "HeapAlloc":1558747920, "HeapSys":9383641088, "HeapIdle":7027343360, "HeapInuse":2356297728,"HeapReleased":2325651456, "HeapObjects":40365442,"GCSys":297445376, "LastGC":1460019245237204721,"PauseTotalNs":29804527, "PauseNs":[], "NumGC":23} 2016-04-07T01:56:09.683-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:4993845129, Disk:12950466560, ExtraSnapshotData:0, Fragmentation:61.43887862369029%, mode : circular 2016-04-07T01:56:09.684-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:56:10.242-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18670000 QueuedCount 26702 2016-04-07T01:56:10.797-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18680000 QueuedCount 16702 2016-04-07T01:56:11.352-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18690000 QueuedCount 6702 2016-04-07T01:56:12.719-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 32.596µs CommitTime 1.014061664s TotalFlushTime 13m53.077090782s TotalCommitTime 1m29.222913634s 2016-04-07T01:56:12.719-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 18696702, 18696702, 143 committed:true 2016-04-07T01:56:12.719-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 18696702, 18696702, 143 committed:true) SnapCreateDur 1.022199115s SnapOpenDur 83.258µs 2016-04-07T01:56:12.867-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18700000 QueuedCount 50000 2016-04-07T01:56:13.396-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18710000 QueuedCount 50000 2016-04-07T01:56:13.919-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18720000 QueuedCount 50000 2016-04-07T01:56:14.465-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18730000 QueuedCount 50000 2016-04-07T01:56:15.019-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18740000 QueuedCount 50000 2016-04-07T01:56:15.58-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18750000 QueuedCount 50000 2016-04-07T01:56:16.077-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18760000 QueuedCount 50000 2016-04-07T01:56:16.661-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18770000 QueuedCount 50000 2016-04-07T01:56:17.26-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18780000 QueuedCount 50000 2016-04-07T01:56:17.836-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18790000 QueuedCount 50000 2016-04-07T01:56:18.428-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18800000 QueuedCount 50000 2016-04-07T01:56:19.02-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18810000 QueuedCount 50000 2016-04-07T01:56:19.621-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18820000 QueuedCount 50000 2016-04-07T01:56:20.223-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18830000 QueuedCount 45543 2016-04-07T01:56:20.692-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18840000 QueuedCount 35543 2016-04-07T01:56:21.255-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18850000 QueuedCount 25543 2016-04-07T01:56:21.81-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18860000 QueuedCount 15543 2016-04-07T01:56:22.379-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18870000 QueuedCount 5543 2016-04-07T01:56:22.763-07:00 [Info] Indexer::ReadMemstats Time Taken 11.485568ms 2016-04-07T01:56:22.803-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:56:22.817-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:56:22.817-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:56:23.755-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 38.611µs CommitTime 1.03356521s TotalFlushTime 14m2.915111644s TotalCommitTime 1m30.256478844s 2016-04-07T01:56:23.755-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 18875543, 18875543, 144 committed:true 2016-04-07T01:56:23.755-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 18875543, 18875543, 144 committed:true) SnapCreateDur 1.041708151s SnapOpenDur 70.76µs 2016-04-07T01:56:24.018-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18880000 QueuedCount 50000 2016-04-07T01:56:24.591-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18890000 QueuedCount 50000 2016-04-07T01:56:25.052-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18900000 QueuedCount 50000 2016-04-07T01:56:25.599-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18910000 QueuedCount 50000 2016-04-07T01:56:26.154-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18920000 QueuedCount 50000 2016-04-07T01:56:26.735-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18930000 QueuedCount 50000 2016-04-07T01:56:27.319-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18940000 QueuedCount 50000 2016-04-07T01:56:27.902-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18950000 QueuedCount 50000 2016-04-07T01:56:28.496-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18960000 QueuedCount 50000 2016-04-07T01:56:29.113-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18970000 QueuedCount 50000 2016-04-07T01:56:29.638-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18980000 QueuedCount 50000 2016-04-07T01:56:30.255-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 18990000 QueuedCount 50000 2016-04-07T01:56:30.885-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19000000 QueuedCount 50000 2016-04-07T01:56:31.506-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19010000 QueuedCount 50000 2016-04-07T01:56:32.128-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19020000 QueuedCount 50000 2016-04-07T01:56:32.738-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19030000 QueuedCount 47986 2016-04-07T01:56:32.838-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:56:32.851-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:56:32.851-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:56:33.298-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19040000 QueuedCount 37986 2016-04-07T01:56:33.859-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19050000 QueuedCount 27986 2016-04-07T01:56:34.396-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19060000 QueuedCount 17986 2016-04-07T01:56:34.866-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19070000 QueuedCount 7986 2016-04-07T01:56:36.525-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.336µs CommitTime 1.221680517s TotalFlushTime 14m14.269963771s TotalCommitTime 1m31.478159361s 2016-04-07T01:56:36.526-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 19077986, 19077986, 145 committed:true 2016-04-07T01:56:36.526-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 19077986, 19077986, 145 committed:true) SnapCreateDur 1.231822074s SnapOpenDur 60.715µs 2016-04-07T01:56:36.562-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19080000 QueuedCount 21452 2016-04-07T01:56:37.072-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19090000 QueuedCount 11452 2016-04-07T01:56:37.576-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19100000 QueuedCount 1452 2016-04-07T01:56:38.082-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.03µs CommitTime 395.424512ms TotalFlushTime 14m15.396780217s TotalCommitTime 1m31.873583873s 2016-04-07T01:56:38.082-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 19101452, 19101452, 146 committed:true 2016-04-07T01:56:38.082-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 19101452, 19101452, 146 committed:true) SnapCreateDur 403.534857ms SnapOpenDur 66.261µs 2016-04-07T01:56:38.099-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:56:38.101-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:56:38.104-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:56:38.109-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:56:38.247-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:56:38.49-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:56:38.616-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19110000 QueuedCount 50000 2016-04-07T01:56:39.166-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19120000 QueuedCount 50000 2016-04-07T01:56:39.633-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19130000 QueuedCount 50000 2016-04-07T01:56:39.685-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5115386077, Disk:14210519040, ExtraSnapshotData:0, Fragmentation:64.00282028685139%, mode : circular 2016-04-07T01:56:39.685-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:56:40.2-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19140000 QueuedCount 50000 2016-04-07T01:56:40.776-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19150000 QueuedCount 50000 2016-04-07T01:56:41.337-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19160000 QueuedCount 50000 2016-04-07T01:56:41.923-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19170000 QueuedCount 50000 2016-04-07T01:56:42.507-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19180000 QueuedCount 50000 2016-04-07T01:56:42.876-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:56:42.889-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:56:42.889-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:56:43.116-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19190000 QueuedCount 50000 2016-04-07T01:56:43.709-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19200000 QueuedCount 50000 2016-04-07T01:56:44.217-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19210000 QueuedCount 50000 2016-04-07T01:56:44.83-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19220000 QueuedCount 50000 2016-04-07T01:56:45.464-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19230000 QueuedCount 50000 2016-04-07T01:56:46.093-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19240000 QueuedCount 50000 2016-04-07T01:56:46.689-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19250000 QueuedCount 50000 2016-04-07T01:56:47.302-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19260000 QueuedCount 50000 2016-04-07T01:56:47.924-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19270000 QueuedCount 50000 2016-04-07T01:56:48.304-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:56:48.545-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19280000 QueuedCount 50000 2016-04-07T01:56:49.181-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19290000 QueuedCount 50000 2016-04-07T01:56:49.725-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19300000 QueuedCount 50000 2016-04-07T01:56:50.36-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19310000 QueuedCount 50000 2016-04-07T01:56:51.005-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19320000 QueuedCount 50000 2016-04-07T01:56:51.628-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19330000 QueuedCount 50000 2016-04-07T01:56:52.272-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19340000 QueuedCount 50000 2016-04-07T01:56:52.916-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19350000 QueuedCount 50000 2016-04-07T01:56:53.557-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19360000 QueuedCount 50000 2016-04-07T01:56:54.201-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19370000 QueuedCount 50000 2016-04-07T01:56:54.742-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19380000 QueuedCount 50000 2016-04-07T01:56:55.379-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19390000 QueuedCount 50000 2016-04-07T01:56:56.022-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19400000 QueuedCount 50000 2016-04-07T01:56:56.673-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19410000 QueuedCount 50000 2016-04-07T01:56:57.333-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19420000 QueuedCount 50000 2016-04-07T01:56:57.93-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:56:57.945-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:56:57.945-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:56:57.976-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19430000 QueuedCount 50000 2016-04-07T01:56:58.65-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19440000 QueuedCount 50000 2016-04-07T01:56:59.298-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19450000 QueuedCount 50000 2016-04-07T01:56:59.913-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19460000 QueuedCount 50000 2016-04-07T01:57:00.44-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19470000 QueuedCount 41975 2016-04-07T01:57:01.033-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19480000 QueuedCount 31975 2016-04-07T01:57:01.612-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19490000 QueuedCount 21975 2016-04-07T01:57:02.188-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19500000 QueuedCount 11975 2016-04-07T01:57:02.768-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19510000 QueuedCount 1975 2016-04-07T01:57:04.022-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 14.568µs CommitTime 1.11991577s TotalFlushTime 14m39.838587653s TotalCommitTime 1m32.993499643s 2016-04-07T01:57:04.022-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 19511975, 19511975, 147 committed:true 2016-04-07T01:57:04.022-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 19511975, 19511975, 147 committed:true) SnapCreateDur 1.128919549s SnapOpenDur 63.838µs 2016-04-07T01:57:04.431-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19520000 QueuedCount 50000 2016-04-07T01:57:04.974-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19530000 QueuedCount 50000 2016-04-07T01:57:05.531-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19540000 QueuedCount 50000 2016-04-07T01:57:06.082-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19550000 QueuedCount 50000 2016-04-07T01:57:06.669-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19560000 QueuedCount 50000 2016-04-07T01:57:07.254-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19570000 QueuedCount 50000 2016-04-07T01:57:07.846-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19580000 QueuedCount 50000 2016-04-07T01:57:08.451-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19590000 QueuedCount 50000 2016-04-07T01:57:08.956-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19600000 QueuedCount 50000 2016-04-07T01:57:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:57:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T01:57:09.374-07:00 [Info] client load stats {} 2016-04-07T01:57:09.556-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19610000 QueuedCount 45892 2016-04-07T01:57:09.646-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":344108,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":16543940210,"bucket-1:myindex:avg_ts_items_count":262434,"bucket-1:myindex:build_progress":97,"bucket-1:myindex:data_size":5231897370,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":14750490624,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":45590,"bucket-1:myindex:frag_percent":64,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":2056070477,"bucket-1:myindex:items_count":19511975,"bucket-1:myindex:num_commits":147,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":19610302,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":19511975,"bucket-1:myindex:num_docs_queued":488025,"bucket-1:myindex:num_flush_queued":19655892,"bucket-1:myindex:num_items_flushed":19610302,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":147,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":25939544705,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"147 92993499643 2297670924494828799","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"19610303 118525901722 696696492986096336","bucket-1:myindex:timings/storage_info":"589 4876298 74185712660","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"146 50065179 19093684970821","bucket-1:myindex:timings/storage_meta_set":"147 50027880 18523834920040","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"147 6226793 288188555227","bucket-1:myindex:timings/storage_set":"39220605 756159805263 -9035043798081345407","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":20000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"202 237537448 2203248528149894","bucket-1:ts_queue_size":26,"indexer_state":"Active","memory_quota":31457280000,"memory_used":16601872384,"memory_used_storage":15052935168,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"3192 548266539 226455870966743","uptime":"28m1.338986512s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T01:57:09.678-07:00 [Info] memstats {"Alloc":669678632, "TotalAlloc":18800639984, "Sys":9854296392, "Lookups":7478, "Mallocs":523159944,"Frees":503500097, "HeapAlloc":669678632, "HeapSys":9385574400, "HeapIdle":8134082560, "HeapInuse":1251491840,"HeapReleased":2325356544, "HeapObjects":19659847,"GCSys":297445376, "LastGC":1460019365423333147,"PauseTotalNs":30723968, "PauseNs":[919441], "NumGC":24} 2016-04-07T01:57:09.685-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5240121487, Disk:14750490624, ExtraSnapshotData:0, Fragmentation:64.47493428812473%, mode : circular 2016-04-07T01:57:09.685-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:57:10.116-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19620000 QueuedCount 35892 2016-04-07T01:57:10.689-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19630000 QueuedCount 25892 2016-04-07T01:57:11.252-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19640000 QueuedCount 15892 2016-04-07T01:57:11.82-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19650000 QueuedCount 5892 2016-04-07T01:57:12.981-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:57:12.995-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:57:12.995-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:57:13.102-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 30.873µs CommitTime 933.892313ms TotalFlushTime 14m47.835301427s TotalCommitTime 1m33.927391956s 2016-04-07T01:57:13.102-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 19655892, 19655892, 148 committed:true 2016-04-07T01:57:13.102-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 19655892, 19655892, 148 committed:true) SnapCreateDur 942.160795ms SnapOpenDur 82.793µs 2016-04-07T01:57:13.321-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19660000 QueuedCount 737 2016-04-07T01:57:13.461-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.213µs CommitTime 116.90814ms TotalFlushTime 14m48.059600951s TotalCommitTime 1m34.044300096s 2016-04-07T01:57:13.461-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 19660737, 19660737, 149 committed:true 2016-04-07T01:57:13.461-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 19660737, 19660737, 149 committed:true) SnapCreateDur 125.949347ms SnapOpenDur 85.715µs 2016-04-07T01:57:13.463-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:13.464-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:13.465-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:13.467-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:13.469-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:13.473-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:13.476-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:13.479-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:13.48-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:13.955-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19670000 QueuedCount 50000 2016-04-07T01:57:14.506-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19680000 QueuedCount 50000 2016-04-07T01:57:15.047-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19690000 QueuedCount 50000 2016-04-07T01:57:15.615-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19700000 QueuedCount 50000 2016-04-07T01:57:16.19-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19710000 QueuedCount 50000 2016-04-07T01:57:16.67-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19720000 QueuedCount 50000 2016-04-07T01:57:17.255-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19730000 QueuedCount 50000 2016-04-07T01:57:17.584-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:17.836-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19740000 QueuedCount 50000 2016-04-07T01:57:18.441-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19750000 QueuedCount 50000 2016-04-07T01:57:19.038-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19760000 QueuedCount 50000 2016-04-07T01:57:19.639-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19770000 QueuedCount 50000 2016-04-07T01:57:20.247-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19780000 QueuedCount 50000 2016-04-07T01:57:20.873-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19790000 QueuedCount 50000 2016-04-07T01:57:21.4-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19800000 QueuedCount 50000 2016-04-07T01:57:22.043-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19810000 QueuedCount 50000 2016-04-07T01:57:22.673-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19820000 QueuedCount 50000 2016-04-07T01:57:22.977-07:00 [Info] Indexer::ReadMemstats Time Taken 11.617903ms 2016-04-07T01:57:23.017-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:57:23.031-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:57:23.031-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:57:23.296-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19830000 QueuedCount 50000 2016-04-07T01:57:23.944-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19840000 QueuedCount 50000 2016-04-07T01:57:24.19-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:24.569-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19850000 QueuedCount 47119 2016-04-07T01:57:25.143-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19860000 QueuedCount 37119 2016-04-07T01:57:25.714-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19870000 QueuedCount 27119 2016-04-07T01:57:26.281-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19880000 QueuedCount 17119 2016-04-07T01:57:26.78-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19890000 QueuedCount 7119 2016-04-07T01:57:31.593-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 40.88µs CommitTime 4.349193525s TotalFlushTime 15m1.632236322s TotalCommitTime 1m38.393493621s 2016-04-07T01:57:31.593-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 19897119, 19897119, 150 committed:true 2016-04-07T01:57:31.594-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 19897119, 19897119, 150 committed:true) SnapCreateDur 4.357683554s SnapOpenDur 74.375µs 2016-04-07T01:57:31.667-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 16.839µs CommitTime 43.797485ms TotalFlushTime 15m1.64917941s TotalCommitTime 1m38.437291106s 2016-04-07T01:57:31.667-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 19898553, 19898553, 151 committed:true 2016-04-07T01:57:31.667-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 19898553, 19898553, 151 committed:true) SnapCreateDur 52.285098ms SnapOpenDur 61.979µs 2016-04-07T01:57:31.668-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:31.67-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:31.671-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:31.671-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:31.672-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:31.677-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:31.678-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:31.679-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:31.68-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:31.682-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:31.682-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T01:57:31.701-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19900000 QueuedCount 50000 2016-04-07T01:57:32.201-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19910000 QueuedCount 50000 2016-04-07T01:57:32.734-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19920000 QueuedCount 50000 2016-04-07T01:57:33.055-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:57:33.069-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:57:33.069-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:57:33.288-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19930000 QueuedCount 50000 2016-04-07T01:57:33.843-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19940000 QueuedCount 50000 2016-04-07T01:57:34.396-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19950000 QueuedCount 49989 2016-04-07T01:57:34.852-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19960000 QueuedCount 40000 2016-04-07T01:57:35.383-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19970000 QueuedCount 30000 2016-04-07T01:57:35.92-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19980000 QueuedCount 20000 2016-04-07T01:57:36.457-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 19990000 QueuedCount 10000 2016-04-07T01:57:37.001-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20000000 QueuedCount 0 2016-04-07T01:57:37.868-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 23.202µs CommitTime 852.959081ms TotalFlushTime 15m6.90422742s TotalCommitTime 1m39.290250187s 2016-04-07T01:57:37.868-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20000000, 20000000, 152 committed:true 2016-04-07T01:57:37.868-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20000000, 20000000, 152 committed:true) SnapCreateDur 861.370158ms SnapOpenDur 45.986µs 2016-04-07T01:57:37.868-07:00 [Info] Timekeeper::checkInitialBuildDone remove index 8343618505678776570 from stream MAINT_STREAM bucket bucket-1 2016-04-07T01:57:37.868-07:00 [Info] Timekeeper::checkInitialBuildDone Initial Build Done Index: 8343618505678776570 Stream: MAINT_STREAM Bucket: bucket-1 BuildTS: [0=19535,1=19525,2=19535,3=19530,4=19535,5=19530,6=19535,7=19525,8=19535,9=19525,10=19535,11=19530,12=19535,13=19530,14=19535,15=19525,16=19535,17=19525,18=19536,19=19530,20=19535,21=19530,22=19535,23=19525,24=19535,25=19525,26=19535,27=19530,28=19535,29=19530,30=19535,31=19525,32=19535,33=19525,34=19535,35=19530,36=19535,37=19530,38=19535,39=19525,40=19535,41=19525,42=19535,43=19530,44=19535,45=19530,46=19535,47=19525,48=19535,49=19525,50=19535,51=19530,52=19535,53=19530,54=19535,55=19525,56=19535,57=19525,58=19535,59=19530,60=19535,61=19530,62=19535,63=19525,64=19530,65=19535,66=19525,67=19535,68=19525,69=19535,70=19530,71=19535,72=19530,73=19535,74=19525,75=19535,76=19525,77=19535,78=19530,79=19535,80=19530,81=19535,82=19525,83=19535,84=19525,85=19535,86=19530,87=19535,88=19530,89=19535,90=19525,91=19535,92=19525,93=19535,94=19530,95=19535,96=19530,97=19535,98=19525,99=19535,100=19525,101=19535,102=19530,103=19535,104=19530,105=19535,106=19525,107=19535,108=19525,109=19535,110=19530,111=19535,112=19530,113=19535,114=19525,115=19535,116=19525,117=19535,118=19530,119=19535,120=19530,121=19535,122=19525,123=19535,124=19525,125=19535,126=19530,127=19535,128=19535,129=19530,130=19535,131=19525,132=19535,133=19525,134=19535,135=19530,136=19535,137=19530,138=19535,139=19525,140=19535,141=19525,142=19535,143=19530,144=19535,145=19530,146=19535,147=19525,148=19535,149=19525,150=19535,151=19530,152=19535,153=19530,154=19535,155=19525,156=19535,157=19525,158=19535,159=19530,160=19535,161=19530,162=19535,163=19525,164=19535,165=19525,166=19535,167=19530,168=19535,169=19530,170=19535,171=19525,172=19535,173=19525,174=19535,175=19530,176=19535,177=19530,178=19535,179=19525,180=19535,181=19525,182=19535,183=19530,184=19535,185=19530,186=19535,187=19525,188=19535,189=19525,190=19535,191=19530,192=19525,193=19535,194=19530,195=19535,196=19530,197=19535,198=19525,199=19535,200=19525,201=19535,202=19530,203=19535,204=19530,205=19535,206=19525,207=19535,208=19525,209=19535,210=19530,211=19535,212=19530,213=19535,214=19525,215=19535,216=19525,217=19535,218=19530,219=19535,220=19530,221=19535,222=19525,223=19535,224=19525,225=19535,226=19530,227=19535,228=19530,229=19535,230=19525,231=19535,232=19525,233=19535,234=19530,235=19535,236=19530,237=19535,238=19525,239=19535,240=19525,241=19535,242=19530,243=19535,244=19530,245=19535,246=19525,247=19535,248=19525,249=19535,250=19530,251=19535,252=19530,253=19535,254=19525,255=19535,256=19530,257=19535,258=19525,259=19535,260=19525,261=19535,262=19530,263=19535,264=19530,265=19535,266=19525,267=19535,268=19525,269=19535,270=19530,271=19535,272=19530,273=19535,274=19525,275=19535,276=19525,277=19535,278=19530,279=19535,280=19530,281=19535,282=19525,283=19535,284=19525,285=19535,286=19530,287=19535,288=19530,289=19535,290=19525,291=19535,292=19525,293=19535,294=19530,295=19535,296=19530,297=19535,298=19525,299=19535,300=19525,301=19535,302=19530,303=19535,304=19530,305=19535,306=19525,307=19535,308=19525,309=19535,310=19530,311=19535,312=19530,313=19535,314=19525,315=19535,316=19525,317=19535,318=19530,319=19535,320=19535,321=19525,322=19535,323=19530,324=19535,325=19530,326=19535,327=19525,328=19535,329=19525,330=19535,331=19530,332=19535,333=19530,334=19535,335=19525,336=19535,337=19525,338=19535,339=19530,340=19535,341=19530,342=19535,343=19525,344=19535,345=19525,346=19535,347=19530,348=19535,349=19530,350=19535,351=19525,352=19535,353=19525,354=19535,355=19530,356=19535,357=19530,358=19535,359=19525,360=19535,361=19525,362=19535,363=19530,364=19535,365=19530,366=19535,367=19525,368=19535,369=19525,370=19535,371=19530,372=19535,373=19530,374=19535,375=19525,376=19535,377=19525,378=19535,379=19530,380=19535,381=19530,382=19535,383=19525,384=19525,385=19535,386=19530,387=19535,388=19530,389=19535,390=19525,391=19535,392=19525,393=19535,394=19530,395=19535,396=19530,397=19535,398=19525,399=19535,400=19525,401=19535,402=19530,403=19535,404=19530,405=19535,406=19525,407=19535,408=19525,409=19535,410=19530,411=19535,412=19530,413=19535,414=19525,415=19535,416=19525,417=19535,418=19530,419=19535,420=19530,421=19535,422=19525,423=19535,424=19525,425=19535,426=19530,427=19535,428=19530,429=19535,430=19525,431=19535,432=19525,433=19535,434=19530,435=19535,436=19530,437=19535,438=19525,439=19535,440=19525,441=19535,442=19530,443=19535,444=19530,445=19535,446=19525,447=19535,448=19535,449=19530,450=19535,451=19525,452=19535,453=19525,454=19535,455=19530,456=19535,457=19530,458=19535,459=19525,460=19535,461=19525,462=19535,463=19530,464=19535,465=19530,466=19535,467=19525,468=19535,469=19525,470=19535,471=19530,472=19535,473=19530,474=19535,475=19525,476=19535,477=19525,478=19535,479=19530,480=19535,481=19530,482=19535,483=19525,484=19535,485=19525,486=19535,487=19530,488=19535,489=19530,490=19535,491=19525,492=19535,493=19525,494=19535,495=19530,496=19535,497=19530,498=19535,499=19525,500=19535,501=19525,502=19535,503=19530,504=19535,505=19530,506=19535,507=19525,508=19535,509=19525,510=19535,511=19530,512=19535,513=19530,514=19535,515=19525,516=19535,517=19525,518=19535,519=19530,520=19535,521=19530,522=19535,523=19525,524=19535,525=19525,526=19535,527=19530,528=19535,529=19530,530=19535,531=19525,532=19535,533=19525,534=19535,535=19530,536=19535,537=19530,538=19535,539=19525,540=19535,541=19525,542=19535,543=19530,544=19535,545=19530,546=19535,547=19525,548=19535,549=19525,550=19535,551=19530,552=19535,553=19530,554=19535,555=19525,556=19535,557=19525,558=19535,559=19530,560=19535,561=19530,562=19535,563=19525,564=19535,565=19525,566=19535,567=19529,568=19535,569=19530,570=19535,571=19525,572=19535,573=19525,574=19535,575=19530,576=19525,577=19535,578=19530,579=19535,580=19530,581=19535,582=19525,583=19535,584=19525,585=19535,586=19530,587=19535,588=19530,589=19535,590=19525,591=19535,592=19525,593=19535,594=19530,595=19535,596=19530,597=19535,598=19525,599=19535,600=19525,601=19535,602=19530,603=19535,604=19530,605=19535,606=19525,607=19535,608=19525,609=19535,610=19530,611=19535,612=19530,613=19535,614=19525,615=19535,616=19525,617=19535,618=19530,619=19535,620=19530,621=19535,622=19525,623=19535,624=19525,625=19535,626=19530,627=19535,628=19530,629=19535,630=19525,631=19535,632=19525,633=19535,634=19530,635=19535,636=19530,637=19535,638=19525,639=19535,640=19535,641=19525,642=19535,643=19530,644=19535,645=19530,646=19535,647=19525,648=19535,649=19525,650=19535,651=19530,652=19535,653=19530,654=19535,655=19525,656=19535,657=19525,658=19535,659=19530,660=19535,661=19530,662=19535,663=19525,664=19535,665=19525,666=19535,667=19530,668=19535,669=19530,670=19535,671=19525,672=19535,673=19525,674=19535,675=19530,676=19535,677=19530,678=19535,679=19525,680=19535,681=19525,682=19535,683=19530,684=19535,685=19530,686=19535,687=19525,688=19535,689=19525,690=19535,691=19530,692=19535,693=19530,694=19535,695=19525,696=19535,697=19525,698=19535,699=19530,700=19535,701=19530,702=19535,703=19525,704=19530,705=19535,706=19525,707=19535,708=19525,709=19535,710=19530,711=19535,712=19530,713=19535,714=19525,715=19535,716=19525,717=19535,718=19530,719=19535,720=19530,721=19535,722=19525,723=19535,724=19525,725=19535,726=19530,727=19535,728=19530,729=19535,730=19525,731=19535,732=19525,733=19535,734=19530,735=19535,736=19530,737=19535,738=19525,739=19535,740=19525,741=19535,742=19530,743=19535,744=19530,745=19535,746=19525,747=19535,748=19525,749=19535,750=19530,751=19535,752=19530,753=19535,754=19525,755=19535,756=19525,757=19535,758=19530,759=19535,760=19530,761=19535,762=19525,763=19535,764=19525,765=19535,766=19530,767=19535,768=19525,769=19535,770=19530,771=19535,772=19530,773=19535,774=19525,775=19535,776=19525,777=19535,778=19530,779=19535,780=19530,781=19535,782=19525,783=19535,784=19525,785=19535,786=19530,787=19535,788=19530,789=19535,790=19525,791=19535,792=19525,793=19535,794=19530,795=19535,796=19530,797=19535,798=19525,799=19535,800=19525,801=19535,802=19530,803=19535,804=19530,805=19535,806=19525,807=19535,808=19525,809=19535,810=19530,811=19535,812=19530,813=195 35,814=19525,815=19535,816=19525,817=19535,818=19530,819=19535,820=19530,821=19535,822=19525,823=19535,824=19525,825=19535,826=19530,827=19535,828=19530,829=19535,830=19525,831=19535,832=19535,833=19530,834=19535,835=19525,836=19535,837=19525,838=19535,839=19530,840=19535,841=19530,842=19535,843=19525,844=19535,845=19525,846=19535,847=19530,848=19535,849=19530,850=19535,851=19525,852=19535,853=19525,854=19535,855=19530,856=19535,857=19530,858=19535,859=19525,860=19535,861=19525,862=19535,863=19530,864=19535,865=19530,866=19535,867=19525,868=19535,869=19525,870=19535,871=19530,872=19535,873=19530,874=19535,875=19525,876=19535,877=19525,878=19535,879=19530,880=19535,881=19530,882=19535,883=19525,884=19535,885=19525,886=19535,887=19530,888=19535,889=19530,890=19535,891=19525,892=19535,893=19525,894=19535,895=19530,896=19530,897=19535,898=19525,899=19535,900=19525,901=19535,902=19530,903=19535,904=19530,905=19535,906=19525,907=19535,908=19525,909=19535,910=19530,911=19535,912=19530,913=19535,914=19525,915=19535,916=19525,917=19535,918=19530,919=19535,920=19530,921=19535,922=19525,923=19535,924=19525,925=19535,926=19530,927=19535,928=19530,929=19535,930=19525,931=19535,932=19525,933=19535,934=19530,935=19535,936=19530,937=19535,938=19525,939=19535,940=19525,941=19535,942=19530,943=19535,944=19530,945=19535,946=19525,947=19535,948=19525,949=19535,950=19530,951=19535,952=19530,953=19535,954=19525,955=19535,956=19525,957=19535,958=19530,959=19535,960=19535,961=19525,962=19535,963=19530,964=19535,965=19530,966=19535,967=19525,968=19535,969=19525,970=19535,971=19530,972=19535,973=19530,974=19535,975=19525,976=19535,977=19525,978=19535,979=19530,980=19535,981=19530,982=19535,983=19525,984=19535,985=19525,986=19535,987=19530,988=19535,989=19530,990=19535,991=19525,992=19535,993=19525,994=19535,995=19530,996=19535,997=19530,998=19535,999=19525,1000=19535,1001=19525,1002=19535,1003=19530,1004=19535,1005=19530,1006=19535,1007=19525,1008=19535,1009=19525,1010=19535,1011=19530,1012=19535,1013=19530,1014=19535,1015=19525,1016=19535,1017=19525,1018=19535,1019=19530,1020=19535,1021=19530,1022=19535,1023=19525] 2016-04-07T01:57:37.871-07:00 [Info] Indexer::handleInitialBuildDone Bucket: bucket-1 Stream: MAINT_STREAM 2016-04-07T01:57:37.871-07:00 [Info] MutationMgr::handleUpdateIndexInstMap Message: MsgUpdateInstMap InstanceId: 8343618505678776570 Name: myindex Bucket: bucket-1 State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM 2016-04-07T01:57:37.871-07:00 [Info] ClustMgr:handleUpdateTopologyForIndex &{47 [{8343618505678776570 {8343618505678776570 myindex forestdb bucket-1 6311cb70dafcf851cc4e85775abf6ceb false [`alt_email`] N1QL SINGLE true false [52:a9:25:ef:79:3b:5c:c0] false} 4 1 0xc823805600 []}] {true true false false} 0} 2016-04-07T01:57:38.072-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:57:38.084-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:57:38.084-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:57:39.686-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5353304353, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:65.94266539023734%, mode : circular 2016-04-07T01:57:39.686-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:57:43.09-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2016-04-07T01:57:43.103-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. 2016-04-07T01:57:43.103-07:00 [Info] Refreshed Indexer List: [172.23.99.247:9100] 2016-04-07T01:58:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:58:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T01:58:09.374-07:00 [Info] client load stats {} 2016-04-07T01:58:09.646-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":0,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":6208593017,"bucket-1:myindex:avg_ts_items_count":93631,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":5353304353,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":15718506496,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":0,"bucket-1:myindex:frag_percent":65,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":2100000000,"bucket-1:myindex:items_count":20000000,"bucket-1:myindex:num_commits":152,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":20000000,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":20000000,"bucket-1:myindex:num_docs_queued":0,"bucket-1:myindex:num_flush_queued":20000000,"bucket-1:myindex:num_items_flushed":20000000,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":152,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":6200994051,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"152 99290250187 4381690947718642163","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"20000000 121030679724 696713583949301932","bucket-1:myindex:timings/storage_info":"609 5037061 76362671955","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"151 51862862 19765243313372","bucket-1:myindex:timings/storage_meta_set":"152 51794519 19181748915473","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"152 6495959 303582011715","bucket-1:myindex:timings/storage_set":"40000000 775093290168 -7529719260312036834","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":20000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"214 247264740 2211145103575020","bucket-1:ts_queue_size":0,"indexer_state":"Active","memory_quota":31457280000,"memory_used":17398835200,"memory_used_storage":15718924288,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"3299 568804529 230537813647409","uptime":"29m1.339715932s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T01:58:09.678-07:00 [Info] memstats {"Alloc":925987592, "TotalAlloc":19056948944, "Sys":9854296392, "Lookups":7911, "Mallocs":532541909,"Frees":505939171, "HeapAlloc":925987592, "HeapSys":9384460288, "HeapIdle":8001994752, "HeapInuse":1382465536,"HeapReleased":2879111168, "HeapObjects":26602738,"GCSys":297445376, "LastGC":1460019365423333147,"PauseTotalNs":30723968, "PauseNs":[], "NumGC":24} 2016-04-07T01:58:09.686-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5353304353, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:65.94266539023734%, mode : circular 2016-04-07T01:58:09.686-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:58:23.202-07:00 [Info] Indexer::ReadMemstats Time Taken 10.337451ms 2016-04-07T01:58:39.686-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5353304353, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:65.94266539023734%, mode : circular 2016-04-07T01:58:39.686-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:59:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T01:59:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T01:59:09.374-07:00 [Info] client load stats {} 2016-04-07T01:59:09.646-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":0,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":6208593017,"bucket-1:myindex:avg_ts_items_count":93631,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":5353304353,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":15718506496,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":0,"bucket-1:myindex:frag_percent":65,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":2100000000,"bucket-1:myindex:items_count":20000000,"bucket-1:myindex:num_commits":152,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":20000000,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":20000000,"bucket-1:myindex:num_docs_queued":0,"bucket-1:myindex:num_flush_queued":20000000,"bucket-1:myindex:num_items_flushed":20000000,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":152,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":6200994051,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"152 99290250187 4381690947718642163","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"20000000 121030679724 696713583949301932","bucket-1:myindex:timings/storage_info":"609 5037061 76362671955","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"151 51862862 19765243313372","bucket-1:myindex:timings/storage_meta_set":"152 51794519 19181748915473","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"152 6495959 303582011715","bucket-1:myindex:timings/storage_set":"40000000 775093290168 -7529719260312036834","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":20000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"226 256864000 2218834190677052","bucket-1:ts_queue_size":0,"indexer_state":"Active","memory_quota":31457280000,"memory_used":16430303232,"memory_used_storage":15718924288,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"3371 581448016 232802883097654","uptime":"30m1.340026886s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T01:59:09.678-07:00 [Info] memstats {"Alloc":212476000, "TotalAlloc":19138228384, "Sys":9854296392, "Lookups":8086, "Mallocs":534866461,"Frees":525676997, "HeapAlloc":212476000, "HeapSys":9387048960, "HeapIdle":8973115392, "HeapInuse":413933568,"HeapReleased":2878324736, "HeapObjects":9189464,"GCSys":297445376, "LastGC":1460019485507034671,"PauseTotalNs":31854349, "PauseNs":[1130381], "NumGC":25} 2016-04-07T01:59:09.686-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5353304353, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:65.94266539023734%, mode : circular 2016-04-07T01:59:09.686-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T01:59:23.431-07:00 [Info] Indexer::ReadMemstats Time Taken 10.86337ms 2016-04-07T01:59:39.686-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5353304353, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:65.94266539023734%, mode : circular 2016-04-07T01:59:39.686-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:00:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:00:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:00:09.374-07:00 [Info] client load stats {} 2016-04-07T02:00:09.647-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":0,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":6208593017,"bucket-1:myindex:avg_ts_items_count":93631,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":5353304353,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":15718506496,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":0,"bucket-1:myindex:frag_percent":65,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":2100000000,"bucket-1:myindex:items_count":20000000,"bucket-1:myindex:num_commits":152,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":20000000,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":20000000,"bucket-1:myindex:num_docs_queued":0,"bucket-1:myindex:num_flush_queued":20000000,"bucket-1:myindex:num_items_flushed":20000000,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":152,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":6200994051,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"152 99290250187 4381690947718642163","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"20000000 121030679724 696713583949301932","bucket-1:myindex:timings/storage_info":"609 5037061 76362671955","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"151 51862862 19765243313372","bucket-1:myindex:timings/storage_meta_set":"152 51794519 19181748915473","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"152 6495959 303582011715","bucket-1:myindex:timings/storage_set":"40000000 775093290168 -7529719260312036834","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":20000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"238 266678820 2226870554126562","bucket-1:ts_queue_size":0,"indexer_state":"Active","memory_quota":31457280000,"memory_used":16449636352,"memory_used_storage":15718924288,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"3443 594373957 235190048217615","uptime":"31m1.340330321s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:00:09.678-07:00 [Info] memstats {"Alloc":240628944, "TotalAlloc":19166381328, "Sys":9854296392, "Lookups":8165, "Mallocs":535627408,"Frees":525848377, "HeapAlloc":240628944, "HeapSys":9387048960, "HeapIdle":8953782272, "HeapInuse":433266688,"HeapReleased":3493339136, "HeapObjects":9779031,"GCSys":297445376, "LastGC":1460019485507034671,"PauseTotalNs":31854349, "PauseNs":[], "NumGC":25} 2016-04-07T02:00:09.687-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5353304353, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:65.94266539023734%, mode : circular 2016-04-07T02:00:09.687-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:00:23.666-07:00 [Info] Indexer::ReadMemstats Time Taken 10.790872ms 2016-04-07T02:00:39.687-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5353304353, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:65.94266539023734%, mode : circular 2016-04-07T02:00:39.687-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:01:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:01:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:01:09.374-07:00 [Info] client load stats {} 2016-04-07T02:01:09.647-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":0,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":6208593017,"bucket-1:myindex:avg_ts_items_count":93631,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":5353304353,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":15718506496,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":0,"bucket-1:myindex:frag_percent":65,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":2100000000,"bucket-1:myindex:items_count":20000000,"bucket-1:myindex:num_commits":152,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":20000000,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":20000000,"bucket-1:myindex:num_docs_queued":0,"bucket-1:myindex:num_flush_queued":20000000,"bucket-1:myindex:num_items_flushed":20000000,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":152,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":6200994051,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"152 99290250187 4381690947718642163","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"20000000 121030679724 696713583949301932","bucket-1:myindex:timings/storage_info":"609 5037061 76362671955","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"151 51862862 19765243313372","bucket-1:myindex:timings/storage_meta_set":"152 51794519 19181748915473","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"152 6495959 303582011715","bucket-1:myindex:timings/storage_set":"40000000 775093290168 -7529719260312036834","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":20000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"250 276578952 2235049266682140","bucket-1:ts_queue_size":0,"indexer_state":"Active","memory_quota":31457280000,"memory_used":16429942784,"memory_used_storage":15718924288,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"3515 607239877 237589442148579","uptime":"32m1.340808614s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:01:09.678-07:00 [Info] memstats {"Alloc":212480744, "TotalAlloc":19194318168, "Sys":9854296392, "Lookups":8243, "Mallocs":536389076,"Frees":527199401, "HeapAlloc":212480744, "HeapSys":9387114496, "HeapIdle":8973541376, "HeapInuse":413573120,"HeapReleased":3478970368, "HeapObjects":9189675,"GCSys":297445376, "LastGC":1460019605577270468,"PauseTotalNs":32447649, "PauseNs":[593300], "NumGC":26} 2016-04-07T02:01:09.687-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5353304353, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:65.94266539023734%, mode : circular 2016-04-07T02:01:09.687-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:01:23.892-07:00 [Info] Indexer::ReadMemstats Time Taken 10.584982ms 2016-04-07T02:01:39.687-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5353304353, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:65.94266539023734%, mode : circular 2016-04-07T02:01:39.687-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:02:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:02:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:02:09.374-07:00 [Info] client load stats {} 2016-04-07T02:02:09.648-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":0,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":6208593017,"bucket-1:myindex:avg_ts_items_count":93631,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":5353304353,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":15718506496,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":0,"bucket-1:myindex:frag_percent":65,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":2100000000,"bucket-1:myindex:items_count":20000000,"bucket-1:myindex:num_commits":152,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":20000000,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":20000000,"bucket-1:myindex:num_docs_queued":0,"bucket-1:myindex:num_flush_queued":20000000,"bucket-1:myindex:num_items_flushed":20000000,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":152,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":6200994051,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"152 99290250187 4381690947718642163","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"20000000 121030679724 696713583949301932","bucket-1:myindex:timings/storage_info":"609 5037061 76362671955","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"151 51862862 19765243313372","bucket-1:myindex:timings/storage_meta_set":"152 51794519 19181748915473","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"152 6495959 303582011715","bucket-1:myindex:timings/storage_set":"40000000 775093290168 -7529719260312036834","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":20000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"262 286605097 2243457212723697","bucket-1:ts_queue_size":0,"indexer_state":"Active","memory_quota":31457280000,"memory_used":16449488896,"memory_used_storage":15718924288,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"3587 619990068 239950330224414","uptime":"33m1.341137156s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:02:09.679-07:00 [Info] memstats {"Alloc":240302280, "TotalAlloc":19222139704, "Sys":9854296392, "Lookups":8322, "Mallocs":537146680,"Frees":527369709, "HeapAlloc":240302280, "HeapSys":9387114496, "HeapIdle":8953995264, "HeapInuse":433119232,"HeapReleased":3478970368, "HeapObjects":9776971,"GCSys":297445376, "LastGC":1460019605577270468,"PauseTotalNs":32447649, "PauseNs":[], "NumGC":26} 2016-04-07T02:02:09.688-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5353304353, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:65.94266539023734%, mode : circular 2016-04-07T02:02:09.688-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:02:24.112-07:00 [Info] Indexer::ReadMemstats Time Taken 10.779322ms 2016-04-07T02:02:39.688-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5353304353, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:65.94266539023734%, mode : circular 2016-04-07T02:02:39.688-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:03:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:03:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:03:09.374-07:00 [Info] client load stats {} 2016-04-07T02:03:09.648-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":0,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":6208593017,"bucket-1:myindex:avg_ts_items_count":93631,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":5353304353,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":15718506496,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":0,"bucket-1:myindex:frag_percent":65,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":2100000000,"bucket-1:myindex:items_count":20000000,"bucket-1:myindex:num_commits":152,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":20000000,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":20000000,"bucket-1:myindex:num_docs_queued":0,"bucket-1:myindex:num_flush_queued":20000000,"bucket-1:myindex:num_items_flushed":20000000,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":152,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":6200994051,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"152 99290250187 4381690947718642163","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"20000000 121030679724 696713583949301932","bucket-1:myindex:timings/storage_info":"609 5037061 76362671955","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"151 51862862 19765243313372","bucket-1:myindex:timings/storage_meta_set":"152 51794519 19181748915473","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"152 6495959 303582011715","bucket-1:myindex:timings/storage_set":"40000000 775093290168 -7529719260312036834","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":20000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"274 296543932 2251712035545296","bucket-1:ts_queue_size":0,"indexer_state":"Active","memory_quota":31457280000,"memory_used":16429975552,"memory_used_storage":15718924288,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"3659 632706305 242289153680427","uptime":"34m1.341539474s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:03:09.679-07:00 [Info] memstats {"Alloc":212662184, "TotalAlloc":19250079952, "Sys":9854296392, "Lookups":8401, "Mallocs":537908305,"Frees":528714351, "HeapAlloc":212662184, "HeapSys":9387147264, "HeapIdle":8973541376, "HeapInuse":413605888,"HeapReleased":4435066880, "HeapObjects":9193954,"GCSys":297445376, "LastGC":1460019725665827284,"PauseTotalNs":33156963, "PauseNs":[709314], "NumGC":27} 2016-04-07T02:03:09.688-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5353304353, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:65.94266539023734%, mode : circular 2016-04-07T02:03:09.688-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:03:24.342-07:00 [Info] Indexer::ReadMemstats Time Taken 10.793342ms 2016-04-07T02:03:39.688-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5353304353, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:65.94266539023734%, mode : circular 2016-04-07T02:03:39.688-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:04:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:04:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:04:09.374-07:00 [Info] client load stats {} 2016-04-07T02:04:09.648-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":0,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":6208593017,"bucket-1:myindex:avg_ts_items_count":93631,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":5353304353,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":15718506496,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":0,"bucket-1:myindex:frag_percent":65,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":2100000000,"bucket-1:myindex:items_count":20000000,"bucket-1:myindex:num_commits":152,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":20000000,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":20000000,"bucket-1:myindex:num_docs_queued":0,"bucket-1:myindex:num_flush_queued":20000000,"bucket-1:myindex:num_items_flushed":20000000,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":152,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":6200994051,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"152 99290250187 4381690947718642163","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"20000000 121030679724 696713583949301932","bucket-1:myindex:timings/storage_info":"609 5037061 76362671955","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"151 51862862 19765243313372","bucket-1:myindex:timings/storage_meta_set":"152 51794519 19181748915473","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"152 6495959 303582011715","bucket-1:myindex:timings/storage_set":"40000000 775093290168 -7529719260312036834","bucket-1:myindex:timings/storage_snapshot_close":"0 0 0","bucket-1:myindex:timings/storage_snapshot_create":"0 0 0","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":20000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"286 306340611 2259721472732291","bucket-1:ts_queue_size":0,"indexer_state":"Active","memory_quota":31457280000,"memory_used":16449669120,"memory_used_storage":15718924288,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"3731 646027401 244873360371687","uptime":"35m1.341838719s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:04:09.679-07:00 [Info] memstats {"Alloc":240445296, "TotalAlloc":19277863064, "Sys":9854296392, "Lookups":8480, "Mallocs":538665288,"Frees":528884602, "HeapAlloc":240445296, "HeapSys":9387147264, "HeapIdle":8953847808, "HeapInuse":433299456,"HeapReleased":4435066880, "HeapObjects":9780686,"GCSys":297445376, "LastGC":1460019725665827284,"PauseTotalNs":33156963, "PauseNs":[], "NumGC":27} 2016-04-07T02:04:09.689-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5353304353, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:65.94266539023734%, mode : circular 2016-04-07T02:04:09.689-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:04:24.575-07:00 [Info] Indexer::ReadMemstats Time Taken 10.745515ms 2016-04-07T02:04:39.689-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5353304353, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:65.94266539023734%, mode : circular 2016-04-07T02:04:39.689-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:04:54.395-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20010000 2016-04-07T02:04:54.567-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 40.193µs CommitTime 88.366471ms TotalFlushTime 15m7.046628356s TotalCommitTime 1m39.378616658s 2016-04-07T02:04:54.567-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20005574, 20005574, 153 committed:true 2016-04-07T02:04:54.568-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20005574, 20005574, 153 committed:true) SnapCreateDur 97.784529ms SnapOpenDur 44.933µs 2016-04-07T02:04:54.633-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20020000 2016-04-07T02:04:54.696-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20010000 QueuedCount 3865 2016-04-07T02:04:54.832-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20013865, 20013865, 0 committed:false 2016-04-07T02:04:54.832-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20013865, 20013865, 0 committed:false) SnapCreateDur 36.756µs SnapOpenDur 160.321µs 2016-04-07T02:04:54.859-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20030000 2016-04-07T02:04:55.037-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20020000 QueuedCount 2347 2016-04-07T02:04:55.08-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20040000 2016-04-07T02:04:55.116-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20022347, 20022347, 0 committed:false 2016-04-07T02:04:55.116-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20022347, 20022347, 0 committed:false) SnapCreateDur 32.699µs SnapOpenDur 374.737µs 2016-04-07T02:04:55.3-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20050000 2016-04-07T02:04:55.399-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20030000 QueuedCount 1166 2016-04-07T02:04:55.47-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20031166, 20031166, 0 committed:false 2016-04-07T02:04:55.471-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20031166, 20031166, 0 committed:false) SnapCreateDur 37.183µs SnapOpenDur 832.55µs 2016-04-07T02:04:55.524-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20060000 2016-04-07T02:04:55.739-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20070000 2016-04-07T02:04:55.784-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20040000 QueuedCount 166 2016-04-07T02:04:55.795-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20040166, 20040166, 0 committed:false 2016-04-07T02:04:55.797-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20040166, 20040166, 0 committed:false) SnapCreateDur 31.07µs SnapOpenDur 1.961728ms 2016-04-07T02:04:55.964-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20080000 2016-04-07T02:04:56.199-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20090000 2016-04-07T02:04:56.261-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20049357, 20049357, 0 committed:false 2016-04-07T02:04:56.262-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20049357, 20049357, 0 committed:false) SnapCreateDur 30.119µs SnapOpenDur 645.174µs 2016-04-07T02:04:56.27-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20050000 QueuedCount 8026 2016-04-07T02:04:56.409-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20100000 2016-04-07T02:04:56.626-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20110000 2016-04-07T02:04:56.666-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20058026, 20058026, 0 committed:false 2016-04-07T02:04:56.667-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20058026, 20058026, 0 committed:false) SnapCreateDur 29.088µs SnapOpenDur 1.170775ms 2016-04-07T02:04:56.842-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20120000 2016-04-07T02:04:56.856-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20060000 QueuedCount 7508 2016-04-07T02:04:57.068-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20130000 2016-04-07T02:04:57.141-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20067508, 20067508, 0 committed:false 2016-04-07T02:04:57.141-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20067508, 20067508, 0 committed:false) SnapCreateDur 35.782µs SnapOpenDur 430.562µs 2016-04-07T02:04:57.145-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:04:57.284-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20140000 2016-04-07T02:04:57.388-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20070000 QueuedCount 15390 2016-04-07T02:04:57.504-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20150000 2016-04-07T02:04:57.728-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20160000 2016-04-07T02:04:57.753-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20080000 QueuedCount 5390 2016-04-07T02:04:57.919-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20085390, 20085390, 0 committed:false 2016-04-07T02:04:57.921-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20085390, 20085390, 0 committed:false) SnapCreateDur 28.449µs SnapOpenDur 1.91493ms 2016-04-07T02:04:57.925-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:04:57.954-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20170000 2016-04-07T02:04:58.18-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20180000 2016-04-07T02:04:58.209-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20090000 QueuedCount 13459 2016-04-07T02:04:58.408-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20190000 2016-04-07T02:04:58.571-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20100000 QueuedCount 3459 2016-04-07T02:04:58.627-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20200000 2016-04-07T02:04:58.669-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20103459, 20103459, 0 committed:false 2016-04-07T02:04:58.67-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20103459, 20103459, 0 committed:false) SnapCreateDur 27.42µs SnapOpenDur 1.435763ms 2016-04-07T02:04:58.674-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:04:58.846-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20210000 2016-04-07T02:04:59.075-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20220000 2016-04-07T02:04:59.099-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20110000 QueuedCount 11938 2016-04-07T02:04:59.288-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20230000 2016-04-07T02:04:59.466-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20120000 QueuedCount 1938 2016-04-07T02:04:59.507-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20240000 2016-04-07T02:04:59.548-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20121938, 20121938, 0 committed:false 2016-04-07T02:04:59.549-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20121938, 20121938, 0 committed:false) SnapCreateDur 33.702µs SnapOpenDur 1.48ms 2016-04-07T02:04:59.728-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20250000 2016-04-07T02:04:59.954-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20260000 2016-04-07T02:05:00.067-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20130000 QueuedCount 811 2016-04-07T02:05:00.183-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20270000 2016-04-07T02:05:00.415-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20280000 2016-04-07T02:05:00.645-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20290000 2016-04-07T02:05:00.801-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 54.308µs CommitTime 531.546241ms TotalFlushTime 15m12.49763249s TotalCommitTime 1m39.910162899s 2016-04-07T02:05:00.801-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20130811, 20130811, 154 committed:true 2016-04-07T02:05:00.801-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20130811, 20130811, 154 committed:true) SnapCreateDur 621.751207ms SnapOpenDur 51.695µs 2016-04-07T02:05:00.806-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:00.81-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:00.87-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20300000 2016-04-07T02:05:01.09-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20310000 2016-04-07T02:05:01.155-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20140000 QueuedCount 18026 2016-04-07T02:05:01.315-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20320000 2016-04-07T02:05:01.503-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20150000 QueuedCount 8026 2016-04-07T02:05:01.533-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20330000 2016-04-07T02:05:01.759-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20340000 2016-04-07T02:05:01.794-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20158026, 20158026, 0 committed:false 2016-04-07T02:05:01.795-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20158026, 20158026, 0 committed:false) SnapCreateDur 28.716µs SnapOpenDur 518.579µs 2016-04-07T02:05:01.799-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:01.803-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:01.93-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20160000 QueuedCount 24577 2016-04-07T02:05:01.98-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20350000 2016-04-07T02:05:02.2-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20360000 2016-04-07T02:05:02.28-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20170000 QueuedCount 14577 2016-04-07T02:05:02.418-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20370000 2016-04-07T02:05:02.64-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20380000 2016-04-07T02:05:02.643-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20180000 QueuedCount 4577 2016-04-07T02:05:02.807-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20184577, 20184577, 0 committed:false 2016-04-07T02:05:02.808-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20184577, 20184577, 0 committed:false) SnapCreateDur 52.21µs SnapOpenDur 570.22µs 2016-04-07T02:05:02.812-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:02.816-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:02.864-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20390000 2016-04-07T02:05:03.09-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20400000 2016-04-07T02:05:03.111-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20190000 QueuedCount 21724 2016-04-07T02:05:03.312-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20410000 2016-04-07T02:05:03.47-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20200000 QueuedCount 11724 2016-04-07T02:05:03.54-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20420000 2016-04-07T02:05:03.764-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20430000 2016-04-07T02:05:03.832-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20210000 QueuedCount 1724 2016-04-07T02:05:03.91-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20211724, 20211724, 0 committed:false 2016-04-07T02:05:03.911-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20211724, 20211724, 0 committed:false) SnapCreateDur 38.795µs SnapOpenDur 938.233µs 2016-04-07T02:05:03.915-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:03.92-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:03.992-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20440000 2016-04-07T02:05:04.22-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20450000 2016-04-07T02:05:04.373-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20220000 QueuedCount 19053 2016-04-07T02:05:04.438-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20460000 2016-04-07T02:05:04.66-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20470000 2016-04-07T02:05:04.744-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20230000 QueuedCount 9053 2016-04-07T02:05:04.89-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20480000 2016-04-07T02:05:05.074-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20239053, 20239053, 0 committed:false 2016-04-07T02:05:05.077-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20239053, 20239053, 0 committed:false) SnapCreateDur 40.981µs SnapOpenDur 3.084226ms 2016-04-07T02:05:05.119-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20490000 2016-04-07T02:05:05.34-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20500000 2016-04-07T02:05:05.417-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20240000 QueuedCount 8022 2016-04-07T02:05:05.562-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20510000 2016-04-07T02:05:05.793-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20520000 2016-04-07T02:05:06.015-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20530000 2016-04-07T02:05:06.238-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20540000 2016-04-07T02:05:06.275-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 39.283µs CommitTime 534.093645ms TotalFlushTime 15m17.324419557s TotalCommitTime 1m40.444256544s 2016-04-07T02:05:06.275-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20248022, 20248022, 155 committed:true 2016-04-07T02:05:06.275-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20248022, 20248022, 155 committed:true) SnapCreateDur 543.414621ms SnapOpenDur 52.58µs 2016-04-07T02:05:06.28-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:06.284-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:06.298-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20250000 QueuedCount 24943 2016-04-07T02:05:06.459-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20550000 2016-04-07T02:05:06.665-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20260000 QueuedCount 14943 2016-04-07T02:05:06.684-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20560000 2016-04-07T02:05:06.909-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20570000 2016-04-07T02:05:07.004-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20270000 QueuedCount 4943 2016-04-07T02:05:07.128-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20580000 2016-04-07T02:05:07.208-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20274943, 20274943, 0 committed:false 2016-04-07T02:05:07.208-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20274943, 20274943, 0 committed:false) SnapCreateDur 32.498µs SnapOpenDur 366.606µs 2016-04-07T02:05:07.212-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:07.218-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:07.35-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20590000 2016-04-07T02:05:07.44-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20280000 QueuedCount 20641 2016-04-07T02:05:07.567-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20600000 2016-04-07T02:05:07.784-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20610000 2016-04-07T02:05:07.807-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20290000 QueuedCount 10641 2016-04-07T02:05:08.009-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20620000 2016-04-07T02:05:08.228-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20630000 2016-04-07T02:05:08.277-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20300000 QueuedCount 641 2016-04-07T02:05:08.292-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20300641, 20300641, 0 committed:false 2016-04-07T02:05:08.293-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20300641, 20300641, 0 committed:false) SnapCreateDur 28.608µs SnapOpenDur 1.223966ms 2016-04-07T02:05:08.297-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:08.3-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:08.445-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20640000 2016-04-07T02:05:08.668-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20650000 2016-04-07T02:05:08.769-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20310000 QueuedCount 17859 2016-04-07T02:05:08.89-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20660000 2016-04-07T02:05:09.117-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20670000 2016-04-07T02:05:09.128-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20320000 QueuedCount 7859 2016-04-07T02:05:09.347-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20680000 2016-04-07T02:05:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:05:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:05:09.374-07:00 [Info] client load stats {} 2016-04-07T02:05:09.374-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20327859, 20327859, 0 committed:false 2016-04-07T02:05:09.376-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20327859, 20327859, 0 committed:false) SnapCreateDur 27.24µs SnapOpenDur 2.175239ms 2016-04-07T02:05:09.381-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:09.385-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:09.575-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20690000 2016-04-07T02:05:09.65-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":338415,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":1074293232,"bucket-1:myindex:avg_ts_items_count":25602,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":5415636013,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":15718506496,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":24787,"bucket-1:myindex:frag_percent":65,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":2124059552,"bucket-1:myindex:items_count":20130811,"bucket-1:myindex:num_commits":155,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":20329961,"bucket-1:myindex:num_docs_pending":191,"bucket-1:myindex:num_docs_processed":20229886,"bucket-1:myindex:num_docs_queued":243016,"bucket-1:myindex:num_flush_queued":20354748,"bucket-1:myindex:num_items_flushed":20329961,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":172,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":1083696652,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"155 100444256544 4957297008866252110","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"20329962 122435370039 697648159764591031","bucket-1:myindex:timings/storage_info":"655 5303855 79137563291","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"154 53138897 20334733691671","bucket-1:myindex:timings/storage_meta_set":"155 52990502 19682091967810","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"155 6595867 306949002347","bucket-1:myindex:timings/storage_set":"40659923 787250156259 -6520805861598031011","bucket-1:myindex:timings/storage_snapshot_close":"16 12475535 28952529899497","bucket-1:myindex:timings/storage_snapshot_create":"17 17107342 24005787802374","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":20693163,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"297 315761674 2267855273667946","bucket-1:ts_queue_size":37,"indexer_state":"Active","memory_quota":31457280000,"memory_used":16430319616,"memory_used_storage":15718916096,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"3805 660087157 247655031895601","uptime":"36m1.34228768s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:05:09.679-07:00 [Info] memstats {"Alloc":213178272, "TotalAlloc":19306172152, "Sys":9854296392, "Lookups":8559, "Mallocs":539430832,"Frees":530230320, "HeapAlloc":213178272, "HeapSys":9387147264, "HeapIdle":8973189120, "HeapInuse":413958144,"HeapReleased":8863801344, "HeapObjects":9200512,"GCSys":297445376, "LastGC":1460019845749734693,"PauseTotalNs":33646084, "PauseNs":[489121], "NumGC":28} 2016-04-07T02:05:09.689-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5442930827, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:65.37246825336045%, mode : circular 2016-04-07T02:05:09.689-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:05:09.701-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20330000 QueuedCount 24748 2016-04-07T02:05:09.803-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20700000 2016-04-07T02:05:10.026-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20340000 QueuedCount 14748 2016-04-07T02:05:10.028-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20710000 2016-04-07T02:05:10.285-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20720000 2016-04-07T02:05:10.401-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20350000 QueuedCount 4748 2016-04-07T02:05:10.505-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20730000 2016-04-07T02:05:10.629-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20354748, 20354748, 0 committed:false 2016-04-07T02:05:10.629-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20354748, 20354748, 0 committed:false) SnapCreateDur 27.866µs SnapOpenDur 205.102µs 2016-04-07T02:05:10.724-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20740000 2016-04-07T02:05:10.95-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20750000 2016-04-07T02:05:10.985-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20360000 QueuedCount 3836 2016-04-07T02:05:11.169-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20760000 2016-04-07T02:05:11.389-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20770000 2016-04-07T02:05:11.614-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20780000 2016-04-07T02:05:11.623-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.252µs CommitTime 480.995337ms TotalFlushTime 15m22.084757791s TotalCommitTime 1m40.925251881s 2016-04-07T02:05:11.624-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20363836, 20363836, 156 committed:true 2016-04-07T02:05:11.624-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20363836, 20363836, 156 committed:true) SnapCreateDur 490.253427ms SnapOpenDur 77.92µs 2016-04-07T02:05:11.628-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:11.632-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:11.838-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20790000 2016-04-07T02:05:11.984-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20370000 QueuedCount 20980 2016-04-07T02:05:12.069-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20800000 2016-04-07T02:05:12.293-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20380000 QueuedCount 10980 2016-04-07T02:05:12.302-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20810000 2016-04-07T02:05:12.526-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20820000 2016-04-07T02:05:12.651-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20390000 QueuedCount 980 2016-04-07T02:05:12.726-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20390980, 20390980, 0 committed:false 2016-04-07T02:05:12.727-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20390980, 20390980, 0 committed:false) SnapCreateDur 28.463µs SnapOpenDur 480.039µs 2016-04-07T02:05:12.731-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:12.735-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:12.75-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20830000 2016-04-07T02:05:12.976-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20840000 2016-04-07T02:05:13.091-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20400000 QueuedCount 17467 2016-04-07T02:05:13.193-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20850000 2016-04-07T02:05:13.413-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20860000 2016-04-07T02:05:13.429-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20410000 QueuedCount 7467 2016-04-07T02:05:13.638-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20870000 2016-04-07T02:05:13.7-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20417467, 20417467, 0 committed:false 2016-04-07T02:05:13.7-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20417467, 20417467, 0 committed:false) SnapCreateDur 30.142µs SnapOpenDur 424.129µs 2016-04-07T02:05:13.704-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:13.708-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:13.864-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20880000 2016-04-07T02:05:13.89-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20420000 QueuedCount 24186 2016-04-07T02:05:14.083-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20890000 2016-04-07T02:05:14.233-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20430000 QueuedCount 14186 2016-04-07T02:05:14.306-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20900000 2016-04-07T02:05:14.532-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20910000 2016-04-07T02:05:14.588-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20440000 QueuedCount 4186 2016-04-07T02:05:14.742-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20444186, 20444186, 0 committed:false 2016-04-07T02:05:14.743-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20444186, 20444186, 0 committed:false) SnapCreateDur 37.636µs SnapOpenDur 501.162µs 2016-04-07T02:05:14.747-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:14.75-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:14.755-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20920000 2016-04-07T02:05:14.981-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20930000 2016-04-07T02:05:15.202-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20450000 QueuedCount 21076 2016-04-07T02:05:15.209-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20940000 2016-04-07T02:05:15.429-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20950000 2016-04-07T02:05:15.52-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20460000 QueuedCount 11076 2016-04-07T02:05:15.648-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20960000 2016-04-07T02:05:15.871-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20970000 2016-04-07T02:05:15.875-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20470000 QueuedCount 1076 2016-04-07T02:05:15.947-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20471076, 20471076, 0 committed:false 2016-04-07T02:05:15.948-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20471076, 20471076, 0 committed:false) SnapCreateDur 65.805µs SnapOpenDur 760.882µs 2016-04-07T02:05:16.091-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20980000 2016-04-07T02:05:16.314-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 20990000 2016-04-07T02:05:16.535-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21000000 2016-04-07T02:05:16.766-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21010000 2016-04-07T02:05:16.836-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.88µs CommitTime 464.613986ms TotalFlushTime 15m26.721753053s TotalCommitTime 1m41.389865867s 2016-04-07T02:05:16.836-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20479729, 20479729, 157 committed:true 2016-04-07T02:05:16.836-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20479729, 20479729, 157 committed:true) SnapCreateDur 474.551389ms SnapOpenDur 39.247µs 2016-04-07T02:05:16.84-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:16.841-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20480000 QueuedCount 10344 2016-04-07T02:05:16.845-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:16.848-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:16.987-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21020000 2016-04-07T02:05:17.206-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21030000 2016-04-07T02:05:17.316-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20490000 QueuedCount 25280 2016-04-07T02:05:17.425-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21040000 2016-04-07T02:05:17.594-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20500000 QueuedCount 15280 2016-04-07T02:05:17.709-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21050000 2016-04-07T02:05:17.872-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21060000 2016-04-07T02:05:18.069-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20510000 QueuedCount 5280 2016-04-07T02:05:18.101-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21070000 2016-04-07T02:05:18.262-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20515280, 20515280, 0 committed:false 2016-04-07T02:05:18.263-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20515280, 20515280, 0 committed:false) SnapCreateDur 41.918µs SnapOpenDur 815.824µs 2016-04-07T02:05:18.267-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:18.27-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:18.274-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:18.324-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21080000 2016-04-07T02:05:18.451-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20520000 QueuedCount 31075 2016-04-07T02:05:18.547-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21090000 2016-04-07T02:05:18.783-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21100000 2016-04-07T02:05:18.812-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20530000 QueuedCount 21075 2016-04-07T02:05:19.01-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21110000 2016-04-07T02:05:19.195-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20540000 QueuedCount 11075 2016-04-07T02:05:19.238-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21120000 2016-04-07T02:05:19.459-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21130000 2016-04-07T02:05:19.57-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20550000 QueuedCount 1075 2016-04-07T02:05:19.588-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20551075, 20551075, 0 committed:false 2016-04-07T02:05:19.59-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20551075, 20551075, 0 committed:false) SnapCreateDur 39.811µs SnapOpenDur 1.63907ms 2016-04-07T02:05:19.594-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:19.597-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:19.601-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:19.686-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21140000 2016-04-07T02:05:19.909-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21150000 2016-04-07T02:05:20.115-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20560000 QueuedCount 27066 2016-04-07T02:05:20.131-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21160000 2016-04-07T02:05:20.363-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21170000 2016-04-07T02:05:20.495-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20570000 QueuedCount 17066 2016-04-07T02:05:20.587-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21180000 2016-04-07T02:05:20.804-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21190000 2016-04-07T02:05:20.818-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20580000 QueuedCount 7066 2016-04-07T02:05:21.033-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21200000 2016-04-07T02:05:21.125-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20587066, 20587066, 0 committed:false 2016-04-07T02:05:21.126-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20587066, 20587066, 0 committed:false) SnapCreateDur 52.615µs SnapOpenDur 1.807909ms 2016-04-07T02:05:21.254-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21210000 2016-04-07T02:05:21.331-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20590000 QueuedCount 6257 2016-04-07T02:05:21.478-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21220000 2016-04-07T02:05:21.709-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21230000 2016-04-07T02:05:21.926-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21240000 2016-04-07T02:05:22.085-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.732µs CommitTime 512.297288ms TotalFlushTime 15m31.357081706s TotalCommitTime 1m41.902163155s 2016-04-07T02:05:22.085-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20596257, 20596257, 158 committed:true 2016-04-07T02:05:22.085-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20596257, 20596257, 158 committed:true) SnapCreateDur 523.54321ms SnapOpenDur 66.603µs 2016-04-07T02:05:22.09-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:22.094-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:22.098-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:22.144-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21250000 2016-04-07T02:05:22.326-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20600000 QueuedCount 32657 2016-04-07T02:05:22.363-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21260000 2016-04-07T02:05:22.588-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21270000 2016-04-07T02:05:22.636-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20610000 QueuedCount 22657 2016-04-07T02:05:22.81-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21280000 2016-04-07T02:05:22.959-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20620000 QueuedCount 12657 2016-04-07T02:05:23.033-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21290000 2016-04-07T02:05:23.26-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21300000 2016-04-07T02:05:23.29-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20630000 QueuedCount 2657 2016-04-07T02:05:23.374-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20632657, 20632657, 0 committed:false 2016-04-07T02:05:23.376-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20632657, 20632657, 0 committed:false) SnapCreateDur 32.399µs SnapOpenDur 1.525672ms 2016-04-07T02:05:23.38-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:23.385-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:23.39-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:23.484-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21310000 2016-04-07T02:05:23.685-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20640000 QueuedCount 28657 2016-04-07T02:05:23.708-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21320000 2016-04-07T02:05:23.938-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21330000 2016-04-07T02:05:24.036-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20650000 QueuedCount 18657 2016-04-07T02:05:24.163-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21340000 2016-04-07T02:05:24.382-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21350000 2016-04-07T02:05:24.397-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20660000 QueuedCount 8657 2016-04-07T02:05:24.616-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21360000 2016-04-07T02:05:24.715-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20668657, 20668657, 0 committed:false 2016-04-07T02:05:24.715-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20668657, 20668657, 0 committed:false) SnapCreateDur 30.022µs SnapOpenDur 688.64µs 2016-04-07T02:05:24.72-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:24.726-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:24.731-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:24.741-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20670000 QueuedCount 33811 2016-04-07T02:05:24.832-07:00 [Info] Indexer::ReadMemstats Time Taken 11.789014ms 2016-04-07T02:05:24.842-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21370000 2016-04-07T02:05:25.058-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21380000 2016-04-07T02:05:25.177-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20680000 QueuedCount 23811 2016-04-07T02:05:25.276-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21390000 2016-04-07T02:05:25.504-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21400000 2016-04-07T02:05:25.542-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20690000 QueuedCount 13811 2016-04-07T02:05:25.727-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21410000 2016-04-07T02:05:25.928-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20700000 QueuedCount 3811 2016-04-07T02:05:25.956-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21420000 2016-04-07T02:05:26.086-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20703811, 20703811, 0 committed:false 2016-04-07T02:05:26.087-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20703811, 20703811, 0 committed:false) SnapCreateDur 52.722µs SnapOpenDur 958.073µs 2016-04-07T02:05:26.092-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:26.096-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:26.102-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:26.183-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21430000 2016-04-07T02:05:26.402-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21440000 2016-04-07T02:05:26.53-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20710000 QueuedCount 28327 2016-04-07T02:05:26.619-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21450000 2016-04-07T02:05:26.843-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21460000 2016-04-07T02:05:26.899-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20720000 QueuedCount 18327 2016-04-07T02:05:27.063-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21470000 2016-04-07T02:05:27.292-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20730000 QueuedCount 8327 2016-04-07T02:05:27.294-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21480000 2016-04-07T02:05:27.525-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21490000 2016-04-07T02:05:27.597-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20738327, 20738327, 0 committed:false 2016-04-07T02:05:27.598-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20738327, 20738327, 0 committed:false) SnapCreateDur 26.098µs SnapOpenDur 602.226µs 2016-04-07T02:05:27.745-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21500000 2016-04-07T02:05:27.824-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20740000 QueuedCount 7057 2016-04-07T02:05:27.975-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21510000 2016-04-07T02:05:28.198-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21520000 2016-04-07T02:05:28.426-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21530000 2016-04-07T02:05:28.65-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21540000 2016-04-07T02:05:28.666-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.219µs CommitTime 593.429363ms TotalFlushTime 15m37.222110345s TotalCommitTime 1m42.495592518s 2016-04-07T02:05:28.666-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20747057, 20747057, 159 committed:true 2016-04-07T02:05:28.666-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20747057, 20747057, 159 committed:true) SnapCreateDur 603.625496ms SnapOpenDur 51.378µs 2016-04-07T02:05:28.67-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:28.675-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:28.678-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:28.871-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21550000 2016-04-07T02:05:28.914-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20750000 QueuedCount 33226 2016-04-07T02:05:29.093-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21560000 2016-04-07T02:05:29.227-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20760000 QueuedCount 23226 2016-04-07T02:05:29.323-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21570000 2016-04-07T02:05:29.552-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21580000 2016-04-07T02:05:29.749-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20770000 QueuedCount 13226 2016-04-07T02:05:29.775-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21590000 2016-04-07T02:05:30.002-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21600000 2016-04-07T02:05:30.088-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20780000 QueuedCount 3226 2016-04-07T02:05:30.173-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20783226, 20783226, 0 committed:false 2016-04-07T02:05:30.174-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20783226, 20783226, 0 committed:false) SnapCreateDur 40.701µs SnapOpenDur 1.256896ms 2016-04-07T02:05:30.178-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:30.182-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:30.186-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:30.224-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21610000 2016-04-07T02:05:30.457-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21620000 2016-04-07T02:05:30.486-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20790000 QueuedCount 28111 2016-04-07T02:05:30.677-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21630000 2016-04-07T02:05:30.835-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20800000 QueuedCount 18111 2016-04-07T02:05:30.901-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21640000 2016-04-07T02:05:31.131-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21650000 2016-04-07T02:05:31.212-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20810000 QueuedCount 8111 2016-04-07T02:05:31.363-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21660000 2016-04-07T02:05:31.531-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20818111, 20818111, 0 committed:false 2016-04-07T02:05:31.532-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20818111, 20818111, 0 committed:false) SnapCreateDur 28.172µs SnapOpenDur 1.258917ms 2016-04-07T02:05:31.537-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:31.542-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:31.546-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:31.588-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21670000 2016-04-07T02:05:31.692-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20820000 QueuedCount 34139 2016-04-07T02:05:31.808-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21680000 2016-04-07T02:05:32.034-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21690000 2016-04-07T02:05:32.053-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20830000 QueuedCount 24139 2016-04-07T02:05:32.253-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21700000 2016-04-07T02:05:32.426-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20840000 QueuedCount 14139 2016-04-07T02:05:32.469-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21710000 2016-04-07T02:05:32.694-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21720000 2016-04-07T02:05:32.807-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20850000 QueuedCount 4139 2016-04-07T02:05:32.919-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21730000 2016-04-07T02:05:32.972-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20854139, 20854139, 0 committed:false 2016-04-07T02:05:32.972-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20854139, 20854139, 0 committed:false) SnapCreateDur 36.489µs SnapOpenDur 500.781µs 2016-04-07T02:05:33.143-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21740000 2016-04-07T02:05:33.369-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21750000 2016-04-07T02:05:33.468-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20860000 QueuedCount 3192 2016-04-07T02:05:33.59-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21760000 2016-04-07T02:05:33.813-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21770000 2016-04-07T02:05:34.044-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21780000 2016-04-07T02:05:34.108-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.752µs CommitTime 530.820465ms TotalFlushTime 15m42.039410974s TotalCommitTime 1m43.026412983s 2016-04-07T02:05:34.108-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20863192, 20863192, 160 committed:true 2016-04-07T02:05:34.108-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20863192, 20863192, 160 committed:true) SnapCreateDur 540.605802ms SnapOpenDur 54.947µs 2016-04-07T02:05:34.113-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:34.118-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:34.123-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:34.271-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21790000 2016-04-07T02:05:34.368-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20870000 QueuedCount 28996 2016-04-07T02:05:34.493-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21800000 2016-04-07T02:05:34.696-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20880000 QueuedCount 18996 2016-04-07T02:05:34.721-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21810000 2016-04-07T02:05:34.95-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21820000 2016-04-07T02:05:35.038-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20890000 QueuedCount 8996 2016-04-07T02:05:35.178-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21830000 2016-04-07T02:05:35.328-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20898996, 20898996, 0 committed:false 2016-04-07T02:05:35.329-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20898996, 20898996, 0 committed:false) SnapCreateDur 51.069µs SnapOpenDur 923.842µs 2016-04-07T02:05:35.334-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:35.338-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:35.343-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:35.35-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20900000 QueuedCount 34554 2016-04-07T02:05:35.403-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21840000 2016-04-07T02:05:35.639-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21850000 2016-04-07T02:05:35.753-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20910000 QueuedCount 24554 2016-04-07T02:05:35.859-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21860000 2016-04-07T02:05:36.087-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21870000 2016-04-07T02:05:36.121-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20920000 QueuedCount 14554 2016-04-07T02:05:36.333-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21880000 2016-04-07T02:05:36.482-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20930000 QueuedCount 4554 2016-04-07T02:05:36.553-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21890000 2016-04-07T02:05:36.648-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20934554, 20934554, 0 committed:false 2016-04-07T02:05:36.65-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20934554, 20934554, 0 committed:false) SnapCreateDur 27.68µs SnapOpenDur 1.568721ms 2016-04-07T02:05:36.654-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:36.659-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:36.663-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:36.771-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21900000 2016-04-07T02:05:36.939-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20940000 QueuedCount 30641 2016-04-07T02:05:37.007-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21910000 2016-04-07T02:05:37.232-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21920000 2016-04-07T02:05:37.306-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20950000 QueuedCount 20641 2016-04-07T02:05:37.454-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21930000 2016-04-07T02:05:37.675-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21940000 2016-04-07T02:05:37.691-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20960000 QueuedCount 10641 2016-04-07T02:05:37.903-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21950000 2016-04-07T02:05:38.078-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20970000 QueuedCount 641 2016-04-07T02:05:38.088-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20970641, 20970641, 0 committed:false 2016-04-07T02:05:38.089-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20970641, 20970641, 0 committed:false) SnapCreateDur 41.081µs SnapOpenDur 793.61µs 2016-04-07T02:05:38.122-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21960000 2016-04-07T02:05:38.344-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21970000 2016-04-07T02:05:38.567-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21980000 2016-04-07T02:05:38.789-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 21990000 2016-04-07T02:05:39.018-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22000000 2016-04-07T02:05:39.145-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.968µs CommitTime 521.923933ms TotalFlushTime 15m46.447022802s TotalCommitTime 1m43.548336916s 2016-04-07T02:05:39.145-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 20979723, 20979723, 161 committed:true 2016-04-07T02:05:39.145-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 20979723, 20979723, 161 committed:true) SnapCreateDur 531.161144ms SnapOpenDur 79.698µs 2016-04-07T02:05:39.15-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:39.15-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20980000 QueuedCount 8859 2016-04-07T02:05:39.155-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:39.159-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:39.25-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22010000 2016-04-07T02:05:39.476-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22020000 2016-04-07T02:05:39.556-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 20990000 QueuedCount 25271 2016-04-07T02:05:39.69-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5622202017, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:64.23195792532374%, mode : circular 2016-04-07T02:05:39.69-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:05:39.703-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22030000 2016-04-07T02:05:39.867-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21000000 QueuedCount 15271 2016-04-07T02:05:39.931-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22040000 2016-04-07T02:05:40.163-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22050000 2016-04-07T02:05:40.226-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21010000 QueuedCount 5271 2016-04-07T02:05:40.384-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22060000 2016-04-07T02:05:40.443-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21015271, 21015271, 0 committed:false 2016-04-07T02:05:40.444-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21015271, 21015271, 0 committed:false) SnapCreateDur 28.51µs SnapOpenDur 674.262µs 2016-04-07T02:05:40.449-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:40.453-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:40.458-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:40.613-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22070000 2016-04-07T02:05:40.673-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21020000 QueuedCount 29970 2016-04-07T02:05:40.839-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22080000 2016-04-07T02:05:41.035-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21030000 QueuedCount 19970 2016-04-07T02:05:41.061-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22090000 2016-04-07T02:05:41.283-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22100000 2016-04-07T02:05:41.401-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21040000 QueuedCount 9970 2016-04-07T02:05:41.509-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22110000 2016-04-07T02:05:41.733-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22120000 2016-04-07T02:05:41.791-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21049970, 21049970, 0 committed:false 2016-04-07T02:05:41.792-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21049970, 21049970, 0 committed:false) SnapCreateDur 26.912µs SnapOpenDur 585.812µs 2016-04-07T02:05:41.793-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21050000 QueuedCount 777 2016-04-07T02:05:41.797-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:41.802-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:41.806-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:41.964-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22130000 2016-04-07T02:05:42.193-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22140000 2016-04-07T02:05:42.305-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21060000 QueuedCount 27286 2016-04-07T02:05:42.415-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22150000 2016-04-07T02:05:42.638-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22160000 2016-04-07T02:05:42.667-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21070000 QueuedCount 17286 2016-04-07T02:05:42.866-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22170000 2016-04-07T02:05:42.978-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21080000 QueuedCount 7286 2016-04-07T02:05:43.094-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22180000 2016-04-07T02:05:43.271-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21087286, 21087286, 0 committed:false 2016-04-07T02:05:43.272-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21087286, 21087286, 0 committed:false) SnapCreateDur 36.783µs SnapOpenDur 1.000974ms 2016-04-07T02:05:43.313-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22190000 2016-04-07T02:05:43.537-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22200000 2016-04-07T02:05:43.686-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21090000 QueuedCount 6178 2016-04-07T02:05:43.892-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22210000 2016-04-07T02:05:43.998-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22220000 2016-04-07T02:05:44.217-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22230000 2016-04-07T02:05:44.444-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22240000 2016-04-07T02:05:44.643-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 45.549µs CommitTime 566.824221ms TotalFlushTime 15m51.277318624s TotalCommitTime 1m44.115161137s 2016-04-07T02:05:44.643-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21096178, 21096178, 162 committed:true 2016-04-07T02:05:44.643-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21096178, 21096178, 162 committed:true) SnapCreateDur 576.017723ms SnapOpenDur 58.644µs 2016-04-07T02:05:44.647-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:44.651-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:44.656-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:44.669-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22250000 2016-04-07T02:05:44.788-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21100000 QueuedCount 30823 2016-04-07T02:05:44.892-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22260000 2016-04-07T02:05:45.122-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22270000 2016-04-07T02:05:45.126-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21110000 QueuedCount 20823 2016-04-07T02:05:45.345-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22280000 2016-04-07T02:05:45.475-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21120000 QueuedCount 10823 2016-04-07T02:05:45.57-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22290000 2016-04-07T02:05:45.8-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22300000 2016-04-07T02:05:45.85-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21130000 QueuedCount 823 2016-04-07T02:05:45.86-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21130823, 21130823, 0 committed:false 2016-04-07T02:05:45.862-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21130823, 21130823, 0 committed:false) SnapCreateDur 28.208µs SnapOpenDur 1.956303ms 2016-04-07T02:05:45.867-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:45.871-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:45.875-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:46.024-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22310000 2016-04-07T02:05:46.245-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22320000 2016-04-07T02:05:46.255-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21140000 QueuedCount 26690 2016-04-07T02:05:46.463-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22330000 2016-04-07T02:05:46.616-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21150000 QueuedCount 16690 2016-04-07T02:05:46.686-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22340000 2016-04-07T02:05:46.912-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22350000 2016-04-07T02:05:46.995-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21160000 QueuedCount 6690 2016-04-07T02:05:47.13-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22360000 2016-04-07T02:05:47.229-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21166690, 21166690, 0 committed:false 2016-04-07T02:05:47.23-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21166690, 21166690, 0 committed:false) SnapCreateDur 29.081µs SnapOpenDur 733.284µs 2016-04-07T02:05:47.234-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:47.238-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:47.242-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:47.35-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22370000 2016-04-07T02:05:47.558-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21170000 QueuedCount 32307 2016-04-07T02:05:47.578-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22380000 2016-04-07T02:05:47.802-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22390000 2016-04-07T02:05:47.885-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21180000 QueuedCount 22307 2016-04-07T02:05:48.039-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22400000 2016-04-07T02:05:48.264-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22410000 2016-04-07T02:05:48.269-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21190000 QueuedCount 12307 2016-04-07T02:05:48.49-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22420000 2016-04-07T02:05:48.646-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21200000 QueuedCount 2307 2016-04-07T02:05:48.726-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22430000 2016-04-07T02:05:48.727-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21202307, 21202307, 0 committed:false 2016-04-07T02:05:48.729-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21202307, 21202307, 0 committed:false) SnapCreateDur 38.763µs SnapOpenDur 1.809773ms 2016-04-07T02:05:48.947-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22440000 2016-04-07T02:05:49.118-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21210000 QueuedCount 1359 2016-04-07T02:05:49.179-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22450000 2016-04-07T02:05:49.404-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22460000 2016-04-07T02:05:49.63-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22470000 2016-04-07T02:05:49.649-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.505µs CommitTime 446.878819ms TotalFlushTime 15m55.746084531s TotalCommitTime 1m44.562039956s 2016-04-07T02:05:49.649-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21211359, 21211359, 163 committed:true 2016-04-07T02:05:49.649-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21211359, 21211359, 163 committed:true) SnapCreateDur 456.248481ms SnapOpenDur 58.442µs 2016-04-07T02:05:49.653-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:49.656-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:49.661-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:49.855-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22480000 2016-04-07T02:05:50.083-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22490000 2016-04-07T02:05:50.085-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21220000 QueuedCount 27295 2016-04-07T02:05:50.308-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22500000 2016-04-07T02:05:50.407-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21230000 QueuedCount 17295 2016-04-07T02:05:50.539-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22510000 2016-04-07T02:05:50.705-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21240000 QueuedCount 7295 2016-04-07T02:05:50.76-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22520000 2016-04-07T02:05:50.984-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21247295, 21247295, 0 committed:false 2016-04-07T02:05:50.985-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21247295, 21247295, 0 committed:false) SnapCreateDur 26.85µs SnapOpenDur 968.441µs 2016-04-07T02:05:50.99-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:50.994-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:50.997-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22530000 2016-04-07T02:05:50.998-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:51.153-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21250000 QueuedCount 33319 2016-04-07T02:05:51.218-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22540000 2016-04-07T02:05:51.439-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22550000 2016-04-07T02:05:51.505-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21260000 QueuedCount 23319 2016-04-07T02:05:51.668-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22560000 2016-04-07T02:05:51.863-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21270000 QueuedCount 13319 2016-04-07T02:05:51.885-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22570000 2016-04-07T02:05:52.111-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22580000 2016-04-07T02:05:52.236-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21280000 QueuedCount 3319 2016-04-07T02:05:52.329-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22590000 2016-04-07T02:05:52.382-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21283319, 21283319, 0 committed:false 2016-04-07T02:05:52.383-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21283319, 21283319, 0 committed:false) SnapCreateDur 63.677µs SnapOpenDur 410.727µs 2016-04-07T02:05:52.387-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:52.392-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:52.396-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:52.552-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22600000 2016-04-07T02:05:52.692-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21290000 QueuedCount 28976 2016-04-07T02:05:52.773-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22610000 2016-04-07T02:05:52.994-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22620000 2016-04-07T02:05:53.054-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21300000 QueuedCount 18976 2016-04-07T02:05:53.219-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22630000 2016-04-07T02:05:53.424-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21310000 QueuedCount 8976 2016-04-07T02:05:53.447-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22640000 2016-04-07T02:05:53.672-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22650000 2016-04-07T02:05:53.751-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21318976, 21318976, 0 committed:false 2016-04-07T02:05:53.752-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21318976, 21318976, 0 committed:false) SnapCreateDur 27.752µs SnapOpenDur 1.08276ms 2016-04-07T02:05:53.911-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22660000 2016-04-07T02:05:54.082-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21320000 QueuedCount 7241 2016-04-07T02:05:54.134-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22670000 2016-04-07T02:05:54.36-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22680000 2016-04-07T02:05:54.594-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22690000 2016-04-07T02:05:54.817-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22700000 2016-04-07T02:05:54.861-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 30.554µs CommitTime 525.317367ms TotalFlushTime 16m0.342645309s TotalCommitTime 1m45.087357323s 2016-04-07T02:05:54.861-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21327241, 21327241, 164 committed:true 2016-04-07T02:05:54.861-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21327241, 21327241, 164 committed:true) SnapCreateDur 535.123786ms SnapOpenDur 47.506µs 2016-04-07T02:05:54.866-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:54.87-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:54.875-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:55-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21330000 QueuedCount 33104 2016-04-07T02:05:55.044-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22710000 2016-04-07T02:05:55.269-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22720000 2016-04-07T02:05:55.328-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21340000 QueuedCount 23104 2016-04-07T02:05:55.494-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22730000 2016-04-07T02:05:55.672-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21350000 QueuedCount 13104 2016-04-07T02:05:55.72-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22740000 2016-04-07T02:05:55.943-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22750000 2016-04-07T02:05:55.983-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21360000 QueuedCount 3104 2016-04-07T02:05:56.129-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21363104, 21363104, 0 committed:false 2016-04-07T02:05:56.13-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21363104, 21363104, 0 committed:false) SnapCreateDur 44.564µs SnapOpenDur 940.26µs 2016-04-07T02:05:56.136-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:56.141-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:56.145-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:56.15-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:56.156-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:56.173-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22760000 2016-04-07T02:05:56.4-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22770000 2016-04-07T02:05:56.441-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21370000 QueuedCount 46899 2016-04-07T02:05:56.625-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22780000 2016-04-07T02:05:56.789-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21380000 QueuedCount 36899 2016-04-07T02:05:56.856-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22790000 2016-04-07T02:05:57.083-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22800000 2016-04-07T02:05:57.147-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21390000 QueuedCount 26899 2016-04-07T02:05:57.308-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22810000 2016-04-07T02:05:57.53-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21400000 QueuedCount 16899 2016-04-07T02:05:57.54-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22820000 2016-04-07T02:05:57.767-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22830000 2016-04-07T02:05:57.917-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21410000 QueuedCount 6899 2016-04-07T02:05:57.994-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22840000 2016-04-07T02:05:58.168-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21416899, 21416899, 0 committed:false 2016-04-07T02:05:58.17-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21416899, 21416899, 0 committed:false) SnapCreateDur 37.896µs SnapOpenDur 2.841399ms 2016-04-07T02:05:58.176-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:58.181-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:58.186-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:58.191-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:58.196-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:05:58.22-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22850000 2016-04-07T02:05:58.453-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22860000 2016-04-07T02:05:58.527-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21420000 QueuedCount 50000 2016-04-07T02:05:58.683-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22870000 2016-04-07T02:05:58.917-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22880000 2016-04-07T02:05:58.924-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21430000 QueuedCount 40923 2016-04-07T02:05:59.15-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22890000 2016-04-07T02:05:59.327-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21440000 QueuedCount 30923 2016-04-07T02:05:59.375-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22900000 2016-04-07T02:05:59.602-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22910000 2016-04-07T02:05:59.678-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21450000 QueuedCount 20923 2016-04-07T02:05:59.835-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22920000 2016-04-07T02:06:00.07-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22930000 2016-04-07T02:06:00.106-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21460000 QueuedCount 10923 2016-04-07T02:06:00.3-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22940000 2016-04-07T02:06:00.542-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22950000 2016-04-07T02:06:00.543-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21470000 QueuedCount 923 2016-04-07T02:06:00.628-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21470923, 21470923, 0 committed:false 2016-04-07T02:06:00.628-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21470923, 21470923, 0 committed:false) SnapCreateDur 29.514µs SnapOpenDur 271.26µs 2016-04-07T02:06:00.769-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22960000 2016-04-07T02:06:01.002-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22970000 2016-04-07T02:06:01.227-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22980000 2016-04-07T02:06:01.459-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 22990000 2016-04-07T02:06:01.689-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23000000 2016-04-07T02:06:01.93-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23010000 2016-04-07T02:06:02.087-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.691µs CommitTime 654.917394ms TotalFlushTime 16m6.789577586s TotalCommitTime 1m45.742274717s 2016-04-07T02:06:02.087-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21479598, 21479598, 165 committed:true 2016-04-07T02:06:02.087-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21479598, 21479598, 165 committed:true) SnapCreateDur 664.321365ms SnapOpenDur 44.117µs 2016-04-07T02:06:02.093-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:02.095-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21480000 QueuedCount 13488 2016-04-07T02:06:02.097-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:02.103-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:02.108-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:02.112-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:02.164-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23020000 2016-04-07T02:06:02.401-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23030000 2016-04-07T02:06:02.639-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23040000 2016-04-07T02:06:02.66-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21490000 QueuedCount 42563 2016-04-07T02:06:02.879-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23050000 2016-04-07T02:06:02.963-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21500000 QueuedCount 32563 2016-04-07T02:06:03.115-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23060000 2016-04-07T02:06:03.328-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21510000 QueuedCount 22563 2016-04-07T02:06:03.347-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23070000 2016-04-07T02:06:03.584-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23080000 2016-04-07T02:06:03.698-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21520000 QueuedCount 12563 2016-04-07T02:06:03.817-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23090000 2016-04-07T02:06:04.056-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23100000 2016-04-07T02:06:04.076-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21530000 QueuedCount 2563 2016-04-07T02:06:04.169-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21532563, 21532563, 0 committed:false 2016-04-07T02:06:04.171-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21532563, 21532563, 0 committed:false) SnapCreateDur 31.999µs SnapOpenDur 1.862415ms 2016-04-07T02:06:04.175-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:04.179-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:04.183-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:04.188-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:04.191-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:04.298-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23110000 2016-04-07T02:06:04.545-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23120000 2016-04-07T02:06:04.59-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21540000 QueuedCount 40845 2016-04-07T02:06:04.779-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23130000 2016-04-07T02:06:04.969-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21550000 QueuedCount 30845 2016-04-07T02:06:05.017-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23140000 2016-04-07T02:06:05.244-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23150000 2016-04-07T02:06:05.35-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21560000 QueuedCount 20845 2016-04-07T02:06:05.476-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23160000 2016-04-07T02:06:05.708-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23170000 2016-04-07T02:06:05.725-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21570000 QueuedCount 10845 2016-04-07T02:06:05.94-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23180000 2016-04-07T02:06:06.105-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21580000 QueuedCount 845 2016-04-07T02:06:06.114-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21580845, 21580845, 0 committed:false 2016-04-07T02:06:06.115-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21580845, 21580845, 0 committed:false) SnapCreateDur 30.98µs SnapOpenDur 894.151µs 2016-04-07T02:06:06.169-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23190000 2016-04-07T02:06:06.395-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23200000 2016-04-07T02:06:06.633-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23210000 2016-04-07T02:06:06.657-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21590000 QueuedCount 4960 2016-04-07T02:06:06.87-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23220000 2016-04-07T02:06:07.104-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23230000 2016-04-07T02:06:07.339-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23240000 2016-04-07T02:06:07.43-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 34.09µs CommitTime 538.251839ms TotalFlushTime 16m11.505596362s TotalCommitTime 1m46.280526556s 2016-04-07T02:06:07.43-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21594960, 21594960, 166 committed:true 2016-04-07T02:06:07.43-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21594960, 21594960, 166 committed:true) SnapCreateDur 547.286038ms SnapOpenDur 37.882µs 2016-04-07T02:06:07.434-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:07.439-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:07.443-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:07.447-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:07.451-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:07.581-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23250000 2016-04-07T02:06:07.764-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21600000 QueuedCount 48003 2016-04-07T02:06:07.824-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23260000 2016-04-07T02:06:08.06-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23270000 2016-04-07T02:06:08.098-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21610000 QueuedCount 38003 2016-04-07T02:06:08.3-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23280000 2016-04-07T02:06:08.443-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21620000 QueuedCount 28003 2016-04-07T02:06:08.542-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23290000 2016-04-07T02:06:08.781-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23300000 2016-04-07T02:06:08.792-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21630000 QueuedCount 18003 2016-04-07T02:06:09.023-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23310000 2016-04-07T02:06:09.096-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21640000 QueuedCount 8003 2016-04-07T02:06:09.258-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23320000 2016-04-07T02:06:09.373-07:00 [Info] connected with 1 indexers 2016-04-07T02:06:09.373-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:06:09.374-07:00 [Info] client load stats {} 2016-04-07T02:06:09.387-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21648003, 21648003, 0 committed:false 2016-04-07T02:06:09.389-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21648003, 21648003, 0 committed:false) SnapCreateDur 41.517µs SnapOpenDur 1.820839ms 2016-04-07T02:06:09.393-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:09.397-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:09.401-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:09.405-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:09.41-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:09.484-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23330000 2016-04-07T02:06:09.523-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21650000 QueuedCount 50000 2016-04-07T02:06:09.65-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":1635382,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":1791205574,"bucket-1:myindex:avg_ts_items_count":41153,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":5772840586,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":15718506496,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":49273,"bucket-1:myindex:frag_percent":63,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":2262885975,"bucket-1:myindex:items_count":21479598,"bucket-1:myindex:num_commits":166,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":21652317,"bucket-1:myindex:num_docs_pending":1160,"bucket-1:myindex:num_docs_processed":21576266,"bucket-1:myindex:num_docs_queued":1561550,"bucket-1:myindex:num_flush_queued":21701590,"bucket-1:myindex:num_items_flushed":21652317,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":216,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":1958978801,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"166 106280526556 8088882104738086750","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"21652318 128399171700 749563442583604736","bucket-1:myindex:timings/storage_info":"765 6018849 88748730205","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"165 57603639 22348498845695","bucket-1:myindex:timings/storage_meta_set":"166 57066052 21308592262246","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"166 7011139 324639329915","bucket-1:myindex:timings/storage_set":"43304635 833645552565 -3078839789504198131","bucket-1:myindex:timings/storage_snapshot_close":"49 39527516 90840328171130","bucket-1:myindex:timings/storage_snapshot_create":"50 47482029 59876813078549","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":23336972,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"309 330596314 2288900093401804","bucket-1:ts_queue_size":186,"indexer_state":"Active","memory_quota":31457280000,"memory_used":16958613504,"memory_used_storage":15718924288,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"3889 677716688 251495749710678","uptime":"37m1.34345163s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:06:09.679-07:00 [Info] memstats {"Alloc":798338576, "TotalAlloc":21155539768, "Sys":9854296392, "Lookups":8638, "Mallocs":592847429,"Frees":569654881, "HeapAlloc":798338576, "HeapSys":9383051264, "HeapIdle":8440807424, "HeapInuse":942243840,"HeapReleased":8315166720, "HeapObjects":23192548,"GCSys":297445376, "LastGC":1460019917767945383,"PauseTotalNs":37631940, "PauseNs":[1343147, 1340988, 1301721], "NumGC":31} 2016-04-07T02:06:09.69-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5800964977, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:63.09468091974124%, mode : circular 2016-04-07T02:06:09.69-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:06:09.738-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23340000 2016-04-07T02:06:09.898-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21660000 QueuedCount 41590 2016-04-07T02:06:09.963-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23350000 2016-04-07T02:06:10.194-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23360000 2016-04-07T02:06:10.281-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21670000 QueuedCount 31590 2016-04-07T02:06:10.428-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23370000 2016-04-07T02:06:10.663-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23380000 2016-04-07T02:06:10.684-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21680000 QueuedCount 21590 2016-04-07T02:06:10.904-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23390000 2016-04-07T02:06:11.082-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21690000 QueuedCount 11590 2016-04-07T02:06:11.14-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23400000 2016-04-07T02:06:11.377-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23410000 2016-04-07T02:06:11.47-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21700000 QueuedCount 1590 2016-04-07T02:06:11.551-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21701590, 21701590, 0 committed:false 2016-04-07T02:06:11.551-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21701590, 21701590, 0 committed:false) SnapCreateDur 29.022µs SnapOpenDur 202.085µs 2016-04-07T02:06:11.606-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23420000 2016-04-07T02:06:11.839-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23430000 2016-04-07T02:06:12.069-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23440000 2016-04-07T02:06:12.093-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21710000 QueuedCount 713 2016-04-07T02:06:12.302-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23450000 2016-04-07T02:06:12.534-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23460000 2016-04-07T02:06:12.646-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.259µs CommitTime 531.735859ms TotalFlushTime 16m16.107322874s TotalCommitTime 1m46.812262415s 2016-04-07T02:06:12.647-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21710713, 21710713, 167 committed:true 2016-04-07T02:06:12.647-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21710713, 21710713, 167 committed:true) SnapCreateDur 541.256653ms SnapOpenDur 56.303µs 2016-04-07T02:06:12.651-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:12.655-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:12.659-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:12.663-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:12.668-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:12.78-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23470000 2016-04-07T02:06:13.004-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23480000 2016-04-07T02:06:13.099-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21720000 QueuedCount 44184 2016-04-07T02:06:13.238-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23490000 2016-04-07T02:06:13.455-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21730000 QueuedCount 34184 2016-04-07T02:06:13.464-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23500000 2016-04-07T02:06:13.697-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23510000 2016-04-07T02:06:13.821-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21740000 QueuedCount 24184 2016-04-07T02:06:13.928-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23520000 2016-04-07T02:06:14.158-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23530000 2016-04-07T02:06:14.201-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21750000 QueuedCount 14184 2016-04-07T02:06:14.388-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23540000 2016-04-07T02:06:14.608-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21760000 QueuedCount 4184 2016-04-07T02:06:14.619-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23550000 2016-04-07T02:06:14.785-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21764184, 21764184, 0 committed:false 2016-04-07T02:06:14.785-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21764184, 21764184, 0 committed:false) SnapCreateDur 29.866µs SnapOpenDur 233.192µs 2016-04-07T02:06:14.789-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:14.792-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:14.796-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:14.799-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:14.803-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:14.859-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23560000 2016-04-07T02:06:15.074-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21770000 QueuedCount 47192 2016-04-07T02:06:15.088-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23570000 2016-04-07T02:06:15.323-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23580000 2016-04-07T02:06:15.443-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21780000 QueuedCount 37192 2016-04-07T02:06:15.555-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23590000 2016-04-07T02:06:15.789-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23600000 2016-04-07T02:06:15.81-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21790000 QueuedCount 27192 2016-04-07T02:06:16.018-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23610000 2016-04-07T02:06:16.191-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21800000 QueuedCount 17192 2016-04-07T02:06:16.248-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23620000 2016-04-07T02:06:16.475-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23630000 2016-04-07T02:06:16.573-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21810000 QueuedCount 7192 2016-04-07T02:06:16.714-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23640000 2016-04-07T02:06:16.831-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21817192, 21817192, 0 committed:false 2016-04-07T02:06:16.833-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21817192, 21817192, 0 committed:false) SnapCreateDur 28.197µs SnapOpenDur 2.072335ms 2016-04-07T02:06:16.95-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23650000 2016-04-07T02:06:17.184-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23660000 2016-04-07T02:06:17.217-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21820000 QueuedCount 5940 2016-04-07T02:06:17.419-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23670000 2016-04-07T02:06:17.655-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23680000 2016-04-07T02:06:17.886-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23690000 2016-04-07T02:06:17.992-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 33.509µs CommitTime 524.309947ms TotalFlushTime 16m20.849824421s TotalCommitTime 1m47.336572362s 2016-04-07T02:06:17.992-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21825940, 21825940, 168 committed:true 2016-04-07T02:06:17.992-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21825940, 21825940, 168 committed:true) SnapCreateDur 534.77977ms SnapOpenDur 32.424µs 2016-04-07T02:06:17.996-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:17.999-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:18.002-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:18.006-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:18.01-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:18.136-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23700000 2016-04-07T02:06:18.261-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21830000 QueuedCount 48025 2016-04-07T02:06:18.365-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23710000 2016-04-07T02:06:18.581-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21840000 QueuedCount 38025 2016-04-07T02:06:18.59-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23720000 2016-04-07T02:06:18.818-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23730000 2016-04-07T02:06:18.916-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21850000 QueuedCount 28025 2016-04-07T02:06:19.055-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23740000 2016-04-07T02:06:19.267-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21860000 QueuedCount 18025 2016-04-07T02:06:19.274-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23750000 2016-04-07T02:06:19.517-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23760000 2016-04-07T02:06:19.622-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21870000 QueuedCount 8025 2016-04-07T02:06:19.74-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23770000 2016-04-07T02:06:19.912-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21878025, 21878025, 0 committed:false 2016-04-07T02:06:19.913-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21878025, 21878025, 0 committed:false) SnapCreateDur 52.741µs SnapOpenDur 811.047µs 2016-04-07T02:06:19.918-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:19.922-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:19.926-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:19.93-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:19.934-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:19.964-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23780000 2016-04-07T02:06:20.038-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21880000 QueuedCount 50000 2016-04-07T02:06:20.184-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23790000 2016-04-07T02:06:20.391-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21890000 QueuedCount 41385 2016-04-07T02:06:20.413-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23800000 2016-04-07T02:06:20.633-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23810000 2016-04-07T02:06:20.752-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21900000 QueuedCount 31385 2016-04-07T02:06:20.859-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23820000 2016-04-07T02:06:21.098-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23830000 2016-04-07T02:06:21.113-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21910000 QueuedCount 21385 2016-04-07T02:06:21.315-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23840000 2016-04-07T02:06:21.43-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21920000 QueuedCount 11385 2016-04-07T02:06:21.538-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23850000 2016-04-07T02:06:21.768-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23860000 2016-04-07T02:06:21.816-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21930000 QueuedCount 1385 2016-04-07T02:06:21.893-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21931385, 21931385, 0 committed:false 2016-04-07T02:06:21.893-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21931385, 21931385, 0 committed:false) SnapCreateDur 51.495µs SnapOpenDur 908.429µs 2016-04-07T02:06:21.989-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23870000 2016-04-07T02:06:22.213-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23880000 2016-04-07T02:06:22.383-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21940000 QueuedCount 501 2016-04-07T02:06:22.467-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23890000 2016-04-07T02:06:22.752-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23900000 2016-04-07T02:06:22.894-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23910000 2016-04-07T02:06:23.015-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 34.397µs CommitTime 558.05086ms TotalFlushTime 16m25.171712388s TotalCommitTime 1m47.894623222s 2016-04-07T02:06:23.015-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21940501, 21940501, 169 committed:true 2016-04-07T02:06:23.015-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21940501, 21940501, 169 committed:true) SnapCreateDur 620.414283ms SnapOpenDur 56.294µs 2016-04-07T02:06:23.019-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:23.023-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:23.027-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:23.031-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:23.035-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:23.118-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23920000 2016-04-07T02:06:23.348-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23930000 2016-04-07T02:06:23.368-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21950000 QueuedCount 44184 2016-04-07T02:06:23.569-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23940000 2016-04-07T02:06:23.711-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21960000 QueuedCount 34184 2016-04-07T02:06:23.79-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23950000 2016-04-07T02:06:24.023-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23960000 2016-04-07T02:06:24.073-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21970000 QueuedCount 24184 2016-04-07T02:06:24.262-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23970000 2016-04-07T02:06:24.437-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21980000 QueuedCount 14184 2016-04-07T02:06:24.49-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23980000 2016-04-07T02:06:24.72-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 23990000 2016-04-07T02:06:24.797-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 21990000 QueuedCount 4184 2016-04-07T02:06:24.942-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24000000 2016-04-07T02:06:24.969-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 21994184, 21994184, 0 committed:false 2016-04-07T02:06:24.97-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 21994184, 21994184, 0 committed:false) SnapCreateDur 30.269µs SnapOpenDur 421.456µs 2016-04-07T02:06:24.974-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:24.977-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:24.981-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:24.984-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:24.988-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:25.084-07:00 [Info] Indexer::ReadMemstats Time Taken 12.881343ms 2016-04-07T02:06:25.167-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24010000 2016-04-07T02:06:25.263-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22000000 QueuedCount 46611 2016-04-07T02:06:25.396-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24020000 2016-04-07T02:06:25.575-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22010000 QueuedCount 36611 2016-04-07T02:06:25.622-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24030000 2016-04-07T02:06:25.84-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24040000 2016-04-07T02:06:25.951-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22020000 QueuedCount 26611 2016-04-07T02:06:26.068-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24050000 2016-04-07T02:06:26.297-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24060000 2016-04-07T02:06:26.329-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22030000 QueuedCount 16611 2016-04-07T02:06:26.528-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24070000 2016-04-07T02:06:26.708-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22040000 QueuedCount 6611 2016-04-07T02:06:26.751-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24080000 2016-04-07T02:06:26.948-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22046611, 22046611, 0 committed:false 2016-04-07T02:06:26.95-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22046611, 22046611, 0 committed:false) SnapCreateDur 25.842µs SnapOpenDur 1.560412ms 2016-04-07T02:06:26.978-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24090000 2016-04-07T02:06:27.2-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24100000 2016-04-07T02:06:27.206-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22050000 QueuedCount 5613 2016-04-07T02:06:27.421-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24110000 2016-04-07T02:06:27.647-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24120000 2016-04-07T02:06:27.864-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24130000 2016-04-07T02:06:27.918-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.943µs CommitTime 454.993911ms TotalFlushTime 16m29.527598723s TotalCommitTime 1m48.349617133s 2016-04-07T02:06:27.918-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22055613, 22055613, 170 committed:true 2016-04-07T02:06:27.918-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22055613, 22055613, 170 committed:true) SnapCreateDur 464.466157ms SnapOpenDur 58.533µs 2016-04-07T02:06:27.922-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:27.926-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:27.93-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:27.934-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:27.937-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:28.087-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24140000 2016-04-07T02:06:28.121-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22060000 QueuedCount 48978 2016-04-07T02:06:28.309-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24150000 2016-04-07T02:06:28.44-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22070000 QueuedCount 38978 2016-04-07T02:06:28.531-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24160000 2016-04-07T02:06:28.724-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22080000 QueuedCount 28978 2016-04-07T02:06:28.76-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24170000 2016-04-07T02:06:28.977-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24180000 2016-04-07T02:06:29.054-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22090000 QueuedCount 18978 2016-04-07T02:06:29.203-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24190000 2016-04-07T02:06:29.409-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22100000 QueuedCount 8978 2016-04-07T02:06:29.439-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24200000 2016-04-07T02:06:29.657-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24210000 2016-04-07T02:06:29.76-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22108978, 22108978, 0 committed:false 2016-04-07T02:06:29.76-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22108978, 22108978, 0 committed:false) SnapCreateDur 29.032µs SnapOpenDur 151.484µs 2016-04-07T02:06:29.764-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:29.768-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:29.772-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:29.776-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:29.78-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22110000 QueuedCount 43157 2016-04-07T02:06:29.78-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:29.893-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24220000 2016-04-07T02:06:30.118-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24230000 2016-04-07T02:06:30.252-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22120000 QueuedCount 42221 2016-04-07T02:06:30.338-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24240000 2016-04-07T02:06:30.564-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24250000 2016-04-07T02:06:30.619-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22130000 QueuedCount 32221 2016-04-07T02:06:30.788-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24260000 2016-04-07T02:06:31.005-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22140000 QueuedCount 22221 2016-04-07T02:06:31.009-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24270000 2016-04-07T02:06:31.237-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24280000 2016-04-07T02:06:31.398-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22150000 QueuedCount 12221 2016-04-07T02:06:31.46-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24290000 2016-04-07T02:06:31.68-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24300000 2016-04-07T02:06:31.722-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22160000 QueuedCount 2221 2016-04-07T02:06:31.864-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22162221, 22162221, 0 committed:false 2016-04-07T02:06:31.864-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22162221, 22162221, 0 committed:false) SnapCreateDur 38.318µs SnapOpenDur 151.735µs 2016-04-07T02:06:31.868-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:31.872-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:31.876-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:31.88-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:31.884-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:31.903-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24310000 2016-04-07T02:06:32.125-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24320000 2016-04-07T02:06:32.292-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22170000 QueuedCount 39890 2016-04-07T02:06:32.343-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24330000 2016-04-07T02:06:32.563-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24340000 2016-04-07T02:06:32.673-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22180000 QueuedCount 29890 2016-04-07T02:06:32.779-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24350000 2016-04-07T02:06:33-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24360000 2016-04-07T02:06:33.062-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22190000 QueuedCount 19890 2016-04-07T02:06:33.218-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24370000 2016-04-07T02:06:33.443-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24380000 2016-04-07T02:06:33.459-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22200000 QueuedCount 9890 2016-04-07T02:06:33.654-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24390000 2016-04-07T02:06:33.866-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22209890, 22209890, 0 committed:false 2016-04-07T02:06:33.866-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22209890, 22209890, 0 committed:false) SnapCreateDur 56.733µs SnapOpenDur 568.48µs 2016-04-07T02:06:33.869-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22210000 QueuedCount 6079 2016-04-07T02:06:33.871-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24400000 2016-04-07T02:06:34.092-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24410000 2016-04-07T02:06:34.468-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24420000 2016-04-07T02:06:34.497-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22220000 QueuedCount 3971 2016-04-07T02:06:34.7-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24430000 2016-04-07T02:06:34.952-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24440000 2016-04-07T02:06:35.165-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.209µs CommitTime 503.98899ms TotalFlushTime 16m36.15539673s TotalCommitTime 1m48.853606123s 2016-04-07T02:06:35.165-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22223971, 22223971, 171 committed:true 2016-04-07T02:06:35.165-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22223971, 22223971, 171 committed:true) SnapCreateDur 513.445892ms SnapOpenDur 60.479µs 2016-04-07T02:06:35.169-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:35.173-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:35.176-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:35.18-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:35.183-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:35.346-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24450000 2016-04-07T02:06:35.53-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22230000 QueuedCount 47308 2016-04-07T02:06:35.564-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24460000 2016-04-07T02:06:35.792-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24470000 2016-04-07T02:06:35.868-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22240000 QueuedCount 37308 2016-04-07T02:06:36.021-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24480000 2016-04-07T02:06:36.219-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22250000 QueuedCount 27308 2016-04-07T02:06:36.24-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24490000 2016-04-07T02:06:36.458-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24500000 2016-04-07T02:06:36.57-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22260000 QueuedCount 17308 2016-04-07T02:06:36.685-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24510000 2016-04-07T02:06:36.91-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24520000 2016-04-07T02:06:36.937-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22270000 QueuedCount 7308 2016-04-07T02:06:37.15-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24530000 2016-04-07T02:06:37.229-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22277308, 22277308, 0 committed:false 2016-04-07T02:06:37.231-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22277308, 22277308, 0 committed:false) SnapCreateDur 37.522µs SnapOpenDur 1.625602ms 2016-04-07T02:06:37.235-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:37.239-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:37.244-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:37.248-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:37.252-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:37.375-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24540000 2016-04-07T02:06:37.384-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22280000 QueuedCount 50000 2016-04-07T02:06:37.598-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24550000 2016-04-07T02:06:37.737-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22290000 QueuedCount 41053 2016-04-07T02:06:37.818-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24560000 2016-04-07T02:06:38.044-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24570000 2016-04-07T02:06:38.097-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22300000 QueuedCount 31053 2016-04-07T02:06:38.262-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24580000 2016-04-07T02:06:38.409-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22310000 QueuedCount 21053 2016-04-07T02:06:38.479-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24590000 2016-04-07T02:06:38.695-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24600000 2016-04-07T02:06:38.775-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22320000 QueuedCount 11053 2016-04-07T02:06:38.92-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24610000 2016-04-07T02:06:39.144-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22330000 QueuedCount 1053 2016-04-07T02:06:39.162-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24620000 2016-04-07T02:06:39.218-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22331053, 22331053, 0 committed:false 2016-04-07T02:06:39.218-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22331053, 22331053, 0 committed:false) SnapCreateDur 28.903µs SnapOpenDur 563.59µs 2016-04-07T02:06:39.38-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24630000 2016-04-07T02:06:39.608-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24640000 2016-04-07T02:06:39.691-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:5986458679, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:61.914583420992216%, mode : circular 2016-04-07T02:06:39.691-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:06:39.832-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24650000 2016-04-07T02:06:40.05-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24660000 2016-04-07T02:06:40.175-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.172µs CommitTime 453.521242ms TotalFlushTime 16m40.627545723s TotalCommitTime 1m49.307127365s 2016-04-07T02:06:40.175-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22339972, 22339972, 172 committed:true 2016-04-07T02:06:40.175-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22339972, 22339972, 172 committed:true) SnapCreateDur 462.543209ms SnapOpenDur 54.413µs 2016-04-07T02:06:40.176-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22340000 QueuedCount 68 2016-04-07T02:06:40.18-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:40.185-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:40.19-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:40.194-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:40.198-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:40.287-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24670000 2016-04-07T02:06:40.512-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24680000 2016-04-07T02:06:40.519-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22350000 QueuedCount 43563 2016-04-07T02:06:40.733-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24690000 2016-04-07T02:06:40.845-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22360000 QueuedCount 33563 2016-04-07T02:06:40.962-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24700000 2016-04-07T02:06:41.179-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24710000 2016-04-07T02:06:41.181-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22370000 QueuedCount 23563 2016-04-07T02:06:41.402-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24720000 2016-04-07T02:06:41.535-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22380000 QueuedCount 13563 2016-04-07T02:06:41.631-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24730000 2016-04-07T02:06:41.874-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24740000 2016-04-07T02:06:41.899-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22390000 QueuedCount 3563 2016-04-07T02:06:42.056-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22393563, 22393563, 0 committed:false 2016-04-07T02:06:42.056-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22393563, 22393563, 0 committed:false) SnapCreateDur 44.576µs SnapOpenDur 365.03µs 2016-04-07T02:06:42.06-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:42.063-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:42.067-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:42.071-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:42.074-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:42.097-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24750000 2016-04-07T02:06:42.318-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24760000 2016-04-07T02:06:42.407-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22400000 QueuedCount 45861 2016-04-07T02:06:42.553-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24770000 2016-04-07T02:06:42.768-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24780000 2016-04-07T02:06:42.792-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22410000 QueuedCount 35861 2016-04-07T02:06:42.993-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24790000 2016-04-07T02:06:43.172-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22420000 QueuedCount 25861 2016-04-07T02:06:43.209-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24800000 2016-04-07T02:06:43.434-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24810000 2016-04-07T02:06:43.501-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22430000 QueuedCount 15861 2016-04-07T02:06:43.658-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24820000 2016-04-07T02:06:43.89-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24830000 2016-04-07T02:06:43.916-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22440000 QueuedCount 5861 2016-04-07T02:06:44.115-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24840000 2016-04-07T02:06:44.155-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22445861, 22445861, 0 committed:false 2016-04-07T02:06:44.156-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22445861, 22445861, 0 committed:false) SnapCreateDur 69.493µs SnapOpenDur 1.390712ms 2016-04-07T02:06:44.16-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:44.163-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:44.167-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:44.171-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:44.174-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:44.335-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24850000 2016-04-07T02:06:44.52-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22450000 QueuedCount 48972 2016-04-07T02:06:44.56-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24860000 2016-04-07T02:06:44.789-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24870000 2016-04-07T02:06:44.92-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22460000 QueuedCount 38972 2016-04-07T02:06:45.012-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24880000 2016-04-07T02:06:45.238-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24890000 2016-04-07T02:06:45.319-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22470000 QueuedCount 28972 2016-04-07T02:06:45.482-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24900000 2016-04-07T02:06:45.716-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22480000 QueuedCount 18972 2016-04-07T02:06:45.769-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24910000 2016-04-07T02:06:45.992-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24920000 2016-04-07T02:06:46.117-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22490000 QueuedCount 8972 2016-04-07T02:06:46.218-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24930000 2016-04-07T02:06:46.433-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24940000 2016-04-07T02:06:46.449-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22498972, 22498972, 0 committed:false 2016-04-07T02:06:46.451-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22498972, 22498972, 0 committed:false) SnapCreateDur 26µs SnapOpenDur 1.192528ms 2016-04-07T02:06:46.658-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24950000 2016-04-07T02:06:46.781-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22500000 QueuedCount 7547 2016-04-07T02:06:46.884-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24960000 2016-04-07T02:06:47.109-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24970000 2016-04-07T02:06:47.33-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24980000 2016-04-07T02:06:47.556-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 24990000 2016-04-07T02:06:47.583-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.282µs CommitTime 548.174776ms TotalFlushTime 16m47.364007681s TotalCommitTime 1m49.855302141s 2016-04-07T02:06:47.583-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22507547, 22507547, 173 committed:true 2016-04-07T02:06:47.583-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22507547, 22507547, 173 committed:true) SnapCreateDur 557.986475ms SnapOpenDur 46.559µs 2016-04-07T02:06:47.587-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:47.592-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:47.596-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:47.6-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:47.604-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:47.792-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25000000 2016-04-07T02:06:48.21-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22510000 QueuedCount 50000 2016-04-07T02:06:48.212-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25010000 2016-04-07T02:06:48.229-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25020000 2016-04-07T02:06:48.459-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25030000 2016-04-07T02:06:48.56-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22520000 QueuedCount 41025 2016-04-07T02:06:48.688-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25040000 2016-04-07T02:06:48.864-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22530000 QueuedCount 31025 2016-04-07T02:06:48.91-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25050000 2016-04-07T02:06:49.134-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25060000 2016-04-07T02:06:49.229-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22540000 QueuedCount 21025 2016-04-07T02:06:49.359-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25070000 2016-04-07T02:06:49.588-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25080000 2016-04-07T02:06:49.609-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22550000 QueuedCount 11025 2016-04-07T02:06:49.813-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25090000 2016-04-07T02:06:49.981-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22560000 QueuedCount 1025 2016-04-07T02:06:50.034-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25100000 2016-04-07T02:06:50.053-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22561025, 22561025, 0 committed:false 2016-04-07T02:06:50.053-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22561025, 22561025, 0 committed:false) SnapCreateDur 28.002µs SnapOpenDur 221.391µs 2016-04-07T02:06:50.058-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:50.062-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:50.067-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:50.072-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:50.077-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:50.259-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25110000 2016-04-07T02:06:50.482-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25120000 2016-04-07T02:06:50.489-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22570000 QueuedCount 45090 2016-04-07T02:06:50.7-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25130000 2016-04-07T02:06:50.866-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22580000 QueuedCount 35090 2016-04-07T02:06:50.924-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25140000 2016-04-07T02:06:51.143-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25150000 2016-04-07T02:06:51.232-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22590000 QueuedCount 25090 2016-04-07T02:06:51.364-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25160000 2016-04-07T02:06:51.593-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25170000 2016-04-07T02:06:51.606-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22600000 QueuedCount 15090 2016-04-07T02:06:51.81-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25180000 2016-04-07T02:06:51.981-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22610000 QueuedCount 5090 2016-04-07T02:06:52.039-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25190000 2016-04-07T02:06:52.152-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22615090, 22615090, 0 committed:false 2016-04-07T02:06:52.153-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22615090, 22615090, 0 committed:false) SnapCreateDur 27.57µs SnapOpenDur 1.245368ms 2016-04-07T02:06:52.258-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25200000 2016-04-07T02:06:52.48-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25210000 2016-04-07T02:06:52.51-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22620000 QueuedCount 3857 2016-04-07T02:06:52.699-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25220000 2016-04-07T02:06:52.927-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25230000 2016-04-07T02:06:53.138-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.85µs CommitTime 471.085257ms TotalFlushTime 16m52.353551576s TotalCommitTime 1m50.326387398s 2016-04-07T02:06:53.138-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22623857, 22623857, 174 committed:true 2016-04-07T02:06:53.138-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22623857, 22623857, 174 committed:true) SnapCreateDur 479.781678ms SnapOpenDur 56.671µs 2016-04-07T02:06:53.143-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:53.146-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25240000 2016-04-07T02:06:53.147-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:53.151-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:53.156-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:53.16-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:53.367-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25250000 2016-04-07T02:06:53.478-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22630000 QueuedCount 46813 2016-04-07T02:06:53.595-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25260000 2016-04-07T02:06:53.819-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25270000 2016-04-07T02:06:53.831-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22640000 QueuedCount 36813 2016-04-07T02:06:54.049-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25280000 2016-04-07T02:06:54.191-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22650000 QueuedCount 26813 2016-04-07T02:06:54.282-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25290000 2016-04-07T02:06:54.504-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25300000 2016-04-07T02:06:54.553-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22660000 QueuedCount 16813 2016-04-07T02:06:54.725-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25310000 2016-04-07T02:06:54.913-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22670000 QueuedCount 6813 2016-04-07T02:06:54.951-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25320000 2016-04-07T02:06:55.15-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22676813, 22676813, 0 committed:false 2016-04-07T02:06:55.151-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22676813, 22676813, 0 committed:false) SnapCreateDur 28.739µs SnapOpenDur 1.754038ms 2016-04-07T02:06:55.155-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:55.159-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:55.163-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:55.167-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:55.171-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:55.179-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25330000 2016-04-07T02:06:55.407-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25340000 2016-04-07T02:06:55.408-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22680000 QueuedCount 49597 2016-04-07T02:06:55.637-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25350000 2016-04-07T02:06:55.775-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22690000 QueuedCount 39597 2016-04-07T02:06:55.915-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25360000 2016-04-07T02:06:56.153-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22700000 QueuedCount 29597 2016-04-07T02:06:56.22-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25370000 2016-04-07T02:06:56.454-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25380000 2016-04-07T02:06:56.546-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22710000 QueuedCount 19597 2016-04-07T02:06:56.68-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25390000 2016-04-07T02:06:56.871-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22720000 QueuedCount 9597 2016-04-07T02:06:56.9-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25400000 2016-04-07T02:06:57.128-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25410000 2016-04-07T02:06:57.264-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22729597, 22729597, 0 committed:false 2016-04-07T02:06:57.265-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22729597, 22729597, 0 committed:false) SnapCreateDur 27.642µs SnapOpenDur 1.152508ms 2016-04-07T02:06:57.272-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22730000 QueuedCount 8440 2016-04-07T02:06:57.349-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25420000 2016-04-07T02:06:57.572-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25430000 2016-04-07T02:06:57.795-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25440000 2016-04-07T02:06:58.03-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25450000 2016-04-07T02:06:58.249-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25460000 2016-04-07T02:06:58.393-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 16.637µs CommitTime 573.202063ms TotalFlushTime 16m56.947738923s TotalCommitTime 1m50.899589461s 2016-04-07T02:06:58.393-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22738440, 22738440, 175 committed:true 2016-04-07T02:06:58.393-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22738440, 22738440, 175 committed:true) SnapCreateDur 582.827152ms SnapOpenDur 48.235µs 2016-04-07T02:06:58.398-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:58.403-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:58.408-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:58.413-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:58.418-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:06:58.424-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22740000 QueuedCount 50000 2016-04-07T02:06:58.474-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25470000 2016-04-07T02:06:58.695-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25480000 2016-04-07T02:06:58.912-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22750000 QueuedCount 41272 2016-04-07T02:06:58.924-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25490000 2016-04-07T02:06:59.146-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25500000 2016-04-07T02:06:59.235-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22760000 QueuedCount 31272 2016-04-07T02:06:59.373-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25510000 2016-04-07T02:06:59.571-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22770000 QueuedCount 21272 2016-04-07T02:06:59.604-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25520000 2016-04-07T02:06:59.835-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25530000 2016-04-07T02:06:59.919-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22780000 QueuedCount 11272 2016-04-07T02:07:00.062-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25540000 2016-04-07T02:07:00.289-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22790000 QueuedCount 1272 2016-04-07T02:07:00.29-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25550000 2016-04-07T02:07:00.304-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22791272, 22791272, 0 committed:false 2016-04-07T02:07:00.306-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22791272, 22791272, 0 committed:false) SnapCreateDur 29.195µs SnapOpenDur 1.438371ms 2016-04-07T02:07:00.31-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:00.315-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:00.319-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:00.323-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:00.327-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:00.514-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25560000 2016-04-07T02:07:00.739-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25570000 2016-04-07T02:07:00.775-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22800000 QueuedCount 44072 2016-04-07T02:07:00.964-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25580000 2016-04-07T02:07:01.084-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22810000 QueuedCount 34072 2016-04-07T02:07:01.183-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25590000 2016-04-07T02:07:01.401-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25600000 2016-04-07T02:07:01.463-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22820000 QueuedCount 24072 2016-04-07T02:07:01.625-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25610000 2016-04-07T02:07:01.848-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22830000 QueuedCount 14072 2016-04-07T02:07:01.849-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25620000 2016-04-07T02:07:02.095-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25630000 2016-04-07T02:07:02.246-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22840000 QueuedCount 4072 2016-04-07T02:07:02.329-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25640000 2016-04-07T02:07:02.413-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22844072, 22844072, 0 committed:false 2016-04-07T02:07:02.414-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22844072, 22844072, 0 committed:false) SnapCreateDur 74.773µs SnapOpenDur 1.385598ms 2016-04-07T02:07:02.547-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25650000 2016-04-07T02:07:02.763-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25660000 2016-04-07T02:07:02.763-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22850000 QueuedCount 2713 2016-04-07T02:07:02.98-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25670000 2016-04-07T02:07:03.203-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25680000 2016-04-07T02:07:03.377-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 28.282µs CommitTime 508.032881ms TotalFlushTime 17m1.329609976s TotalCommitTime 1m51.407622342s 2016-04-07T02:07:03.377-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22852713, 22852713, 176 committed:true 2016-04-07T02:07:03.377-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22852713, 22852713, 176 committed:true) SnapCreateDur 517.966146ms SnapOpenDur 50.594µs 2016-04-07T02:07:03.382-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:03.386-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:03.389-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:03.393-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:03.397-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:03.429-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25690000 2016-04-07T02:07:03.653-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25700000 2016-04-07T02:07:03.785-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22860000 QueuedCount 44850 2016-04-07T02:07:03.881-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25710000 2016-04-07T02:07:04.101-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25720000 2016-04-07T02:07:04.127-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22870000 QueuedCount 34850 2016-04-07T02:07:04.32-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25730000 2016-04-07T02:07:04.486-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22880000 QueuedCount 24850 2016-04-07T02:07:04.55-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25740000 2016-04-07T02:07:04.775-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25750000 2016-04-07T02:07:04.857-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22890000 QueuedCount 14850 2016-04-07T02:07:04.999-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25760000 2016-04-07T02:07:05.22-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25770000 2016-04-07T02:07:05.227-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22900000 QueuedCount 4850 2016-04-07T02:07:05.392-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22904850, 22904850, 0 committed:false 2016-04-07T02:07:05.393-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22904850, 22904850, 0 committed:false) SnapCreateDur 39.019µs SnapOpenDur 940.158µs 2016-04-07T02:07:05.398-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:05.402-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:05.406-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:05.41-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:05.413-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:05.445-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25780000 2016-04-07T02:07:05.665-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25790000 2016-04-07T02:07:05.667-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22910000 QueuedCount 46334 2016-04-07T02:07:05.902-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25800000 2016-04-07T02:07:06.048-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22920000 QueuedCount 36334 2016-04-07T02:07:06.127-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25810000 2016-04-07T02:07:06.417-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22930000 QueuedCount 26334 2016-04-07T02:07:06.422-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25820000 2016-04-07T02:07:06.772-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25830000 2016-04-07T02:07:06.773-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22940000 QueuedCount 16334 2016-04-07T02:07:06.998-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25840000 2016-04-07T02:07:07.149-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22950000 QueuedCount 6334 2016-04-07T02:07:07.218-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25850000 2016-04-07T02:07:07.385-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 22956334, 22956334, 0 committed:false 2016-04-07T02:07:07.386-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 22956334, 22956334, 0 committed:false) SnapCreateDur 30.275µs SnapOpenDur 947.639µs 2016-04-07T02:07:07.389-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:07.393-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:07.397-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:07.401-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:07.405-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:07.444-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25860000 2016-04-07T02:07:07.669-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25870000 2016-04-07T02:07:07.729-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22960000 QueuedCount 48293 2016-04-07T02:07:07.892-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25880000 2016-04-07T02:07:08.105-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22970000 QueuedCount 38293 2016-04-07T02:07:08.117-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25890000 2016-04-07T02:07:08.34-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25900000 2016-04-07T02:07:08.49-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22980000 QueuedCount 28293 2016-04-07T02:07:08.562-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25910000 2016-04-07T02:07:08.784-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25920000 2016-04-07T02:07:08.874-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 22990000 QueuedCount 18293 2016-04-07T02:07:09.019-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25930000 2016-04-07T02:07:09.198-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23000000 QueuedCount 8293 2016-04-07T02:07:09.237-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25940000 2016-04-07T02:07:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:07:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:07:09.374-07:00 [Info] client load stats {} 2016-04-07T02:07:09.459-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25950000 2016-04-07T02:07:09.519-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23008293, 23008293, 0 committed:false 2016-04-07T02:07:09.522-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23008293, 23008293, 0 committed:false) SnapCreateDur 30.628µs SnapOpenDur 2.263988ms 2016-04-07T02:07:09.651-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":2941875,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":1997943771,"bucket-1:myindex:avg_ts_items_count":48817,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":6162058301,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":15718506496,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":8239,"bucket-1:myindex:frag_percent":60,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":2413946010,"bucket-1:myindex:items_count":22852713,"bucket-1:myindex:num_commits":176,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":23008398,"bucket-1:myindex:num_docs_pending":880,"bucket-1:myindex:num_docs_processed":22999799,"bucket-1:myindex:num_docs_queued":2921138,"bucket-1:myindex:num_flush_queued":23016637,"bucket-1:myindex:num_items_flushed":23008398,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":250,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":2135538028,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"176 111407622342 -7713016466711573636","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"23008399 134144297063 752550244335439769","bucket-1:myindex:timings/storage_info":"853 6592288 96183914166","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"175 61604349 24172835532479","bucket-1:myindex:timings/storage_meta_set":"176 60910113 22898279776909","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"176 7356131 337096398607","bucket-1:myindex:timings/storage_set":"46016797 880984629661 422477585201070385","bucket-1:myindex:timings/storage_snapshot_close":"73 57192974 127291899219922","bucket-1:myindex:timings/storage_snapshot_create":"74 68159829 86852388128887","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":25958512,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"321 553611139 27022191871512225","bucket-1:ts_queue_size":337,"indexer_state":"Active","memory_quota":31457280000,"memory_used":17444165632,"memory_used_storage":15718912000,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"3973 695368927 255498354699409","uptime":"38m1.34433571s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:07:09.679-07:00 [Info] memstats {"Alloc":954869656, "TotalAlloc":24685843456, "Sys":9854296392, "Lookups":8717, "Mallocs":695300945,"Frees":668829743, "HeapAlloc":954869656, "HeapSys":9382297600, "HeapIdle":7954489344, "HeapInuse":1427808256,"HeapReleased":7400759296, "HeapObjects":26471202,"GCSys":297445376, "LastGC":1460019982580352988,"PauseTotalNs":44030692, "PauseNs":[1812276, 1630147, 1511018, 1445311], "NumGC":35} 2016-04-07T02:07:09.683-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25960000 2016-04-07T02:07:09.691-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:6167510983, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:60.76274177467503%, mode : circular 2016-04-07T02:07:09.691-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:07:09.914-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25970000 2016-04-07T02:07:09.938-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23010000 QueuedCount 6637 2016-04-07T02:07:10.143-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25980000 2016-04-07T02:07:10.381-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 25990000 2016-04-07T02:07:10.614-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26000000 2016-04-07T02:07:10.839-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26010000 2016-04-07T02:07:10.89-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 16.773µs CommitTime 613.262232ms TotalFlushTime 17m8.110702697s TotalCommitTime 1m52.020884574s 2016-04-07T02:07:10.89-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23016637, 23016637, 177 committed:true 2016-04-07T02:07:10.89-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23016637, 23016637, 177 committed:true) SnapCreateDur 623.629659ms SnapOpenDur 49.339µs 2016-04-07T02:07:10.895-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:10.899-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:10.903-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:10.908-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:10.912-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:11.05-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23020000 QueuedCount 47382 2016-04-07T02:07:11.065-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26020000 2016-04-07T02:07:11.287-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26030000 2016-04-07T02:07:11.38-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23030000 QueuedCount 37382 2016-04-07T02:07:11.513-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26040000 2016-04-07T02:07:11.727-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23040000 QueuedCount 27382 2016-04-07T02:07:11.74-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26050000 2016-04-07T02:07:11.979-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26060000 2016-04-07T02:07:12.077-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23050000 QueuedCount 17382 2016-04-07T02:07:12.206-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26070000 2016-04-07T02:07:12.439-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26080000 2016-04-07T02:07:12.45-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23060000 QueuedCount 7382 2016-04-07T02:07:12.665-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26090000 2016-04-07T02:07:12.697-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23067382, 23067382, 0 committed:false 2016-04-07T02:07:12.698-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23067382, 23067382, 0 committed:false) SnapCreateDur 52.507µs SnapOpenDur 1.492475ms 2016-04-07T02:07:12.702-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:12.706-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:12.71-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:12.714-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:12.718-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:12.877-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23070000 QueuedCount 47368 2016-04-07T02:07:12.899-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26100000 2016-04-07T02:07:13.118-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26110000 2016-04-07T02:07:13.18-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23080000 QueuedCount 37368 2016-04-07T02:07:13.334-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26120000 2016-04-07T02:07:13.541-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23090000 QueuedCount 27368 2016-04-07T02:07:13.55-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26130000 2016-04-07T02:07:13.77-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26140000 2016-04-07T02:07:13.91-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23100000 QueuedCount 17368 2016-04-07T02:07:13.987-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26150000 2016-04-07T02:07:14.218-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26160000 2016-04-07T02:07:14.29-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23110000 QueuedCount 7368 2016-04-07T02:07:14.444-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26170000 2016-04-07T02:07:14.612-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23117368, 23117368, 0 committed:false 2016-04-07T02:07:14.612-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23117368, 23117368, 0 committed:false) SnapCreateDur 54.005µs SnapOpenDur 428.002µs 2016-04-07T02:07:14.673-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26180000 2016-04-07T02:07:14.787-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23120000 QueuedCount 5789 2016-04-07T02:07:14.9-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26190000 2016-04-07T02:07:15.125-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26200000 2016-04-07T02:07:15.347-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26210000 2016-04-07T02:07:15.481-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.253µs CommitTime 454.703595ms TotalFlushTime 17m12.15727357s TotalCommitTime 1m52.475588169s 2016-04-07T02:07:15.481-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23125789, 23125789, 178 committed:true 2016-04-07T02:07:15.481-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23125789, 23125789, 178 committed:true) SnapCreateDur 465.017833ms SnapOpenDur 32.471µs 2016-04-07T02:07:15.486-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:15.49-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:15.494-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:15.498-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:15.502-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:15.569-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26220000 2016-04-07T02:07:15.675-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23130000 QueuedCount 47523 2016-04-07T02:07:15.798-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26230000 2016-04-07T02:07:15.951-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23140000 QueuedCount 37523 2016-04-07T02:07:16.046-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26240000 2016-04-07T02:07:16.292-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26250000 2016-04-07T02:07:16.299-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23150000 QueuedCount 27523 2016-04-07T02:07:16.518-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26260000 2016-04-07T02:07:16.657-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23160000 QueuedCount 17523 2016-04-07T02:07:16.751-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26270000 2016-04-07T02:07:16.99-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26280000 2016-04-07T02:07:17.035-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23170000 QueuedCount 7523 2016-04-07T02:07:17.219-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26290000 2016-04-07T02:07:17.339-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23177523, 23177523, 0 committed:false 2016-04-07T02:07:17.34-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23177523, 23177523, 0 committed:false) SnapCreateDur 52.721µs SnapOpenDur 626.66µs 2016-04-07T02:07:17.345-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:17.349-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:17.354-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:17.358-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:17.362-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:17.442-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26300000 2016-04-07T02:07:17.482-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23180000 QueuedCount 49106 2016-04-07T02:07:17.67-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26310000 2016-04-07T02:07:17.836-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23190000 QueuedCount 39106 2016-04-07T02:07:17.893-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26320000 2016-04-07T02:07:18.131-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26330000 2016-04-07T02:07:18.338-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23200000 QueuedCount 29106 2016-04-07T02:07:18.34-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26340000 2016-04-07T02:07:18.661-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26350000 2016-04-07T02:07:18.774-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26360000 2016-04-07T02:07:18.972-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23210000 QueuedCount 19106 2016-04-07T02:07:19.003-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26370000 2016-04-07T02:07:19.224-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26380000 2016-04-07T02:07:19.303-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23220000 QueuedCount 9106 2016-04-07T02:07:19.45-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26390000 2016-04-07T02:07:19.674-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26400000 2016-04-07T02:07:19.703-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23229106, 23229106, 0 committed:false 2016-04-07T02:07:19.704-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23229106, 23229106, 0 committed:false) SnapCreateDur 29.054µs SnapOpenDur 841.322µs 2016-04-07T02:07:19.716-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23230000 QueuedCount 7703 2016-04-07T02:07:19.898-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26410000 2016-04-07T02:07:20.114-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26420000 2016-04-07T02:07:20.345-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26430000 2016-04-07T02:07:20.558-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26440000 2016-04-07T02:07:20.585-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 31.348µs CommitTime 456.945218ms TotalFlushTime 17m16.698710526s TotalCommitTime 1m52.932533387s 2016-04-07T02:07:20.585-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23237703, 23237703, 179 committed:true 2016-04-07T02:07:20.585-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23237703, 23237703, 179 committed:true) SnapCreateDur 465.736965ms SnapOpenDur 52.158µs 2016-04-07T02:07:20.59-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:20.595-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:20.6-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:20.604-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:20.609-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:20.775-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26450000 2016-04-07T02:07:20.783-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23240000 QueuedCount 47670 2016-04-07T02:07:20.995-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26460000 2016-04-07T02:07:21.107-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23250000 QueuedCount 37670 2016-04-07T02:07:21.219-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26470000 2016-04-07T02:07:21.399-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23260000 QueuedCount 27670 2016-04-07T02:07:21.444-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26480000 2016-04-07T02:07:21.668-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26490000 2016-04-07T02:07:21.759-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23270000 QueuedCount 17670 2016-04-07T02:07:21.886-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26500000 2016-04-07T02:07:22.104-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26510000 2016-04-07T02:07:22.139-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23280000 QueuedCount 7670 2016-04-07T02:07:22.332-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26520000 2016-04-07T02:07:22.444-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23287670, 23287670, 0 committed:false 2016-04-07T02:07:22.445-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23287670, 23287670, 0 committed:false) SnapCreateDur 29.223µs SnapOpenDur 668.225µs 2016-04-07T02:07:22.45-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:22.455-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:22.46-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:22.465-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:22.47-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:22.553-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26530000 2016-04-07T02:07:22.593-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23290000 QueuedCount 48167 2016-04-07T02:07:22.771-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26540000 2016-04-07T02:07:22.941-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23300000 QueuedCount 38167 2016-04-07T02:07:22.994-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26550000 2016-04-07T02:07:23.217-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26560000 2016-04-07T02:07:23.298-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23310000 QueuedCount 28167 2016-04-07T02:07:23.438-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26570000 2016-04-07T02:07:23.667-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26580000 2016-04-07T02:07:23.675-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23320000 QueuedCount 18167 2016-04-07T02:07:23.893-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26590000 2016-04-07T02:07:24.053-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23330000 QueuedCount 8167 2016-04-07T02:07:24.118-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26600000 2016-04-07T02:07:24.344-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26610000 2016-04-07T02:07:24.375-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23338167, 23338167, 0 committed:false 2016-04-07T02:07:24.375-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23338167, 23338167, 0 committed:false) SnapCreateDur 39.074µs SnapOpenDur 181.22µs 2016-04-07T02:07:24.38-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:24.385-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:24.39-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:24.396-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:24.401-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:24.411-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23340000 QueuedCount 49138 2016-04-07T02:07:24.569-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26620000 2016-04-07T02:07:24.793-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26630000 2016-04-07T02:07:24.967-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23350000 QueuedCount 39138 2016-04-07T02:07:25.018-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26640000 2016-04-07T02:07:25.243-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26650000 2016-04-07T02:07:25.342-07:00 [Info] Indexer::ReadMemstats Time Taken 12.571492ms 2016-04-07T02:07:25.357-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23360000 QueuedCount 29138 2016-04-07T02:07:25.468-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26660000 2016-04-07T02:07:25.689-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26670000 2016-04-07T02:07:25.756-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23370000 QueuedCount 19138 2016-04-07T02:07:25.923-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26680000 2016-04-07T02:07:26.151-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23380000 QueuedCount 9138 2016-04-07T02:07:26.158-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26690000 2016-04-07T02:07:26.399-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26700000 2016-04-07T02:07:26.486-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23389138, 23389138, 0 committed:false 2016-04-07T02:07:26.488-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23389138, 23389138, 0 committed:false) SnapCreateDur 28.322µs SnapOpenDur 2.033095ms 2016-04-07T02:07:26.622-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26710000 2016-04-07T02:07:26.817-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23390000 QueuedCount 7624 2016-04-07T02:07:26.845-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26720000 2016-04-07T02:07:27.083-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26730000 2016-04-07T02:07:27.313-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26740000 2016-04-07T02:07:27.53-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26750000 2016-04-07T02:07:27.751-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26760000 2016-04-07T02:07:27.773-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 32.254µs CommitTime 599.782111ms TotalFlushTime 17m23.152912375s TotalCommitTime 1m53.532315498s 2016-04-07T02:07:27.773-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23397624, 23397624, 180 committed:true 2016-04-07T02:07:27.773-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23397624, 23397624, 180 committed:true) SnapCreateDur 609.316277ms SnapOpenDur 57.318µs 2016-04-07T02:07:27.778-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:27.783-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:27.788-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:27.793-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:27.798-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:27.931-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23400000 QueuedCount 49318 2016-04-07T02:07:27.976-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26770000 2016-04-07T02:07:28.205-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26780000 2016-04-07T02:07:28.265-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23410000 QueuedCount 39318 2016-04-07T02:07:28.429-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26790000 2016-04-07T02:07:28.567-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23420000 QueuedCount 29318 2016-04-07T02:07:28.65-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26800000 2016-04-07T02:07:28.886-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26810000 2016-04-07T02:07:28.941-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23430000 QueuedCount 19318 2016-04-07T02:07:29.115-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26820000 2016-04-07T02:07:29.296-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23440000 QueuedCount 9318 2016-04-07T02:07:29.338-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26830000 2016-04-07T02:07:29.567-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26840000 2016-04-07T02:07:29.654-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23449318, 23449318, 0 committed:false 2016-04-07T02:07:29.654-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23449318, 23449318, 0 committed:false) SnapCreateDur 31.314µs SnapOpenDur 404.463µs 2016-04-07T02:07:29.659-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:29.664-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:29.669-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:29.67-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23450000 QueuedCount 26073 2016-04-07T02:07:29.674-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:29.678-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:29.807-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26850000 2016-04-07T02:07:30.039-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26860000 2016-04-07T02:07:30.102-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23460000 QueuedCount 40571 2016-04-07T02:07:30.269-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26870000 2016-04-07T02:07:30.468-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23470000 QueuedCount 30571 2016-04-07T02:07:30.488-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26880000 2016-04-07T02:07:30.72-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26890000 2016-04-07T02:07:30.832-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23480000 QueuedCount 20571 2016-04-07T02:07:30.944-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26900000 2016-04-07T02:07:31.179-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26910000 2016-04-07T02:07:31.236-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23490000 QueuedCount 10571 2016-04-07T02:07:31.407-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26920000 2016-04-07T02:07:31.582-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23500000 QueuedCount 571 2016-04-07T02:07:31.631-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26930000 2016-04-07T02:07:31.654-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23500571, 23500571, 0 committed:false 2016-04-07T02:07:31.654-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23500571, 23500571, 0 committed:false) SnapCreateDur 27.779µs SnapOpenDur 497.07µs 2016-04-07T02:07:31.859-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26940000 2016-04-07T02:07:32.086-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26950000 2016-04-07T02:07:32.314-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26960000 2016-04-07T02:07:32.545-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26970000 2016-04-07T02:07:32.775-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26980000 2016-04-07T02:07:32.798-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 29.253µs CommitTime 538.460059ms TotalFlushTime 17m27.537752383s TotalCommitTime 1m54.070775557s 2016-04-07T02:07:32.798-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23509496, 23509496, 181 committed:true 2016-04-07T02:07:32.798-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23509496, 23509496, 181 committed:true) SnapCreateDur 548.024363ms SnapOpenDur 36.133µs 2016-04-07T02:07:32.802-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:32.805-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23510000 QueuedCount 14525 2016-04-07T02:07:32.807-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:32.811-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:32.816-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:32.821-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:32.996-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 26990000 2016-04-07T02:07:33.209-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23520000 QueuedCount 41092 2016-04-07T02:07:33.221-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27000000 2016-04-07T02:07:33.48-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27010000 2016-04-07T02:07:33.551-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23530000 QueuedCount 31092 2016-04-07T02:07:33.701-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27020000 2016-04-07T02:07:33.859-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23540000 QueuedCount 21092 2016-04-07T02:07:33.932-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27030000 2016-04-07T02:07:34.16-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27040000 2016-04-07T02:07:34.223-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23550000 QueuedCount 11092 2016-04-07T02:07:34.396-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27050000 2016-04-07T02:07:34.592-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23560000 QueuedCount 1092 2016-04-07T02:07:34.621-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27060000 2016-04-07T02:07:34.667-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23561092, 23561092, 0 committed:false 2016-04-07T02:07:34.668-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23561092, 23561092, 0 committed:false) SnapCreateDur 27.777µs SnapOpenDur 434.606µs 2016-04-07T02:07:34.673-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:34.678-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:34.682-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:34.688-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:34.692-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:34.861-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27070000 2016-04-07T02:07:35.082-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23570000 QueuedCount 42970 2016-04-07T02:07:35.089-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27080000 2016-04-07T02:07:35.311-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27090000 2016-04-07T02:07:35.451-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23580000 QueuedCount 32970 2016-04-07T02:07:35.537-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27100000 2016-04-07T02:07:35.761-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27110000 2016-04-07T02:07:35.824-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23590000 QueuedCount 22970 2016-04-07T02:07:35.985-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27120000 2016-04-07T02:07:36.198-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23600000 QueuedCount 12970 2016-04-07T02:07:36.206-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27130000 2016-04-07T02:07:36.442-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27140000 2016-04-07T02:07:36.594-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23610000 QueuedCount 2970 2016-04-07T02:07:36.675-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27150000 2016-04-07T02:07:36.692-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23612970, 23612970, 0 committed:false 2016-04-07T02:07:36.693-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23612970, 23612970, 0 committed:false) SnapCreateDur 41.988µs SnapOpenDur 892.677µs 2016-04-07T02:07:36.902-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27160000 2016-04-07T02:07:37.137-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27170000 2016-04-07T02:07:37.177-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23620000 QueuedCount 1624 2016-04-07T02:07:37.376-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27180000 2016-04-07T02:07:37.604-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27190000 2016-04-07T02:07:37.808-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 32.239µs CommitTime 540.868022ms TotalFlushTime 17m31.901539381s TotalCommitTime 1m54.611643579s 2016-04-07T02:07:37.808-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23621624, 23621624, 182 committed:true 2016-04-07T02:07:37.808-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23621624, 23621624, 182 committed:true) SnapCreateDur 549.827292ms SnapOpenDur 65.248µs 2016-04-07T02:07:37.813-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:37.818-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:37.822-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:37.827-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:37.831-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:37.836-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27200000 2016-04-07T02:07:38.064-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27210000 2016-04-07T02:07:38.273-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23630000 QueuedCount 42813 2016-04-07T02:07:38.298-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27220000 2016-04-07T02:07:38.524-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27230000 2016-04-07T02:07:38.555-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23640000 QueuedCount 32813 2016-04-07T02:07:38.75-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27240000 2016-04-07T02:07:38.891-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23650000 QueuedCount 22813 2016-04-07T02:07:38.976-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27250000 2016-04-07T02:07:39.214-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27260000 2016-04-07T02:07:39.247-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23660000 QueuedCount 12813 2016-04-07T02:07:39.443-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27270000 2016-04-07T02:07:39.6-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23670000 QueuedCount 2813 2016-04-07T02:07:39.67-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27280000 2016-04-07T02:07:39.686-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23672813, 23672813, 0 committed:false 2016-04-07T02:07:39.688-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23672813, 23672813, 0 committed:false) SnapCreateDur 63.69µs SnapOpenDur 2.042246ms 2016-04-07T02:07:39.692-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:6346090343, Disk:15718506496, ExtraSnapshotData:0, Fragmentation:59.62663281899629%, mode : circular 2016-04-07T02:07:39.692-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:07:39.692-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:39.697-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:39.702-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:39.706-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:39.711-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:39.905-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27290000 2016-04-07T02:07:40.009-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23680000 QueuedCount 44215 2016-04-07T02:07:40.127-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27300000 2016-04-07T02:07:40.359-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27310000 2016-04-07T02:07:40.37-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23690000 QueuedCount 34215 2016-04-07T02:07:40.58-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27320000 2016-04-07T02:07:40.736-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23700000 QueuedCount 24215 2016-04-07T02:07:40.808-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27330000 2016-04-07T02:07:41.033-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27340000 2016-04-07T02:07:41.104-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23710000 QueuedCount 14215 2016-04-07T02:07:41.254-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27350000 2016-04-07T02:07:41.48-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27360000 2016-04-07T02:07:41.507-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23720000 QueuedCount 4215 2016-04-07T02:07:41.673-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23724215, 23724215, 0 committed:false 2016-04-07T02:07:41.674-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23724215, 23724215, 0 committed:false) SnapCreateDur 55.919µs SnapOpenDur 287.985µs 2016-04-07T02:07:41.678-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:41.683-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:41.687-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:41.691-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:41.696-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:41.705-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27370000 2016-04-07T02:07:41.924-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27380000 2016-04-07T02:07:41.949-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23730000 QueuedCount 46519 2016-04-07T02:07:42.147-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27390000 2016-04-07T02:07:42.331-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23740000 QueuedCount 36519 2016-04-07T02:07:42.378-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27400000 2016-04-07T02:07:42.598-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27410000 2016-04-07T02:07:42.719-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23750000 QueuedCount 26519 2016-04-07T02:07:42.83-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27420000 2016-04-07T02:07:43.049-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27430000 2016-04-07T02:07:43.117-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23760000 QueuedCount 16519 2016-04-07T02:07:43.278-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27440000 2016-04-07T02:07:43.506-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27450000 2016-04-07T02:07:43.53-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23770000 QueuedCount 6519 2016-04-07T02:07:43.731-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27460000 2016-04-07T02:07:43.959-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27470000 2016-04-07T02:07:44.18-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27480000 2016-04-07T02:07:44.388-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.958µs CommitTime 597.405558ms TotalFlushTime 17m37.772856154s TotalCommitTime 1m55.209049137s 2016-04-07T02:07:44.388-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23776519, 23776519, 183 committed:true 2016-04-07T02:07:44.388-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23776519, 23776519, 183 committed:true) SnapCreateDur 607.578391ms SnapOpenDur 55.518µs 2016-04-07T02:07:44.392-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:44.396-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:44.4-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:44.404-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:44.405-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27490000 2016-04-07T02:07:44.408-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:44.544-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23780000 QueuedCount 49469 2016-04-07T02:07:44.625-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27500000 2016-04-07T02:07:44.85-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27510000 2016-04-07T02:07:44.878-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23790000 QueuedCount 39469 2016-04-07T02:07:45.076-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27520000 2016-04-07T02:07:45.253-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23800000 QueuedCount 29469 2016-04-07T02:07:45.309-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27530000 2016-04-07T02:07:45.549-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27540000 2016-04-07T02:07:45.647-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23810000 QueuedCount 19469 2016-04-07T02:07:45.795-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27550000 2016-04-07T02:07:45.981-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27560000 2016-04-07T02:07:46.044-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23820000 QueuedCount 9469 2016-04-07T02:07:46.2-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27570000 2016-04-07T02:07:46.38-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23829469, 23829469, 0 committed:false 2016-04-07T02:07:46.381-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23829469, 23829469, 0 committed:false) SnapCreateDur 54.416µs SnapOpenDur 1.762862ms 2016-04-07T02:07:46.386-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:46.39-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:46.394-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:46.398-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:46.402-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:46.439-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27580000 2016-04-07T02:07:46.509-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23830000 QueuedCount 50000 2016-04-07T02:07:46.671-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27590000 2016-04-07T02:07:46.838-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23840000 QueuedCount 43237 2016-04-07T02:07:46.906-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27600000 2016-04-07T02:07:47.144-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27610000 2016-04-07T02:07:47.225-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23850000 QueuedCount 33237 2016-04-07T02:07:47.369-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27620000 2016-04-07T02:07:47.597-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27630000 2016-04-07T02:07:47.62-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23860000 QueuedCount 23237 2016-04-07T02:07:47.819-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27640000 2016-04-07T02:07:48.022-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23870000 QueuedCount 13237 2016-04-07T02:07:48.042-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27650000 2016-04-07T02:07:48.261-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27660000 2016-04-07T02:07:48.43-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23880000 QueuedCount 3237 2016-04-07T02:07:48.484-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27670000 2016-04-07T02:07:48.589-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23883237, 23883237, 0 committed:false 2016-04-07T02:07:48.589-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23883237, 23883237, 0 committed:false) SnapCreateDur 25.903µs SnapOpenDur 305.44µs 2016-04-07T02:07:48.705-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27680000 2016-04-07T02:07:48.931-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27690000 2016-04-07T02:07:49.07-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23890000 QueuedCount 308 2016-04-07T02:07:49.158-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27700000 2016-04-07T02:07:49.379-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27710000 2016-04-07T02:07:49.597-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27720000 2016-04-07T02:07:49.648-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 26.181µs CommitTime 565.069144ms TotalFlushTime 17m42.390282391s TotalCommitTime 1m55.774118281s 2016-04-07T02:07:49.648-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23890308, 23890308, 184 committed:true 2016-04-07T02:07:49.648-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23890308, 23890308, 184 committed:true) SnapCreateDur 574.626527ms SnapOpenDur 53.111µs 2016-04-07T02:07:49.651-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:49.657-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:49.661-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:49.665-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:49.669-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:49.825-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27730000 2016-04-07T02:07:50.045-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27740000 2016-04-07T02:07:50.132-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23900000 QueuedCount 45201 2016-04-07T02:07:50.269-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27750000 2016-04-07T02:07:50.494-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27760000 2016-04-07T02:07:50.505-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23910000 QueuedCount 35201 2016-04-07T02:07:50.712-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27770000 2016-04-07T02:07:50.864-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23920000 QueuedCount 25201 2016-04-07T02:07:50.938-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27780000 2016-04-07T02:07:51.156-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27790000 2016-04-07T02:07:51.227-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23930000 QueuedCount 15201 2016-04-07T02:07:51.379-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27800000 2016-04-07T02:07:51.6-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27810000 2016-04-07T02:07:51.62-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23940000 QueuedCount 5201 2016-04-07T02:07:51.788-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23945201, 23945201, 0 committed:false 2016-04-07T02:07:51.789-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23945201, 23945201, 0 committed:false) SnapCreateDur 28.782µs SnapOpenDur 1.489334ms 2016-04-07T02:07:51.793-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:51.797-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:51.801-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:51.805-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:51.809-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:51.832-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27820000 2016-04-07T02:07:52.058-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27830000 2016-04-07T02:07:52.062-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23950000 QueuedCount 47590 2016-04-07T02:07:52.287-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27840000 2016-04-07T02:07:52.456-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23960000 QueuedCount 37590 2016-04-07T02:07:52.507-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27850000 2016-04-07T02:07:52.729-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27860000 2016-04-07T02:07:52.785-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23970000 QueuedCount 27590 2016-04-07T02:07:52.948-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27870000 2016-04-07T02:07:53.174-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27880000 2016-04-07T02:07:53.179-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23980000 QueuedCount 17590 2016-04-07T02:07:53.394-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27890000 2016-04-07T02:07:53.593-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 23990000 QueuedCount 7590 2016-04-07T02:07:53.619-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27900000 2016-04-07T02:07:53.847-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27910000 2016-04-07T02:07:53.929-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 23997590, 23997590, 0 committed:false 2016-04-07T02:07:53.929-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 23997590, 23997590, 0 committed:false) SnapCreateDur 29.067µs SnapOpenDur 626.832µs 2016-04-07T02:07:54.078-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27920000 2016-04-07T02:07:54.116-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24000000 QueuedCount 6113 2016-04-07T02:07:54.295-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27930000 2016-04-07T02:07:54.528-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27940000 2016-04-07T02:07:54.773-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27950000 2016-04-07T02:07:54.984-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27960000 2016-04-07T02:07:55.044-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.399µs CommitTime 679.074532ms TotalFlushTime 17m47.017963921s TotalCommitTime 1m56.453192813s 2016-04-07T02:07:55.044-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24006113, 24006113, 185 committed:true 2016-04-07T02:07:55.045-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24006113, 24006113, 185 committed:true) SnapCreateDur 689.577114ms SnapOpenDur 137.432µs 2016-04-07T02:07:55.261-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:55.266-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:55.272-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:55.278-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:55.281-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27970000 2016-04-07T02:07:55.283-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:55.423-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27980000 2016-04-07T02:07:55.43-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24010000 QueuedCount 49425 2016-04-07T02:07:55.64-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 27990000 2016-04-07T02:07:55.799-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24020000 QueuedCount 39425 2016-04-07T02:07:55.862-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28000000 2016-04-07T02:07:56.156-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24030000 QueuedCount 29425 2016-04-07T02:07:56.207-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28010000 2016-04-07T02:07:56.435-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28020000 2016-04-07T02:07:56.546-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24040000 QueuedCount 19425 2016-04-07T02:07:56.756-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28030000 2016-04-07T02:07:56.917-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24050000 QueuedCount 9425 2016-04-07T02:07:56.979-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28040000 2016-04-07T02:07:57.199-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28050000 2016-04-07T02:07:57.304-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24059425, 24059425, 0 committed:false 2016-04-07T02:07:57.304-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24059425, 24059425, 0 committed:false) SnapCreateDur 55.501µs SnapOpenDur 199.372µs 2016-04-07T02:07:57.309-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:57.314-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:57.315-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24060000 QueuedCount 19245 2016-04-07T02:07:57.319-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:57.323-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:57.328-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:07:57.42-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28060000 2016-04-07T02:07:57.642-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28070000 2016-04-07T02:07:57.781-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24070000 QueuedCount 43059 2016-04-07T02:07:57.865-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28080000 2016-04-07T02:07:58.092-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28090000 2016-04-07T02:07:58.173-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24080000 QueuedCount 33059 2016-04-07T02:07:58.315-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28100000 2016-04-07T02:07:58.513-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24090000 QueuedCount 23059 2016-04-07T02:07:58.529-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28110000 2016-04-07T02:07:58.75-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28120000 2016-04-07T02:07:58.91-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24100000 QueuedCount 13059 2016-04-07T02:07:58.971-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28130000 2016-04-07T02:07:59.203-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28140000 2016-04-07T02:07:59.332-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24110000 QueuedCount 3059 2016-04-07T02:07:59.424-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28150000 2016-04-07T02:07:59.498-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24113059, 24113059, 0 committed:false 2016-04-07T02:07:59.499-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24113059, 24113059, 0 committed:false) SnapCreateDur 30.071µs SnapOpenDur 440.612µs 2016-04-07T02:07:59.64-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28160000 2016-04-07T02:07:59.869-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28170000 2016-04-07T02:07:59.946-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24120000 QueuedCount 1861 2016-04-07T02:08:00.1-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28180000 2016-04-07T02:08:00.329-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28190000 2016-04-07T02:08:00.554-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28200000 2016-04-07T02:08:00.605-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.658µs CommitTime 570.689132ms TotalFlushTime 17m51.908474865s TotalCommitTime 1m57.023881945s 2016-04-07T02:08:00.605-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24121861, 24121861, 186 committed:true 2016-04-07T02:08:00.605-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24121861, 24121861, 186 committed:true) SnapCreateDur 580.408345ms SnapOpenDur 67.538µs 2016-04-07T02:08:00.61-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:00.615-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:00.62-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:00.624-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:00.629-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:00.777-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28210000 2016-04-07T02:08:00.929-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24130000 QueuedCount 45754 2016-04-07T02:08:00.998-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28220000 2016-04-07T02:08:01.225-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28230000 2016-04-07T02:08:01.316-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24140000 QueuedCount 35754 2016-04-07T02:08:01.444-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28240000 2016-04-07T02:08:01.663-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28250000 2016-04-07T02:08:01.698-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24150000 QueuedCount 25754 2016-04-07T02:08:01.887-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28260000 2016-04-07T02:08:02.082-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24160000 QueuedCount 15754 2016-04-07T02:08:02.112-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28270000 2016-04-07T02:08:02.334-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28280000 2016-04-07T02:08:02.472-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24170000 QueuedCount 5754 2016-04-07T02:08:02.559-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28290000 2016-04-07T02:08:02.718-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24175754, 24175754, 0 committed:false 2016-04-07T02:08:02.718-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24175754, 24175754, 0 committed:false) SnapCreateDur 25.874µs SnapOpenDur 486.214µs 2016-04-07T02:08:02.723-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:02.727-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:02.732-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:02.736-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:02.74-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:02.781-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28300000 2016-04-07T02:08:02.99-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24180000 QueuedCount 48551 2016-04-07T02:08:03.008-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28310000 2016-04-07T02:08:03.234-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28320000 2016-04-07T02:08:03.376-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24190000 QueuedCount 38551 2016-04-07T02:08:03.459-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28330000 2016-04-07T02:08:03.694-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28340000 2016-04-07T02:08:03.757-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24200000 QueuedCount 28551 2016-04-07T02:08:03.917-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28350000 2016-04-07T02:08:04.144-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28360000 2016-04-07T02:08:04.146-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24210000 QueuedCount 18551 2016-04-07T02:08:04.368-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28370000 2016-04-07T02:08:04.486-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24220000 QueuedCount 8551 2016-04-07T02:08:04.595-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28380000 2016-04-07T02:08:04.829-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28390000 2016-04-07T02:08:04.868-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24228551, 24228551, 0 committed:false 2016-04-07T02:08:04.868-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24228551, 24228551, 0 committed:false) SnapCreateDur 52.54µs SnapOpenDur 252.628µs 2016-04-07T02:08:04.885-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24230000 QueuedCount 7713 2016-04-07T02:08:05.058-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28400000 2016-04-07T02:08:05.275-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28410000 2016-04-07T02:08:05.498-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28420000 2016-04-07T02:08:05.723-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28430000 2016-04-07T02:08:05.943-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 28.834µs CommitTime 540.807705ms TotalFlushTime 17m56.609753136s TotalCommitTime 1m57.56468965s 2016-04-07T02:08:05.943-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24237713, 24237713, 187 committed:true 2016-04-07T02:08:05.943-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24237713, 24237713, 187 committed:true) SnapCreateDur 550.466052ms SnapOpenDur 70.081µs 2016-04-07T02:08:05.948-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:05.953-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:05.956-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:05.957-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28440000 2016-04-07T02:08:05.96-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:05.964-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:06.191-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28450000 2016-04-07T02:08:06.217-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24240000 QueuedCount 50000 2016-04-07T02:08:06.421-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28460000 2016-04-07T02:08:06.501-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24250000 QueuedCount 41132 2016-04-07T02:08:06.649-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28470000 2016-04-07T02:08:06.834-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24260000 QueuedCount 31132 2016-04-07T02:08:06.895-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28480000 2016-04-07T02:08:07.174-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24270000 QueuedCount 21132 2016-04-07T02:08:07.182-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28490000 2016-04-07T02:08:07.405-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28500000 2016-04-07T02:08:07.528-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24280000 QueuedCount 11132 2016-04-07T02:08:07.633-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28510000 2016-04-07T02:08:07.859-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28520000 2016-04-07T02:08:07.889-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24290000 QueuedCount 1132 2016-04-07T02:08:07.969-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24291132, 24291132, 0 committed:false 2016-04-07T02:08:07.969-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24291132, 24291132, 0 committed:false) SnapCreateDur 48.464µs SnapOpenDur 296.78µs 2016-04-07T02:08:07.973-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:07.977-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:07.98-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:07.984-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:07.987-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:08.094-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28530000 2016-04-07T02:08:08.314-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28540000 2016-04-07T02:08:08.369-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24300000 QueuedCount 45635 2016-04-07T02:08:08.545-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28550000 2016-04-07T02:08:08.773-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28560000 2016-04-07T02:08:08.776-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24310000 QueuedCount 35635 2016-04-07T02:08:09.004-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28570000 2016-04-07T02:08:09.178-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24320000 QueuedCount 25635 2016-04-07T02:08:09.229-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28580000 2016-04-07T02:08:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:08:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:08:09.374-07:00 [Info] client load stats {} 2016-04-07T02:08:09.458-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28590000 2016-04-07T02:08:09.61-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24330000 QueuedCount 15635 2016-04-07T02:08:09.652-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":4252866,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":1781369249,"bucket-1:myindex:avg_ts_items_count":40715,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":6498814963,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":16467521536,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":13692,"bucket-1:myindex:frag_percent":60,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":2544959865,"bucket-1:myindex:items_count":24121861,"bucket-1:myindex:num_commits":187,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":24331943,"bucket-1:myindex:num_docs_pending":856,"bucket-1:myindex:num_docs_processed":24228551,"bucket-1:myindex:num_docs_queued":4190735,"bucket-1:myindex:num_flush_queued":24345635,"bucket-1:myindex:num_items_flushed":24331943,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":283,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":2026157624,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"187 117564689650 -4223657759507384404","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"24331944 139900577175 790726395876840213","bucket-1:myindex:timings/storage_info":"941 7228981 105943086413","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"186 65578176 25748583958576","bucket-1:myindex:timings/storage_meta_set":"187 65305481 24884236253059","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"187 7831187 363411116451","bucket-1:myindex:timings/storage_set":"48663887 927440523569 3568734328781578525","bucket-1:myindex:timings/storage_snapshot_close":"95 71890198 162949736175172","bucket-1:myindex:timings/storage_snapshot_create":"96 83491134 104984158093258","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":28598501,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"332 565616241 27035715802423457","bucket-1:ts_queue_size":484,"indexer_state":"Active","memory_quota":31457280000,"memory_used":19181975552,"memory_used_storage":16739463168,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"4057 721849911 332822322022675","uptime":"39m1.34468045s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:08:09.679-07:00 [Info] memstats {"Alloc":1570034104, "TotalAlloc":28224038552, "Sys":9854296392, "Lookups":8793, "Mallocs":798224777,"Frees":757150584, "HeapAlloc":1570034104, "HeapSys":9382985728, "HeapIdle":7237918720, "HeapInuse":2145067008,"HeapReleased":6614867968, "HeapObjects":41074193,"GCSys":297445376, "LastGC":1460020038459560045,"PauseTotalNs":47514864, "PauseNs":[1677544, 1806628], "NumGC":37} 2016-04-07T02:08:09.688-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28600000 2016-04-07T02:08:09.692-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:6523945944, Disk:16740401152, ExtraSnapshotData:0, Fragmentation:61.02873590206305%, mode : circular 2016-04-07T02:08:09.692-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:08:09.909-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28610000 2016-04-07T02:08:09.974-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24340000 QueuedCount 5635 2016-04-07T02:08:10.145-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28620000 2016-04-07T02:08:10.236-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24345635, 24345635, 0 committed:false 2016-04-07T02:08:10.237-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24345635, 24345635, 0 committed:false) SnapCreateDur 39.326µs SnapOpenDur 1.221507ms 2016-04-07T02:08:10.37-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28630000 2016-04-07T02:08:10.588-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24350000 QueuedCount 4807 2016-04-07T02:08:10.595-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28640000 2016-04-07T02:08:10.817-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28650000 2016-04-07T02:08:11.038-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28660000 2016-04-07T02:08:11.263-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28670000 2016-04-07T02:08:11.382-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 35.489µs CommitTime 550.711451ms TotalFlushTime 18m1.412871568s TotalCommitTime 1m58.115401101s 2016-04-07T02:08:11.382-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24354807, 24354807, 188 committed:true 2016-04-07T02:08:11.382-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24354807, 24354807, 188 committed:true) SnapCreateDur 560.122095ms SnapOpenDur 63.538µs 2016-04-07T02:08:11.386-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:11.389-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:11.393-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:11.396-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:11.4-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:11.484-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28680000 2016-04-07T02:08:11.708-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24360000 QueuedCount 49708 2016-04-07T02:08:11.709-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28690000 2016-04-07T02:08:11.928-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28700000 2016-04-07T02:08:12.104-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24370000 QueuedCount 39708 2016-04-07T02:08:12.158-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28710000 2016-04-07T02:08:12.378-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28720000 2016-04-07T02:08:12.488-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24380000 QueuedCount 29708 2016-04-07T02:08:12.601-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28730000 2016-04-07T02:08:12.818-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28740000 2016-04-07T02:08:12.888-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24390000 QueuedCount 19708 2016-04-07T02:08:13.043-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28750000 2016-04-07T02:08:13.269-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28760000 2016-04-07T02:08:13.291-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24400000 QueuedCount 9708 2016-04-07T02:08:13.493-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28770000 2016-04-07T02:08:13.625-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24409708, 24409708, 0 committed:false 2016-04-07T02:08:13.626-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24409708, 24409708, 0 committed:false) SnapCreateDur 31.205µs SnapOpenDur 1.50345ms 2016-04-07T02:08:13.629-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:13.631-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:13.631-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24410000 QueuedCount 10436 2016-04-07T02:08:13.634-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:13.638-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:13.64-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:13.714-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28780000 2016-04-07T02:08:13.943-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28790000 2016-04-07T02:08:14.117-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24420000 QueuedCount 27186 2016-04-07T02:08:14.169-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28800000 2016-04-07T02:08:14.393-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28810000 2016-04-07T02:08:14.516-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24430000 QueuedCount 17186 2016-04-07T02:08:14.618-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28820000 2016-04-07T02:08:14.842-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28830000 2016-04-07T02:08:14.916-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24440000 QueuedCount 7186 2016-04-07T02:08:15.064-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28840000 2016-04-07T02:08:15.222-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24447186, 24447186, 0 committed:false 2016-04-07T02:08:15.222-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24447186, 24447186, 0 committed:false) SnapCreateDur 29.132µs SnapOpenDur 241.027µs 2016-04-07T02:08:15.226-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:15.23-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:15.233-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:15.237-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:15.24-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:15.289-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28850000 2016-04-07T02:08:15.512-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24450000 QueuedCount 50000 2016-04-07T02:08:15.524-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28860000 2016-04-07T02:08:15.746-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28870000 2016-04-07T02:08:15.905-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24460000 QueuedCount 41046 2016-04-07T02:08:15.972-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28880000 2016-04-07T02:08:16.194-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28890000 2016-04-07T02:08:16.306-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24470000 QueuedCount 31046 2016-04-07T02:08:16.419-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28900000 2016-04-07T02:08:16.644-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28910000 2016-04-07T02:08:16.693-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24480000 QueuedCount 21046 2016-04-07T02:08:16.863-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28920000 2016-04-07T02:08:17.023-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24490000 QueuedCount 11046 2016-04-07T02:08:17.095-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28930000 2016-04-07T02:08:17.361-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28940000 2016-04-07T02:08:17.409-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24500000 QueuedCount 1046 2016-04-07T02:08:17.63-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28950000 2016-04-07T02:08:17.854-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28960000 2016-04-07T02:08:18.073-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28970000 2016-04-07T02:08:18.101-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 33.362µs CommitTime 607.227541ms TotalFlushTime 18m7.429115502s TotalCommitTime 1m58.722628642s 2016-04-07T02:08:18.101-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24501046, 24501046, 189 committed:true 2016-04-07T02:08:18.102-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24501046, 24501046, 189 committed:true) SnapCreateDur 616.266575ms SnapOpenDur 77.361µs 2016-04-07T02:08:18.106-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:18.11-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:18.113-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:18.117-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:18.121-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:18.295-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28980000 2016-04-07T02:08:18.522-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 28990000 2016-04-07T02:08:18.559-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24510000 QueuedCount 44052 2016-04-07T02:08:18.737-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29000000 2016-04-07T02:08:18.91-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24520000 QueuedCount 34052 2016-04-07T02:08:18.964-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29010000 2016-04-07T02:08:19.186-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29020000 2016-04-07T02:08:19.268-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24530000 QueuedCount 24052 2016-04-07T02:08:19.413-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29030000 2016-04-07T02:08:19.628-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24540000 QueuedCount 14052 2016-04-07T02:08:19.65-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29040000 2016-04-07T02:08:19.87-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29050000 2016-04-07T02:08:19.941-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24550000 QueuedCount 4052 2016-04-07T02:08:20.099-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29060000 2016-04-07T02:08:20.107-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24554052, 24554052, 0 committed:false 2016-04-07T02:08:20.109-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24554052, 24554052, 0 committed:false) SnapCreateDur 32.299µs SnapOpenDur 2.219718ms 2016-04-07T02:08:20.114-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:20.118-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:20.122-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:20.127-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:20.131-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:20.328-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29070000 2016-04-07T02:08:20.399-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24560000 QueuedCount 48569 2016-04-07T02:08:20.55-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29080000 2016-04-07T02:08:20.776-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29090000 2016-04-07T02:08:20.781-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24570000 QueuedCount 38569 2016-04-07T02:08:21.004-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29100000 2016-04-07T02:08:21.16-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24580000 QueuedCount 28569 2016-04-07T02:08:21.225-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29110000 2016-04-07T02:08:21.45-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29120000 2016-04-07T02:08:21.551-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24590000 QueuedCount 18569 2016-04-07T02:08:21.665-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29130000 2016-04-07T02:08:21.887-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29140000 2016-04-07T02:08:21.951-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24600000 QueuedCount 8569 2016-04-07T02:08:22.109-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29150000 2016-04-07T02:08:22.295-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24608569, 24608569, 0 committed:false 2016-04-07T02:08:22.295-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24608569, 24608569, 0 committed:false) SnapCreateDur 38.804µs SnapOpenDur 863.074µs 2016-04-07T02:08:22.34-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29160000 2016-04-07T02:08:22.565-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24610000 QueuedCount 6645 2016-04-07T02:08:22.57-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29170000 2016-04-07T02:08:22.794-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29180000 2016-04-07T02:08:23.02-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29190000 2016-04-07T02:08:23.243-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29200000 2016-04-07T02:08:23.376-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.96µs CommitTime 545.474589ms TotalFlushTime 18m12.066864205s TotalCommitTime 1m59.268103231s 2016-04-07T02:08:23.376-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24616645, 24616645, 190 committed:true 2016-04-07T02:08:23.376-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24616645, 24616645, 190 committed:true) SnapCreateDur 555.320519ms SnapOpenDur 79.906µs 2016-04-07T02:08:23.381-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:23.385-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:23.39-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:23.394-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:23.399-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:23.465-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29210000 2016-04-07T02:08:23.543-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24620000 QueuedCount 49918 2016-04-07T02:08:23.69-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29220000 2016-04-07T02:08:23.913-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29230000 2016-04-07T02:08:23.914-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24630000 QueuedCount 39918 2016-04-07T02:08:24.141-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29240000 2016-04-07T02:08:24.298-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24640000 QueuedCount 29918 2016-04-07T02:08:24.369-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29250000 2016-04-07T02:08:24.592-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29260000 2016-04-07T02:08:24.691-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24650000 QueuedCount 19918 2016-04-07T02:08:24.818-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29270000 2016-04-07T02:08:25.04-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29280000 2016-04-07T02:08:25.071-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24660000 QueuedCount 9918 2016-04-07T02:08:25.269-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29290000 2016-04-07T02:08:25.462-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24669918, 24669918, 0 committed:false 2016-04-07T02:08:25.463-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24669918, 24669918, 0 committed:false) SnapCreateDur 42.526µs SnapOpenDur 102.059µs 2016-04-07T02:08:25.464-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24670000 QueuedCount 2309 2016-04-07T02:08:25.467-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:25.471-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:25.476-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:25.48-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:25.485-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:25.492-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29300000 2016-04-07T02:08:25.599-07:00 [Info] Indexer::ReadMemstats Time Taken 13.702849ms 2016-04-07T02:08:25.714-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29310000 2016-04-07T02:08:25.856-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24680000 QueuedCount 43631 2016-04-07T02:08:25.932-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29320000 2016-04-07T02:08:26.156-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29330000 2016-04-07T02:08:26.235-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24690000 QueuedCount 33631 2016-04-07T02:08:26.38-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29340000 2016-04-07T02:08:26.609-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29350000 2016-04-07T02:08:26.632-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24700000 QueuedCount 23631 2016-04-07T02:08:26.834-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29360000 2016-04-07T02:08:27.03-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24710000 QueuedCount 13631 2016-04-07T02:08:27.062-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29370000 2016-04-07T02:08:27.298-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29380000 2016-04-07T02:08:27.422-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24720000 QueuedCount 3631 2016-04-07T02:08:27.522-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29390000 2016-04-07T02:08:27.579-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24723631, 24723631, 0 committed:false 2016-04-07T02:08:27.579-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24723631, 24723631, 0 committed:false) SnapCreateDur 29.087µs SnapOpenDur 435.43µs 2016-04-07T02:08:27.774-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29400000 2016-04-07T02:08:28.049-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29410000 2016-04-07T02:08:28.057-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24730000 QueuedCount 2099 2016-04-07T02:08:28.28-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29420000 2016-04-07T02:08:28.507-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29430000 2016-04-07T02:08:28.734-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29440000 2016-04-07T02:08:28.737-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.437µs CommitTime 582.176465ms TotalFlushTime 18m16.747163959s TotalCommitTime 1m59.850279696s 2016-04-07T02:08:28.737-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24732099, 24732099, 191 committed:true 2016-04-07T02:08:28.737-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24732099, 24732099, 191 committed:true) SnapCreateDur 592.318741ms SnapOpenDur 79.886µs 2016-04-07T02:08:28.741-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:28.746-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:28.75-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:28.754-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:28.759-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:28.957-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29450000 2016-04-07T02:08:29.074-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24740000 QueuedCount 45167 2016-04-07T02:08:29.184-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29460000 2016-04-07T02:08:29.419-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29470000 2016-04-07T02:08:29.471-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24750000 QueuedCount 35167 2016-04-07T02:08:29.642-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29480000 2016-04-07T02:08:29.861-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24760000 QueuedCount 25167 2016-04-07T02:08:29.869-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29490000 2016-04-07T02:08:30.093-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29500000 2016-04-07T02:08:30.255-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24770000 QueuedCount 15167 2016-04-07T02:08:30.321-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29510000 2016-04-07T02:08:30.538-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29520000 2016-04-07T02:08:30.664-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24780000 QueuedCount 5167 2016-04-07T02:08:30.757-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29530000 2016-04-07T02:08:30.843-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24785167, 24785167, 0 committed:false 2016-04-07T02:08:30.845-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24785167, 24785167, 0 committed:false) SnapCreateDur 26.944µs SnapOpenDur 1.922889ms 2016-04-07T02:08:30.849-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:30.853-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:30.857-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:30.861-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:30.865-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:30.98-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29540000 2016-04-07T02:08:31.162-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24790000 QueuedCount 48680 2016-04-07T02:08:31.206-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29550000 2016-04-07T02:08:31.427-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29560000 2016-04-07T02:08:31.55-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24800000 QueuedCount 38680 2016-04-07T02:08:31.651-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29570000 2016-04-07T02:08:31.87-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29580000 2016-04-07T02:08:31.945-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24810000 QueuedCount 28680 2016-04-07T02:08:32.094-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29590000 2016-04-07T02:08:32.28-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24820000 QueuedCount 18680 2016-04-07T02:08:32.318-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29600000 2016-04-07T02:08:32.542-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29610000 2016-04-07T02:08:32.673-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24830000 QueuedCount 8680 2016-04-07T02:08:32.775-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29620000 2016-04-07T02:08:33-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29630000 2016-04-07T02:08:33.06-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24838680, 24838680, 0 committed:false 2016-04-07T02:08:33.061-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24838680, 24838680, 0 committed:false) SnapCreateDur 56.845µs SnapOpenDur 189.86µs 2016-04-07T02:08:33.075-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24840000 QueuedCount 7667 2016-04-07T02:08:33.23-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29640000 2016-04-07T02:08:33.455-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29650000 2016-04-07T02:08:33.679-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29660000 2016-04-07T02:08:33.904-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29670000 2016-04-07T02:08:34.128-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29680000 2016-04-07T02:08:34.211-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.835µs CommitTime 585.448038ms TotalFlushTime 18m21.553114465s TotalCommitTime 2m0.435727734s 2016-04-07T02:08:34.211-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24847667, 24847667, 192 committed:true 2016-04-07T02:08:34.211-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24847667, 24847667, 192 committed:true) SnapCreateDur 595.449847ms SnapOpenDur 52.72µs 2016-04-07T02:08:34.216-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:34.22-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:34.225-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:34.229-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:34.233-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:34.35-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29690000 2016-04-07T02:08:34.494-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24850000 QueuedCount 50000 2016-04-07T02:08:34.588-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29700000 2016-04-07T02:08:34.805-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29710000 2016-04-07T02:08:34.841-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24860000 QueuedCount 40015 2016-04-07T02:08:35.028-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29720000 2016-04-07T02:08:35.137-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24870000 QueuedCount 30015 2016-04-07T02:08:35.25-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29730000 2016-04-07T02:08:35.473-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29740000 2016-04-07T02:08:35.497-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24880000 QueuedCount 20015 2016-04-07T02:08:35.694-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29750000 2016-04-07T02:08:35.866-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24890000 QueuedCount 10015 2016-04-07T02:08:35.919-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29760000 2016-04-07T02:08:36.149-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29770000 2016-04-07T02:08:36.235-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24900000 QueuedCount 15 2016-04-07T02:08:36.245-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24900015, 24900015, 0 committed:false 2016-04-07T02:08:36.246-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24900015, 24900015, 0 committed:false) SnapCreateDur 29.014µs SnapOpenDur 1.260378ms 2016-04-07T02:08:36.25-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:36.254-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:36.258-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:36.262-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:36.266-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:36.369-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29780000 2016-04-07T02:08:36.598-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29790000 2016-04-07T02:08:36.68-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24910000 QueuedCount 41209 2016-04-07T02:08:36.83-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29800000 2016-04-07T02:08:37.054-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29810000 2016-04-07T02:08:37.076-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24920000 QueuedCount 31209 2016-04-07T02:08:37.287-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29820000 2016-04-07T02:08:37.465-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24930000 QueuedCount 21209 2016-04-07T02:08:37.511-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29830000 2016-04-07T02:08:37.731-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29840000 2016-04-07T02:08:37.868-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24940000 QueuedCount 11209 2016-04-07T02:08:37.961-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29850000 2016-04-07T02:08:38.179-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29860000 2016-04-07T02:08:38.214-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24950000 QueuedCount 1209 2016-04-07T02:08:38.29-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24951209, 24951209, 0 committed:false 2016-04-07T02:08:38.291-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24951209, 24951209, 0 committed:false) SnapCreateDur 27.469µs SnapOpenDur 960.332µs 2016-04-07T02:08:38.402-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29870000 2016-04-07T02:08:38.626-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29880000 2016-04-07T02:08:38.715-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24960000 QueuedCount 70 2016-04-07T02:08:38.859-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29890000 2016-04-07T02:08:39.092-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29900000 2016-04-07T02:08:39.326-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29910000 2016-04-07T02:08:39.332-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.231µs CommitTime 596.182967ms TotalFlushTime 18m25.979328429s TotalCommitTime 2m1.031910701s 2016-04-07T02:08:39.332-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 24960070, 24960070, 193 committed:true 2016-04-07T02:08:39.332-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 24960070, 24960070, 193 committed:true) SnapCreateDur 606.478624ms SnapOpenDur 110.451µs 2016-04-07T02:08:39.961-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:6692415518, Disk:18407759872, ExtraSnapshotData:0, Fragmentation:63.643509234495085%, mode : circular 2016-04-07T02:08:39.961-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:39.961-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:08:39.966-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:39.971-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:39.975-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:39.976-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29920000 2016-04-07T02:08:39.982-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:39.994-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29930000 2016-04-07T02:08:40.009-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29940000 2016-04-07T02:08:40.218-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29950000 2016-04-07T02:08:40.331-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24970000 QueuedCount 37972 2016-04-07T02:08:40.449-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29960000 2016-04-07T02:08:40.668-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29970000 2016-04-07T02:08:40.684-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24980000 QueuedCount 27972 2016-04-07T02:08:40.899-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29980000 2016-04-07T02:08:41.064-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 24990000 QueuedCount 17972 2016-04-07T02:08:41.118-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 29990000 2016-04-07T02:08:41.343-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30000000 2016-04-07T02:08:41.417-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25000000 QueuedCount 7972 2016-04-07T02:08:41.565-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30010000 2016-04-07T02:08:41.705-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25007972, 25007972, 0 committed:false 2016-04-07T02:08:41.706-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25007972, 25007972, 0 committed:false) SnapCreateDur 33.869µs SnapOpenDur 875.677µs 2016-04-07T02:08:41.713-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:41.718-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:41.722-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:41.727-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:41.731-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:41.789-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30020000 2016-04-07T02:08:41.856-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25010000 QueuedCount 50000 2016-04-07T02:08:42.01-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30030000 2016-04-07T02:08:42.236-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30040000 2016-04-07T02:08:42.25-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25020000 QueuedCount 46739 2016-04-07T02:08:42.471-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30050000 2016-04-07T02:08:42.664-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25030000 QueuedCount 36739 2016-04-07T02:08:42.699-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30060000 2016-04-07T02:08:42.929-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30070000 2016-04-07T02:08:43.076-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25040000 QueuedCount 26739 2016-04-07T02:08:43.158-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30080000 2016-04-07T02:08:43.393-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30090000 2016-04-07T02:08:43.433-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25050000 QueuedCount 16739 2016-04-07T02:08:43.627-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30100000 2016-04-07T02:08:43.84-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25060000 QueuedCount 6739 2016-04-07T02:08:43.863-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30110000 2016-04-07T02:08:44.09-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30120000 2016-04-07T02:08:44.151-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25066739, 25066739, 0 committed:false 2016-04-07T02:08:44.152-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25066739, 25066739, 0 committed:false) SnapCreateDur 31.88µs SnapOpenDur 223.144µs 2016-04-07T02:08:44.327-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30130000 2016-04-07T02:08:44.409-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25070000 QueuedCount 5451 2016-04-07T02:08:44.558-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30140000 2016-04-07T02:08:44.788-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30150000 2016-04-07T02:08:45.022-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30160000 2016-04-07T02:08:45.141-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.597µs CommitTime 486.095118ms TotalFlushTime 18m30.919008675s TotalCommitTime 2m1.518005819s 2016-04-07T02:08:45.141-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25075451, 25075451, 194 committed:true 2016-04-07T02:08:45.141-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25075451, 25075451, 194 committed:true) SnapCreateDur 495.067903ms SnapOpenDur 97.218µs 2016-04-07T02:08:45.145-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:45.149-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:45.153-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:45.157-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:45.161-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:45.251-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30170000 2016-04-07T02:08:45.365-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25080000 QueuedCount 49291 2016-04-07T02:08:45.483-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30180000 2016-04-07T02:08:45.715-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30190000 2016-04-07T02:08:45.729-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25090000 QueuedCount 39291 2016-04-07T02:08:45.939-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30200000 2016-04-07T02:08:46.046-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25100000 QueuedCount 29291 2016-04-07T02:08:46.163-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30210000 2016-04-07T02:08:46.385-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30220000 2016-04-07T02:08:46.421-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25110000 QueuedCount 19291 2016-04-07T02:08:46.609-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30230000 2016-04-07T02:08:46.799-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25120000 QueuedCount 9291 2016-04-07T02:08:46.839-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30240000 2016-04-07T02:08:47.074-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30250000 2016-04-07T02:08:47.179-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25129291, 25129291, 0 committed:false 2016-04-07T02:08:47.179-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25129291, 25129291, 0 committed:false) SnapCreateDur 29.313µs SnapOpenDur 532.784µs 2016-04-07T02:08:47.184-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:47.189-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:47.193-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:47.194-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25130000 QueuedCount 26551 2016-04-07T02:08:47.202-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:47.205-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:47.294-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30260000 2016-04-07T02:08:47.527-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30270000 2016-04-07T02:08:47.649-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25140000 QueuedCount 43339 2016-04-07T02:08:47.763-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30280000 2016-04-07T02:08:48-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30290000 2016-04-07T02:08:48.049-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25150000 QueuedCount 33339 2016-04-07T02:08:48.219-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30300000 2016-04-07T02:08:48.435-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25160000 QueuedCount 23339 2016-04-07T02:08:48.439-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30310000 2016-04-07T02:08:48.673-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30320000 2016-04-07T02:08:48.848-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25170000 QueuedCount 13339 2016-04-07T02:08:48.908-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30330000 2016-04-07T02:08:49.143-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30340000 2016-04-07T02:08:49.272-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25180000 QueuedCount 3339 2016-04-07T02:08:49.368-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30350000 2016-04-07T02:08:49.373-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25183339, 25183339, 0 committed:false 2016-04-07T02:08:49.375-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25183339, 25183339, 0 committed:false) SnapCreateDur 41.147µs SnapOpenDur 1.462596ms 2016-04-07T02:08:49.594-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30360000 2016-04-07T02:08:49.814-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25190000 QueuedCount 2238 2016-04-07T02:08:49.823-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30370000 2016-04-07T02:08:50.058-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30380000 2016-04-07T02:08:50.29-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30390000 2016-04-07T02:08:50.508-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30400000 2016-04-07T02:08:50.541-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 33.58µs CommitTime 572.71877ms TotalFlushTime 18m35.645996973s TotalCommitTime 2m2.090724589s 2016-04-07T02:08:50.541-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25192238, 25192238, 195 committed:true 2016-04-07T02:08:50.541-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25192238, 25192238, 195 committed:true) SnapCreateDur 581.909011ms SnapOpenDur 68.454µs 2016-04-07T02:08:50.546-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:50.55-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:50.554-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:50.558-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:50.562-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:50.73-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30410000 2016-04-07T02:08:50.87-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25200000 QueuedCount 46361 2016-04-07T02:08:50.965-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30420000 2016-04-07T02:08:51.192-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30430000 2016-04-07T02:08:51.262-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25210000 QueuedCount 36361 2016-04-07T02:08:51.414-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30440000 2016-04-07T02:08:51.64-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30450000 2016-04-07T02:08:51.647-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25220000 QueuedCount 26361 2016-04-07T02:08:51.86-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30460000 2016-04-07T02:08:52.035-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25230000 QueuedCount 16361 2016-04-07T02:08:52.093-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30470000 2016-04-07T02:08:52.312-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30480000 2016-04-07T02:08:52.431-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25240000 QueuedCount 6361 2016-04-07T02:08:52.529-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30490000 2016-04-07T02:08:52.684-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25246361, 25246361, 0 committed:false 2016-04-07T02:08:52.684-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25246361, 25246361, 0 committed:false) SnapCreateDur 40.07µs SnapOpenDur 708.733µs 2016-04-07T02:08:52.689-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:52.692-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:52.696-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:52.699-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:52.703-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:52.748-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30500000 2016-04-07T02:08:52.941-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25250000 QueuedCount 49110 2016-04-07T02:08:52.979-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30510000 2016-04-07T02:08:53.204-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30520000 2016-04-07T02:08:53.313-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25260000 QueuedCount 39110 2016-04-07T02:08:53.42-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30530000 2016-04-07T02:08:53.619-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25270000 QueuedCount 29110 2016-04-07T02:08:53.641-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30540000 2016-04-07T02:08:53.86-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30550000 2016-04-07T02:08:53.982-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25280000 QueuedCount 19110 2016-04-07T02:08:54.08-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30560000 2016-04-07T02:08:54.3-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30570000 2016-04-07T02:08:54.351-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25290000 QueuedCount 9110 2016-04-07T02:08:54.528-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30580000 2016-04-07T02:08:54.727-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25299110, 25299110, 0 committed:false 2016-04-07T02:08:54.727-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25299110, 25299110, 0 committed:false) SnapCreateDur 32.836µs SnapOpenDur 427.676µs 2016-04-07T02:08:54.738-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25300000 QueuedCount 8228 2016-04-07T02:08:54.75-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30590000 2016-04-07T02:08:54.979-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30600000 2016-04-07T02:08:55.205-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30610000 2016-04-07T02:08:55.436-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30620000 2016-04-07T02:08:55.665-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30630000 2016-04-07T02:08:55.846-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.458µs CommitTime 574.082323ms TotalFlushTime 18m40.286072796s TotalCommitTime 2m2.664806912s 2016-04-07T02:08:55.846-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25308228, 25308228, 196 committed:true 2016-04-07T02:08:55.846-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25308228, 25308228, 196 committed:true) SnapCreateDur 583.772237ms SnapOpenDur 66.463µs 2016-04-07T02:08:55.85-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:55.854-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:55.858-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:55.862-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:55.866-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:55.873-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25310000 QueuedCount 48740 2016-04-07T02:08:55.902-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30640000 2016-04-07T02:08:56.12-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30650000 2016-04-07T02:08:56.343-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30660000 2016-04-07T02:08:56.437-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25320000 QueuedCount 38740 2016-04-07T02:08:56.583-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30670000 2016-04-07T02:08:56.8-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25330000 QueuedCount 28740 2016-04-07T02:08:56.814-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30680000 2016-04-07T02:08:57.094-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30690000 2016-04-07T02:08:57.167-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25340000 QueuedCount 18740 2016-04-07T02:08:57.334-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30700000 2016-04-07T02:08:57.56-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25350000 QueuedCount 8740 2016-04-07T02:08:57.654-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30710000 2016-04-07T02:08:57.889-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25358740, 25358740, 0 committed:false 2016-04-07T02:08:57.89-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25358740, 25358740, 0 committed:false) SnapCreateDur 30.29µs SnapOpenDur 1.158008ms 2016-04-07T02:08:57.894-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:57.897-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:57.901-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:57.905-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:57.909-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:08:57.977-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30720000 2016-04-07T02:08:58.012-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25360000 QueuedCount 48129 2016-04-07T02:08:58.165-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30730000 2016-04-07T02:08:58.385-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25370000 QueuedCount 38129 2016-04-07T02:08:58.503-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30740000 2016-04-07T02:08:58.717-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30750000 2016-04-07T02:08:58.767-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25380000 QueuedCount 28129 2016-04-07T02:08:58.947-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30760000 2016-04-07T02:08:59.098-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25390000 QueuedCount 18129 2016-04-07T02:08:59.168-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30770000 2016-04-07T02:08:59.394-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30780000 2016-04-07T02:08:59.482-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25400000 QueuedCount 8129 2016-04-07T02:08:59.615-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30790000 2016-04-07T02:08:59.803-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25408129, 25408129, 0 committed:false 2016-04-07T02:08:59.804-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25408129, 25408129, 0 committed:false) SnapCreateDur 27.784µs SnapOpenDur 1.604438ms 2016-04-07T02:08:59.838-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30800000 2016-04-07T02:09:00.052-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30810000 2016-04-07T02:09:00.074-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25410000 QueuedCount 6762 2016-04-07T02:09:00.276-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30820000 2016-04-07T02:09:00.495-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30830000 2016-04-07T02:09:00.715-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30840000 2016-04-07T02:09:00.942-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30850000 2016-04-07T02:09:00.969-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.936µs CommitTime 630.941531ms TotalFlushTime 18m44.694750321s TotalCommitTime 2m3.295748443s 2016-04-07T02:09:00.969-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25416762, 25416762, 197 committed:true 2016-04-07T02:09:00.969-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25416762, 25416762, 197 committed:true) SnapCreateDur 640.636293ms SnapOpenDur 54.132µs 2016-04-07T02:09:00.974-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:00.978-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:00.981-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:00.985-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:00.989-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:01.128-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25420000 QueuedCount 50000 2016-04-07T02:09:01.169-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30860000 2016-04-07T02:09:01.393-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30870000 2016-04-07T02:09:01.508-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25430000 QueuedCount 40436 2016-04-07T02:09:01.62-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30880000 2016-04-07T02:09:01.848-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30890000 2016-04-07T02:09:01.883-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25440000 QueuedCount 30436 2016-04-07T02:09:02.073-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30900000 2016-04-07T02:09:02.244-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25450000 QueuedCount 20436 2016-04-07T02:09:02.295-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30910000 2016-04-07T02:09:02.518-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30920000 2016-04-07T02:09:02.611-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25460000 QueuedCount 10436 2016-04-07T02:09:02.738-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30930000 2016-04-07T02:09:02.923-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25470000 QueuedCount 436 2016-04-07T02:09:02.963-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30940000 2016-04-07T02:09:02.993-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25470436, 25470436, 0 committed:false 2016-04-07T02:09:02.993-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25470436, 25470436, 0 committed:false) SnapCreateDur 30.917µs SnapOpenDur 450.693µs 2016-04-07T02:09:02.997-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:03.001-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:03.005-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:03.009-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:03.013-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:03.188-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30950000 2016-04-07T02:09:03.418-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30960000 2016-04-07T02:09:03.438-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25480000 QueuedCount 43671 2016-04-07T02:09:03.645-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30970000 2016-04-07T02:09:03.83-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25490000 QueuedCount 33671 2016-04-07T02:09:03.871-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30980000 2016-04-07T02:09:04.1-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 30990000 2016-04-07T02:09:04.213-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25500000 QueuedCount 23671 2016-04-07T02:09:04.322-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31000000 2016-04-07T02:09:04.547-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31010000 2016-04-07T02:09:04.602-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25510000 QueuedCount 13671 2016-04-07T02:09:04.778-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31020000 2016-04-07T02:09:04.992-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25520000 QueuedCount 3671 2016-04-07T02:09:04.999-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31030000 2016-04-07T02:09:05.154-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25523671, 25523671, 0 committed:false 2016-04-07T02:09:05.154-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25523671, 25523671, 0 committed:false) SnapCreateDur 44.365µs SnapOpenDur 407.19µs 2016-04-07T02:09:05.221-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31040000 2016-04-07T02:09:05.443-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31050000 2016-04-07T02:09:05.616-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25530000 QueuedCount 2316 2016-04-07T02:09:05.665-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31060000 2016-04-07T02:09:05.887-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31070000 2016-04-07T02:09:06.108-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31080000 2016-04-07T02:09:06.281-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 29.38µs CommitTime 562.369477ms TotalFlushTime 18m49.350210184s TotalCommitTime 2m3.85811792s 2016-04-07T02:09:06.281-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25532316, 25532316, 198 committed:true 2016-04-07T02:09:06.281-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25532316, 25532316, 198 committed:true) SnapCreateDur 571.807762ms SnapOpenDur 51.916µs 2016-04-07T02:09:06.285-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:06.289-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:06.294-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:06.298-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:06.302-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:06.336-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31090000 2016-04-07T02:09:06.562-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31100000 2016-04-07T02:09:06.698-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25540000 QueuedCount 45366 2016-04-07T02:09:06.788-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31110000 2016-04-07T02:09:07.014-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31120000 2016-04-07T02:09:07.027-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25550000 QueuedCount 35366 2016-04-07T02:09:07.241-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31130000 2016-04-07T02:09:07.366-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25560000 QueuedCount 25366 2016-04-07T02:09:07.459-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31140000 2016-04-07T02:09:07.72-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25570000 QueuedCount 15366 2016-04-07T02:09:07.758-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31150000 2016-04-07T02:09:07.997-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31160000 2016-04-07T02:09:08.074-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25580000 QueuedCount 5366 2016-04-07T02:09:08.242-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25585366, 25585366, 0 committed:false 2016-04-07T02:09:08.244-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25585366, 25585366, 0 committed:false) SnapCreateDur 53.689µs SnapOpenDur 2.07748ms 2016-04-07T02:09:08.249-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:08.253-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:08.257-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:08.26-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:08.264-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:08.293-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31170000 2016-04-07T02:09:08.527-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31180000 2016-04-07T02:09:08.578-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25590000 QueuedCount 47851 2016-04-07T02:09:08.764-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31190000 2016-04-07T02:09:09.007-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25600000 QueuedCount 37851 2016-04-07T02:09:09.009-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31200000 2016-04-07T02:09:09.225-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31210000 2016-04-07T02:09:09.364-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25610000 QueuedCount 27851 2016-04-07T02:09:09.373-07:00 [Info] connected with 1 indexers 2016-04-07T02:09:09.373-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:09:09.374-07:00 [Info] client load stats {} 2016-04-07T02:09:09.451-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31220000 2016-04-07T02:09:09.653-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":5590664,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":1754316492,"bucket-1:myindex:avg_ts_items_count":40350,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":6846252428,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":19499196416,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":21547,"bucket-1:myindex:frag_percent":64,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":2680893180,"bucket-1:myindex:items_count":25416762,"bucket-1:myindex:num_commits":198,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":25616304,"bucket-1:myindex:num_docs_pending":1574,"bucket-1:myindex:num_docs_processed":25523671,"bucket-1:myindex:num_docs_queued":5538980,"bucket-1:myindex:num_flush_queued":25637851,"bucket-1:myindex:num_items_flushed":25616304,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":316,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":1963538237,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"198 123858117920 -608781460675664000","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"25616305 145583894406 906991427218481570","bucket-1:myindex:timings/storage_info":"1029 7784158 112861818762","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"197 69100711 26962117101363","bucket-1:myindex:timings/storage_meta_set":"198 69778293 26791001854803","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"198 8395699 394011670229","bucket-1:myindex:timings/storage_set":"51232608 973522710024 6714202343170667472","bucket-1:myindex:timings/storage_snapshot_close":"117 88826442 200350462779638","bucket-1:myindex:timings/storage_snapshot_create":"118 103132966 131170996184148","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":31228515,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"344 578055415 27049790709948911","bucket-1:ts_queue_size":635,"indexer_state":"Active","memory_quota":31457280000,"memory_used":22438447104,"memory_used_storage":18874368000,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"4141 738960092 336394610280798","uptime":"40m1.345440693s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:09:09.68-07:00 [Info] memstats {"Alloc":3202522696, "TotalAlloc":31765237648, "Sys":9854296392, "Lookups":8872, "Mallocs":901241402,"Frees":820842160, "HeapAlloc":3202522696, "HeapSys":9381183488, "HeapIdle":6114549760, "HeapInuse":3266633728,"HeapReleased":6043860992, "HeapObjects":80399242,"GCSys":297445376, "LastGC":1460020075065723701,"PauseTotalNs":49226677, "PauseNs":[1711813], "NumGC":38} 2016-04-07T02:09:09.684-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31230000 2016-04-07T02:09:09.797-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25620000 QueuedCount 17851 2016-04-07T02:09:09.914-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31240000 2016-04-07T02:09:09.962-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:6870222894, Disk:19774464000, ExtraSnapshotData:0, Fragmentation:65.25709675872884%, mode : circular 2016-04-07T02:09:09.962-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:09:10.136-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31250000 2016-04-07T02:09:10.232-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25630000 QueuedCount 7851 2016-04-07T02:09:10.359-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31260000 2016-04-07T02:09:10.566-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25637851, 25637851, 0 committed:false 2016-04-07T02:09:10.567-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25637851, 25637851, 0 committed:false) SnapCreateDur 30.206µs SnapOpenDur 1.005836ms 2016-04-07T02:09:10.578-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31270000 2016-04-07T02:09:10.798-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31280000 2016-04-07T02:09:10.849-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25640000 QueuedCount 7288 2016-04-07T02:09:11.015-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31290000 2016-04-07T02:09:11.238-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31300000 2016-04-07T02:09:11.453-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31310000 2016-04-07T02:09:11.675-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31320000 2016-04-07T02:09:11.818-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.345µs CommitTime 617.299544ms TotalFlushTime 18m54.182671207s TotalCommitTime 2m4.475417464s 2016-04-07T02:09:11.818-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25647288, 25647288, 199 committed:true 2016-04-07T02:09:11.818-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25647288, 25647288, 199 committed:true) SnapCreateDur 626.61737ms SnapOpenDur 60.615µs 2016-04-07T02:09:11.822-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:11.826-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:11.829-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:11.833-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:11.836-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:11.908-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31330000 2016-04-07T02:09:12.002-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25650000 QueuedCount 50000 2016-04-07T02:09:12.137-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31340000 2016-04-07T02:09:12.368-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31350000 2016-04-07T02:09:12.389-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25660000 QueuedCount 41340 2016-04-07T02:09:12.59-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31360000 2016-04-07T02:09:12.772-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25670000 QueuedCount 31340 2016-04-07T02:09:12.807-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31370000 2016-04-07T02:09:13.02-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31380000 2016-04-07T02:09:13.098-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25680000 QueuedCount 21340 2016-04-07T02:09:13.24-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31390000 2016-04-07T02:09:13.455-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31400000 2016-04-07T02:09:13.486-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25690000 QueuedCount 11340 2016-04-07T02:09:13.672-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31410000 2016-04-07T02:09:13.876-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25700000 QueuedCount 1340 2016-04-07T02:09:13.893-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31420000 2016-04-07T02:09:13.946-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25701340, 25701340, 0 committed:false 2016-04-07T02:09:13.947-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25701340, 25701340, 0 committed:false) SnapCreateDur 44.63µs SnapOpenDur 882.007µs 2016-04-07T02:09:13.951-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:13.955-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:13.958-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:13.962-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:13.966-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:14.113-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31430000 2016-04-07T02:09:14.316-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25710000 QueuedCount 44980 2016-04-07T02:09:14.334-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31440000 2016-04-07T02:09:14.549-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31450000 2016-04-07T02:09:14.671-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25720000 QueuedCount 34980 2016-04-07T02:09:14.768-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31460000 2016-04-07T02:09:14.997-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31470000 2016-04-07T02:09:15.029-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25730000 QueuedCount 24980 2016-04-07T02:09:15.234-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31480000 2016-04-07T02:09:15.398-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25740000 QueuedCount 14980 2016-04-07T02:09:15.462-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31490000 2016-04-07T02:09:15.684-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31500000 2016-04-07T02:09:15.794-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25750000 QueuedCount 4980 2016-04-07T02:09:15.913-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31510000 2016-04-07T02:09:15.964-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25754980, 25754980, 0 committed:false 2016-04-07T02:09:15.966-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25754980, 25754980, 0 committed:false) SnapCreateDur 28.128µs SnapOpenDur 1.435237ms 2016-04-07T02:09:16.14-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31520000 2016-04-07T02:09:16.357-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31530000 2016-04-07T02:09:16.448-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25760000 QueuedCount 3946 2016-04-07T02:09:16.576-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31540000 2016-04-07T02:09:16.795-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31550000 2016-04-07T02:09:17.022-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31560000 2016-04-07T02:09:17.247-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31570000 2016-04-07T02:09:17.468-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31580000 2016-04-07T02:09:17.695-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31590000 2016-04-07T02:09:17.925-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31600000 2016-04-07T02:09:18.154-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31610000 2016-04-07T02:09:18.402-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31620000 2016-04-07T02:09:18.609-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31630000 2016-04-07T02:09:18.83-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31640000 2016-04-07T02:09:19.054-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31650000 2016-04-07T02:09:19.275-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31660000 2016-04-07T02:09:19.493-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31670000 2016-04-07T02:09:19.713-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31680000 2016-04-07T02:09:19.945-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31690000 2016-04-07T02:09:20.168-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31700000 2016-04-07T02:09:20.384-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31710000 2016-04-07T02:09:20.608-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31720000 2016-04-07T02:09:20.825-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31730000 2016-04-07T02:09:21.044-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31740000 2016-04-07T02:09:21.262-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31750000 2016-04-07T02:09:21.484-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31760000 2016-04-07T02:09:21.709-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31770000 2016-04-07T02:09:21.935-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31780000 2016-04-07T02:09:22.159-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31790000 2016-04-07T02:09:22.379-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31800000 2016-04-07T02:09:22.598-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31810000 2016-04-07T02:09:22.826-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31820000 2016-04-07T02:09:23.043-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31830000 2016-04-07T02:09:23.275-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31840000 2016-04-07T02:09:23.496-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31850000 2016-04-07T02:09:23.715-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31860000 2016-04-07T02:09:23.938-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31870000 2016-04-07T02:09:24.11-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.983µs CommitTime 7.540975946s TotalFlushTime 18m58.849678385s TotalCommitTime 2m12.01639341s 2016-04-07T02:09:24.11-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25763946, 25763946, 200 committed:true 2016-04-07T02:09:24.11-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25763946, 25763946, 200 committed:true) SnapCreateDur 7.550907521s SnapOpenDur 76.434µs 2016-04-07T02:09:24.163-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31880000 2016-04-07T02:09:24.311-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25770000 QueuedCount 2572 2016-04-07T02:09:24.386-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31890000 2016-04-07T02:09:24.529-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 28.938µs CommitTime 84.689187ms TotalFlushTime 18m59.159858151s TotalCommitTime 2m12.101082597s 2016-04-07T02:09:24.529-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25772572, 25772572, 201 committed:true 2016-04-07T02:09:24.53-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25772572, 25772572, 201 committed:true) SnapCreateDur 94.819877ms SnapOpenDur 53.97µs 2016-04-07T02:09:24.535-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:24.54-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:24.545-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:24.549-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:24.553-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:24.606-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31900000 2016-04-07T02:09:24.761-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25780000 QueuedCount 42469 2016-04-07T02:09:24.831-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31910000 2016-04-07T02:09:25.05-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31920000 2016-04-07T02:09:25.115-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25790000 QueuedCount 32469 2016-04-07T02:09:25.273-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31930000 2016-04-07T02:09:25.48-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25800000 QueuedCount 22469 2016-04-07T02:09:25.493-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31940000 2016-04-07T02:09:25.719-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31950000 2016-04-07T02:09:25.837-07:00 [Info] Indexer::ReadMemstats Time Taken 14.617002ms 2016-04-07T02:09:25.854-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25810000 QueuedCount 12469 2016-04-07T02:09:25.944-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31960000 2016-04-07T02:09:26.169-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31970000 2016-04-07T02:09:26.244-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25820000 QueuedCount 2469 2016-04-07T02:09:26.328-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25822469, 25822469, 0 committed:false 2016-04-07T02:09:26.328-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25822469, 25822469, 0 committed:false) SnapCreateDur 28.448µs SnapOpenDur 682.455µs 2016-04-07T02:09:26.332-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:26.336-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:26.341-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:26.346-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:26.351-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:26.39-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31980000 2016-04-07T02:09:26.618-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 31990000 2016-04-07T02:09:26.74-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25830000 QueuedCount 40912 2016-04-07T02:09:26.848-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32000000 2016-04-07T02:09:27.054-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25840000 QueuedCount 30912 2016-04-07T02:09:27.065-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32010000 2016-04-07T02:09:27.293-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32020000 2016-04-07T02:09:27.424-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25850000 QueuedCount 20912 2016-04-07T02:09:27.521-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32030000 2016-04-07T02:09:27.734-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32040000 2016-04-07T02:09:27.806-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25860000 QueuedCount 10912 2016-04-07T02:09:27.956-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32050000 2016-04-07T02:09:28.175-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32060000 2016-04-07T02:09:28.197-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25870000 QueuedCount 912 2016-04-07T02:09:28.276-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25870912, 25870912, 0 committed:false 2016-04-07T02:09:28.276-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25870912, 25870912, 0 committed:false) SnapCreateDur 26.871µs SnapOpenDur 85.092µs 2016-04-07T02:09:28.281-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:28.285-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:28.29-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:28.294-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:28.299-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:28.412-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32070000 2016-04-07T02:09:28.645-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32080000 2016-04-07T02:09:28.759-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25880000 QueuedCount 44386 2016-04-07T02:09:28.877-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32090000 2016-04-07T02:09:29.101-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32100000 2016-04-07T02:09:29.105-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25890000 QueuedCount 34386 2016-04-07T02:09:29.318-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32110000 2016-04-07T02:09:29.465-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25900000 QueuedCount 24386 2016-04-07T02:09:29.54-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32120000 2016-04-07T02:09:29.755-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32130000 2016-04-07T02:09:29.833-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25910000 QueuedCount 14386 2016-04-07T02:09:29.974-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32140000 2016-04-07T02:09:30.14-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25920000 QueuedCount 4386 2016-04-07T02:09:30.184-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32150000 2016-04-07T02:09:30.343-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25924386, 25924386, 0 committed:false 2016-04-07T02:09:30.344-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25924386, 25924386, 0 committed:false) SnapCreateDur 30.787µs SnapOpenDur 318.509µs 2016-04-07T02:09:30.4-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32160000 2016-04-07T02:09:30.62-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32170000 2016-04-07T02:09:30.79-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25930000 QueuedCount 3204 2016-04-07T02:09:30.84-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32180000 2016-04-07T02:09:31.062-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32190000 2016-04-07T02:09:31.279-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32200000 2016-04-07T02:09:31.5-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32210000 2016-04-07T02:09:31.534-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.958µs CommitTime 565.286362ms TotalFlushTime 19m5.472450382s TotalCommitTime 2m12.666368959s 2016-04-07T02:09:31.534-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25933204, 25933204, 202 committed:true 2016-04-07T02:09:31.534-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25933204, 25933204, 202 committed:true) SnapCreateDur 575.734689ms SnapOpenDur 48.313µs 2016-04-07T02:09:31.538-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:31.542-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:31.546-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:31.55-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:31.554-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:31.72-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32220000 2016-04-07T02:09:31.947-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32230000 2016-04-07T02:09:31.964-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25940000 QueuedCount 45978 2016-04-07T02:09:32.178-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32240000 2016-04-07T02:09:32.331-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25950000 QueuedCount 35978 2016-04-07T02:09:32.41-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32250000 2016-04-07T02:09:32.635-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32260000 2016-04-07T02:09:32.691-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25960000 QueuedCount 25978 2016-04-07T02:09:32.855-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32270000 2016-04-07T02:09:32.999-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25970000 QueuedCount 15978 2016-04-07T02:09:33.079-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32280000 2016-04-07T02:09:33.304-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32290000 2016-04-07T02:09:33.369-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25980000 QueuedCount 5978 2016-04-07T02:09:33.523-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32300000 2016-04-07T02:09:33.606-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 25985978, 25985978, 0 committed:false 2016-04-07T02:09:33.607-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 25985978, 25985978, 0 committed:false) SnapCreateDur 23.894µs SnapOpenDur 1.380735ms 2016-04-07T02:09:33.611-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:33.615-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:33.619-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:33.623-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:33.626-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:33.785-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32310000 2016-04-07T02:09:33.971-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 25990000 QueuedCount 47449 2016-04-07T02:09:34.331-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32320000 2016-04-07T02:09:34.483-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32330000 2016-04-07T02:09:34.84-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32340000 2016-04-07T02:09:34.856-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32350000 2016-04-07T02:09:34.879-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32360000 2016-04-07T02:09:34.91-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26000000 QueuedCount 38598 2016-04-07T02:09:35.108-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32370000 2016-04-07T02:09:35.303-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26010000 QueuedCount 28598 2016-04-07T02:09:35.337-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32380000 2016-04-07T02:09:35.56-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32390000 2016-04-07T02:09:35.686-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26020000 QueuedCount 18598 2016-04-07T02:09:35.779-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32400000 2016-04-07T02:09:36.004-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32410000 2016-04-07T02:09:36.064-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26030000 QueuedCount 8598 2016-04-07T02:09:36.225-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32420000 2016-04-07T02:09:36.37-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26038598, 26038598, 0 committed:false 2016-04-07T02:09:36.371-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26038598, 26038598, 0 committed:false) SnapCreateDur 42.734µs SnapOpenDur 1.003149ms 2016-04-07T02:09:36.453-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32430000 2016-04-07T02:09:36.679-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32440000 2016-04-07T02:09:36.7-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26040000 QueuedCount 7482 2016-04-07T02:09:36.898-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32450000 2016-04-07T02:09:37.119-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32460000 2016-04-07T02:09:37.345-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32470000 2016-04-07T02:09:37.517-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.083µs CommitTime 551.416497ms TotalFlushTime 19m10.800757111s TotalCommitTime 2m13.217785456s 2016-04-07T02:09:37.517-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26047482, 26047482, 203 committed:true 2016-04-07T02:09:37.517-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26047482, 26047482, 203 committed:true) SnapCreateDur 560.483975ms SnapOpenDur 62.711µs 2016-04-07T02:09:37.522-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:37.527-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:37.532-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:37.536-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:37.541-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:37.573-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32480000 2016-04-07T02:09:37.676-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26050000 QueuedCount 49443 2016-04-07T02:09:37.802-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32490000 2016-04-07T02:09:38.009-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26060000 QueuedCount 39443 2016-04-07T02:09:38.03-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32500000 2016-04-07T02:09:38.257-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32510000 2016-04-07T02:09:38.302-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26070000 QueuedCount 29443 2016-04-07T02:09:38.489-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32520000 2016-04-07T02:09:38.668-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26080000 QueuedCount 19443 2016-04-07T02:09:38.74-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32530000 2016-04-07T02:09:38.97-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32540000 2016-04-07T02:09:39.019-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26090000 QueuedCount 9443 2016-04-07T02:09:39.191-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32550000 2016-04-07T02:09:39.382-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26099443, 26099443, 0 committed:false 2016-04-07T02:09:39.382-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26099443, 26099443, 0 committed:false) SnapCreateDur 27.31µs SnapOpenDur 588.566µs 2016-04-07T02:09:39.387-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:39.391-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:39.394-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26100000 QueuedCount 24669 2016-04-07T02:09:39.396-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:39.4-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:39.404-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:39.418-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32560000 2016-04-07T02:09:39.652-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32570000 2016-04-07T02:09:39.829-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26110000 QueuedCount 44563 2016-04-07T02:09:39.875-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32580000 2016-04-07T02:09:39.963-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:7001939882, Disk:20360749056, ExtraSnapshotData:0, Fragmentation:65.61059780884321%, mode : circular 2016-04-07T02:09:39.963-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:09:40.092-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32590000 2016-04-07T02:09:40.201-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26120000 QueuedCount 34563 2016-04-07T02:09:40.32-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32600000 2016-04-07T02:09:40.544-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32610000 2016-04-07T02:09:40.596-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26130000 QueuedCount 24563 2016-04-07T02:09:40.763-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32620000 2016-04-07T02:09:40.982-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32630000 2016-04-07T02:09:40.994-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26140000 QueuedCount 14563 2016-04-07T02:09:41.21-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32640000 2016-04-07T02:09:41.406-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26150000 QueuedCount 4563 2016-04-07T02:09:41.431-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32650000 2016-04-07T02:09:41.582-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26154563, 26154563, 0 committed:false 2016-04-07T02:09:41.582-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26154563, 26154563, 0 committed:false) SnapCreateDur 28.785µs SnapOpenDur 520.809µs 2016-04-07T02:09:41.65-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32660000 2016-04-07T02:09:41.854-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26160000 QueuedCount 2962 2016-04-07T02:09:41.879-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32670000 2016-04-07T02:09:42.101-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32680000 2016-04-07T02:09:42.33-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32690000 2016-04-07T02:09:42.515-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.66µs CommitTime 498.288417ms TotalFlushTime 19m15.201067668s TotalCommitTime 2m13.716073873s 2016-04-07T02:09:42.515-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26162962, 26162962, 204 committed:true 2016-04-07T02:09:42.515-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26162962, 26162962, 204 committed:true) SnapCreateDur 507.898039ms SnapOpenDur 64.717µs 2016-04-07T02:09:42.519-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:42.522-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:42.526-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:42.529-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:42.533-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:42.554-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32700000 2016-04-07T02:09:42.775-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32710000 2016-04-07T02:09:42.817-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26170000 QueuedCount 45990 2016-04-07T02:09:42.998-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32720000 2016-04-07T02:09:43.18-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26180000 QueuedCount 35990 2016-04-07T02:09:43.23-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32730000 2016-04-07T02:09:43.455-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32740000 2016-04-07T02:09:43.563-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26190000 QueuedCount 25990 2016-04-07T02:09:43.68-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32750000 2016-04-07T02:09:43.91-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32760000 2016-04-07T02:09:43.936-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26200000 QueuedCount 15990 2016-04-07T02:09:44.138-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32770000 2016-04-07T02:09:44.26-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26210000 QueuedCount 5990 2016-04-07T02:09:44.363-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32780000 2016-04-07T02:09:44.497-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26215990, 26215990, 0 committed:false 2016-04-07T02:09:44.499-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26215990, 26215990, 0 committed:false) SnapCreateDur 28.53µs SnapOpenDur 1.782412ms 2016-04-07T02:09:44.503-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:44.506-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:44.509-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:44.513-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:44.516-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:44.583-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32790000 2016-04-07T02:09:44.747-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26220000 QueuedCount 47190 2016-04-07T02:09:44.814-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32800000 2016-04-07T02:09:45.036-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32810000 2016-04-07T02:09:45.13-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26230000 QueuedCount 37190 2016-04-07T02:09:45.258-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32820000 2016-04-07T02:09:45.484-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32830000 2016-04-07T02:09:45.518-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26240000 QueuedCount 27190 2016-04-07T02:09:45.712-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32840000 2016-04-07T02:09:45.923-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32850000 2016-04-07T02:09:45.942-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26250000 QueuedCount 17190 2016-04-07T02:09:46.145-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32860000 2016-04-07T02:09:46.344-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26260000 QueuedCount 7190 2016-04-07T02:09:46.372-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32870000 2016-04-07T02:09:46.6-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32880000 2016-04-07T02:09:46.617-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26267190, 26267190, 0 committed:false 2016-04-07T02:09:46.619-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26267190, 26267190, 0 committed:false) SnapCreateDur 28.023µs SnapOpenDur 1.90137ms 2016-04-07T02:09:46.824-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32890000 2016-04-07T02:09:46.999-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26270000 QueuedCount 5408 2016-04-07T02:09:47.045-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32900000 2016-04-07T02:09:47.258-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32910000 2016-04-07T02:09:47.48-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32920000 2016-04-07T02:09:47.708-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32930000 2016-04-07T02:09:47.77-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 31.044µs CommitTime 577.897665ms TotalFlushTime 19m19.797057191s TotalCommitTime 2m14.293971538s 2016-04-07T02:09:47.77-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26275408, 26275408, 205 committed:true 2016-04-07T02:09:47.77-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26275408, 26275408, 205 committed:true) SnapCreateDur 587.609678ms SnapOpenDur 33.145µs 2016-04-07T02:09:47.774-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:47.779-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:47.782-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:47.786-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:47.79-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:47.925-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32940000 2016-04-07T02:09:48.108-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26280000 QueuedCount 46552 2016-04-07T02:09:48.144-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32950000 2016-04-07T02:09:48.373-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32960000 2016-04-07T02:09:48.479-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26290000 QueuedCount 36552 2016-04-07T02:09:48.605-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32970000 2016-04-07T02:09:48.834-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32980000 2016-04-07T02:09:48.869-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26300000 QueuedCount 26552 2016-04-07T02:09:49.059-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 32990000 2016-04-07T02:09:49.185-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26310000 QueuedCount 16552 2016-04-07T02:09:49.28-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33000000 2016-04-07T02:09:49.503-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33010000 2016-04-07T02:09:49.57-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26320000 QueuedCount 6552 2016-04-07T02:09:49.725-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33020000 2016-04-07T02:09:49.814-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26326552, 26326552, 0 committed:false 2016-04-07T02:09:49.816-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26326552, 26326552, 0 committed:false) SnapCreateDur 26.637µs SnapOpenDur 1.967367ms 2016-04-07T02:09:49.821-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:49.824-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:49.829-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:49.833-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:49.837-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:49.95-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33030000 2016-04-07T02:09:50.069-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26330000 QueuedCount 50000 2016-04-07T02:09:50.17-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33040000 2016-04-07T02:09:50.405-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33050000 2016-04-07T02:09:50.46-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26340000 QueuedCount 42731 2016-04-07T02:09:50.637-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33060000 2016-04-07T02:09:50.856-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26350000 QueuedCount 32731 2016-04-07T02:09:50.863-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33070000 2016-04-07T02:09:51.088-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33080000 2016-04-07T02:09:51.256-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26360000 QueuedCount 22731 2016-04-07T02:09:51.313-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33090000 2016-04-07T02:09:51.534-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33100000 2016-04-07T02:09:51.659-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26370000 QueuedCount 12731 2016-04-07T02:09:51.757-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33110000 2016-04-07T02:09:51.975-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33120000 2016-04-07T02:09:52.057-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26380000 QueuedCount 2731 2016-04-07T02:09:52.151-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26382731, 26382731, 0 committed:false 2016-04-07T02:09:52.152-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26382731, 26382731, 0 committed:false) SnapCreateDur 51.313µs SnapOpenDur 817.343µs 2016-04-07T02:09:52.2-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33130000 2016-04-07T02:09:52.417-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33140000 2016-04-07T02:09:52.579-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26390000 QueuedCount 1606 2016-04-07T02:09:52.638-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33150000 2016-04-07T02:09:52.869-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33160000 2016-04-07T02:09:53.099-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33170000 2016-04-07T02:09:53.18-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.471µs CommitTime 514.302266ms TotalFlushTime 19m24.606194476s TotalCommitTime 2m14.808273804s 2016-04-07T02:09:53.18-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26391606, 26391606, 206 committed:true 2016-04-07T02:09:53.18-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26391606, 26391606, 206 committed:true) SnapCreateDur 523.601629ms SnapOpenDur 67.365µs 2016-04-07T02:09:53.185-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:53.19-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:53.194-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:53.198-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:53.203-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:53.322-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33180000 2016-04-07T02:09:53.561-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33190000 2016-04-07T02:09:53.566-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26400000 QueuedCount 45963 2016-04-07T02:09:53.778-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33200000 2016-04-07T02:09:53.867-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26410000 QueuedCount 35963 2016-04-07T02:09:53.997-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33210000 2016-04-07T02:09:54.22-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33220000 2016-04-07T02:09:54.228-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26420000 QueuedCount 25963 2016-04-07T02:09:54.442-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33230000 2016-04-07T02:09:54.595-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26430000 QueuedCount 15963 2016-04-07T02:09:54.669-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33240000 2016-04-07T02:09:54.899-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33250000 2016-04-07T02:09:54.985-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26440000 QueuedCount 5963 2016-04-07T02:09:55.129-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33260000 2016-04-07T02:09:55.224-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26445963, 26445963, 0 committed:false 2016-04-07T02:09:55.225-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26445963, 26445963, 0 committed:false) SnapCreateDur 59.342µs SnapOpenDur 1.190982ms 2016-04-07T02:09:55.229-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:55.233-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:55.237-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:55.241-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:55.245-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:55.355-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33270000 2016-04-07T02:09:55.447-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26450000 QueuedCount 49978 2016-04-07T02:09:55.579-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33280000 2016-04-07T02:09:55.801-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33290000 2016-04-07T02:09:55.816-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26460000 QueuedCount 39978 2016-04-07T02:09:56.023-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33300000 2016-04-07T02:09:56.195-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26470000 QueuedCount 29978 2016-04-07T02:09:56.244-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33310000 2016-04-07T02:09:56.468-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33320000 2016-04-07T02:09:56.572-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26480000 QueuedCount 19978 2016-04-07T02:09:56.687-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33330000 2016-04-07T02:09:56.909-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33340000 2016-04-07T02:09:56.954-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26490000 QueuedCount 9978 2016-04-07T02:09:57.135-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33350000 2016-04-07T02:09:57.307-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26499978, 26499978, 0 committed:false 2016-04-07T02:09:57.309-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26499978, 26499978, 0 committed:false) SnapCreateDur 28.41µs SnapOpenDur 1.890586ms 2016-04-07T02:09:57.31-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26500000 QueuedCount 495 2016-04-07T02:09:57.36-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33360000 2016-04-07T02:09:57.584-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33370000 2016-04-07T02:09:57.806-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33380000 2016-04-07T02:09:58.034-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33390000 2016-04-07T02:09:58.256-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33400000 2016-04-07T02:09:58.483-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33410000 2016-04-07T02:09:58.517-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 28.677µs CommitTime 553.435292ms TotalFlushTime 19m29.287482416s TotalCommitTime 2m15.361709096s 2016-04-07T02:09:58.517-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26509117, 26509117, 207 committed:true 2016-04-07T02:09:58.517-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26509117, 26509117, 207 committed:true) SnapCreateDur 562.621025ms SnapOpenDur 44.246µs 2016-04-07T02:09:58.522-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:58.526-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:58.531-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26510000 QueuedCount 26043 2016-04-07T02:09:58.531-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:58.537-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:58.542-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:09:58.719-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33420000 2016-04-07T02:09:58.927-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26520000 QueuedCount 43231 2016-04-07T02:09:58.944-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33430000 2016-04-07T02:09:59.167-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33440000 2016-04-07T02:09:59.243-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26530000 QueuedCount 33231 2016-04-07T02:09:59.397-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33450000 2016-04-07T02:09:59.576-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26540000 QueuedCount 23231 2016-04-07T02:09:59.624-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33460000 2016-04-07T02:09:59.85-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33470000 2016-04-07T02:09:59.868-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26550000 QueuedCount 13231 2016-04-07T02:10:00.084-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33480000 2016-04-07T02:10:00.217-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26560000 QueuedCount 3231 2016-04-07T02:10:00.297-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33490000 2016-04-07T02:10:00.358-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26563231, 26563231, 0 committed:false 2016-04-07T02:10:00.359-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26563231, 26563231, 0 committed:false) SnapCreateDur 28.238µs SnapOpenDur 818.182µs 2016-04-07T02:10:00.364-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:00.369-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:00.375-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:00.38-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:00.385-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:00.525-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33500000 2016-04-07T02:10:00.721-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26570000 QueuedCount 46177 2016-04-07T02:10:00.748-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33510000 2016-04-07T02:10:00.973-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33520000 2016-04-07T02:10:01.119-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26580000 QueuedCount 36177 2016-04-07T02:10:01.198-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33530000 2016-04-07T02:10:01.424-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33540000 2016-04-07T02:10:01.51-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26590000 QueuedCount 26177 2016-04-07T02:10:01.659-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33550000 2016-04-07T02:10:01.882-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33560000 2016-04-07T02:10:01.912-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26600000 QueuedCount 16177 2016-04-07T02:10:02.108-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33570000 2016-04-07T02:10:02.311-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26610000 QueuedCount 6177 2016-04-07T02:10:02.338-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33580000 2016-04-07T02:10:02.559-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33590000 2016-04-07T02:10:02.561-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26616177, 26616177, 0 committed:false 2016-04-07T02:10:02.562-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26616177, 26616177, 0 committed:false) SnapCreateDur 30.289µs SnapOpenDur 456.453µs 2016-04-07T02:10:02.784-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33600000 2016-04-07T02:10:02.894-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26620000 QueuedCount 4868 2016-04-07T02:10:03.015-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33610000 2016-04-07T02:10:03.239-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33620000 2016-04-07T02:10:03.463-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33630000 2016-04-07T02:10:03.626-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.782µs CommitTime 539.631688ms TotalFlushTime 19m33.75595901s TotalCommitTime 2m15.901340784s 2016-04-07T02:10:03.627-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26624868, 26624868, 208 committed:true 2016-04-07T02:10:03.627-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26624868, 26624868, 208 committed:true) SnapCreateDur 549.652838ms SnapOpenDur 45.577µs 2016-04-07T02:10:03.631-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:03.636-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:03.64-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:03.644-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:03.648-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:03.68-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33640000 2016-04-07T02:10:03.91-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33650000 2016-04-07T02:10:03.947-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26630000 QueuedCount 48357 2016-04-07T02:10:04.135-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33660000 2016-04-07T02:10:04.303-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26640000 QueuedCount 38357 2016-04-07T02:10:04.359-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33670000 2016-04-07T02:10:04.599-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33680000 2016-04-07T02:10:04.667-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26650000 QueuedCount 28357 2016-04-07T02:10:04.825-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33690000 2016-04-07T02:10:05.025-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26660000 QueuedCount 18357 2016-04-07T02:10:05.049-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33700000 2016-04-07T02:10:05.284-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33710000 2016-04-07T02:10:05.404-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26670000 QueuedCount 8357 2016-04-07T02:10:05.506-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33720000 2016-04-07T02:10:05.725-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26678357, 26678357, 0 committed:false 2016-04-07T02:10:05.725-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26678357, 26678357, 0 committed:false) SnapCreateDur 28.248µs SnapOpenDur 326.659µs 2016-04-07T02:10:05.729-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33730000 2016-04-07T02:10:05.73-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:05.734-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:05.739-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:05.744-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:05.748-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:05.757-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26680000 QueuedCount 50000 2016-04-07T02:10:05.956-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33740000 2016-04-07T02:10:06.175-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33750000 2016-04-07T02:10:06.238-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26690000 QueuedCount 40096 2016-04-07T02:10:06.394-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33760000 2016-04-07T02:10:06.61-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26700000 QueuedCount 30096 2016-04-07T02:10:06.618-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33770000 2016-04-07T02:10:06.845-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33780000 2016-04-07T02:10:06.991-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26710000 QueuedCount 20096 2016-04-07T02:10:07.067-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33790000 2016-04-07T02:10:07.298-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33800000 2016-04-07T02:10:07.393-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26720000 QueuedCount 10096 2016-04-07T02:10:07.525-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33810000 2016-04-07T02:10:07.745-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33820000 2016-04-07T02:10:07.792-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26730000 QueuedCount 96 2016-04-07T02:10:07.798-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26730096, 26730096, 0 committed:false 2016-04-07T02:10:07.799-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26730096, 26730096, 0 committed:false) SnapCreateDur 31.705µs SnapOpenDur 624.229µs 2016-04-07T02:10:07.974-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33830000 2016-04-07T02:10:08.202-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33840000 2016-04-07T02:10:08.424-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33850000 2016-04-07T02:10:08.643-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33860000 2016-04-07T02:10:08.842-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 33.659µs CommitTime 508.228138ms TotalFlushTime 19m38.362208891s TotalCommitTime 2m16.409568922s 2016-04-07T02:10:08.842-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26739035, 26739035, 209 committed:true 2016-04-07T02:10:08.842-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26739035, 26739035, 209 committed:true) SnapCreateDur 517.967754ms SnapOpenDur 46.518µs 2016-04-07T02:10:08.846-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:08.851-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:08.855-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:08.859-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26740000 QueuedCount 34241 2016-04-07T02:10:08.86-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:08.864-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:08.871-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33870000 2016-04-07T02:10:09.093-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33880000 2016-04-07T02:10:09.319-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33890000 2016-04-07T02:10:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:10:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:10:09.374-07:00 [Info] client load stats {} 2016-04-07T02:10:09.388-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26750000 QueuedCount 42567 2016-04-07T02:10:09.552-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33900000 2016-04-07T02:10:09.654-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":7111829,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":1491078110,"bucket-1:myindex:avg_ts_items_count":27577,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":7154262501,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":20360749056,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":35096,"bucket-1:myindex:frag_percent":64,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":2801848770,"bucket-1:myindex:items_count":26624868,"bucket-1:myindex:num_commits":209,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":26757471,"bucket-1:myindex:num_docs_pending":849,"bucket-1:myindex:num_docs_processed":26721510,"bucket-1:myindex:num_docs_queued":7020832,"bucket-1:myindex:num_flush_queued":26792567,"bucket-1:myindex:num_items_flushed":26757471,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":347,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":1043095940,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"209 136409568922 3631697045911905548","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"26757472 150706522103 1014177387797794207","bucket-1:myindex:timings/storage_info":"1113 8326713 119938738473","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"208 73147375 28658938520933","bucket-1:myindex:timings/storage_meta_set":"209 73977965 28583623829871","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"209 8835702 412906087630","bucket-1:myindex:timings/storage_set":"53514942 1014336438819 -8806673509928517857","bucket-1:myindex:timings/storage_snapshot_close":"138 101183602 219338630067568","bucket-1:myindex:timings/storage_snapshot_create":"138 121704209 154726142551279","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":33904396,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"356 596952199 27084985436935305","bucket-1:ts_queue_size":803,"indexer_state":"Active","memory_quota":31457280000,"memory_used":23568906240,"memory_used_storage":18874363904,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"4225 756045560 339966328873294","uptime":"41m1.346695468s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:10:09.68-07:00 [Info] memstats {"Alloc":4376173888, "TotalAlloc":35248751664, "Sys":9854296392, "Lookups":8950, "Mallocs":1002256979,"Frees":893727094, "HeapAlloc":4376173888, "HeapSys":9382232064, "HeapIdle":4985135104, "HeapInuse":4397096960,"HeapReleased":4983062528, "HeapObjects":108529885,"GCSys":297445376, "LastGC":1460020119732575202,"PauseTotalNs":51078956, "PauseNs":[1852279], "NumGC":39} 2016-04-07T02:10:09.753-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26760000 QueuedCount 32567 2016-04-07T02:10:09.777-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33910000 2016-04-07T02:10:09.964-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:7176102812, Disk:20360749056, ExtraSnapshotData:0, Fragmentation:64.75521213751557%, mode : circular 2016-04-07T02:10:09.964-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:10:10-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33920000 2016-04-07T02:10:10.128-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26770000 QueuedCount 22567 2016-04-07T02:10:10.22-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33930000 2016-04-07T02:10:10.44-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26780000 QueuedCount 12567 2016-04-07T02:10:10.443-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33940000 2016-04-07T02:10:10.667-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33950000 2016-04-07T02:10:10.808-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26790000 QueuedCount 2567 2016-04-07T02:10:10.895-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33960000 2016-04-07T02:10:10.955-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26792567, 26792567, 0 committed:false 2016-04-07T02:10:10.955-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26792567, 26792567, 0 committed:false) SnapCreateDur 28.701µs SnapOpenDur 244.205µs 2016-04-07T02:10:10.959-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:10.964-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:10.968-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:10.972-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:10.977-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:11.124-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33970000 2016-04-07T02:10:11.319-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26800000 QueuedCount 44814 2016-04-07T02:10:11.352-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33980000 2016-04-07T02:10:11.58-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 33990000 2016-04-07T02:10:11.696-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26810000 QueuedCount 34814 2016-04-07T02:10:11.805-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34000000 2016-04-07T02:10:12.027-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34010000 2016-04-07T02:10:12.073-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26820000 QueuedCount 24814 2016-04-07T02:10:12.25-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34020000 2016-04-07T02:10:12.45-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26830000 QueuedCount 14814 2016-04-07T02:10:12.48-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34030000 2016-04-07T02:10:12.713-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34040000 2016-04-07T02:10:12.84-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26840000 QueuedCount 4814 2016-04-07T02:10:12.932-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34050000 2016-04-07T02:10:13-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26844814, 26844814, 0 committed:false 2016-04-07T02:10:13.001-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26844814, 26844814, 0 committed:false) SnapCreateDur 29.761µs SnapOpenDur 1.46722ms 2016-04-07T02:10:13.17-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34060000 2016-04-07T02:10:13.393-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34070000 2016-04-07T02:10:13.461-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26850000 QueuedCount 3341 2016-04-07T02:10:13.617-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34080000 2016-04-07T02:10:13.838-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34090000 2016-04-07T02:10:14.065-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34100000 2016-04-07T02:10:14.13-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 16.854µs CommitTime 565.160026ms TotalFlushTime 19m42.995792448s TotalCommitTime 2m16.974728948s 2016-04-07T02:10:14.13-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26853341, 26853341, 210 committed:true 2016-04-07T02:10:14.13-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26853341, 26853341, 210 committed:true) SnapCreateDur 574.226022ms SnapOpenDur 37.884µs 2016-04-07T02:10:14.135-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:14.14-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:14.144-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:14.149-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:14.154-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:14.288-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34110000 2016-04-07T02:10:14.517-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34120000 2016-04-07T02:10:14.539-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26860000 QueuedCount 45993 2016-04-07T02:10:14.744-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34130000 2016-04-07T02:10:14.911-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26870000 QueuedCount 35993 2016-04-07T02:10:14.969-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34140000 2016-04-07T02:10:15.194-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34150000 2016-04-07T02:10:15.281-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26880000 QueuedCount 25993 2016-04-07T02:10:15.423-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34160000 2016-04-07T02:10:15.611-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26890000 QueuedCount 15993 2016-04-07T02:10:15.644-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34170000 2016-04-07T02:10:15.863-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34180000 2016-04-07T02:10:15.985-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26900000 QueuedCount 5993 2016-04-07T02:10:16.085-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34190000 2016-04-07T02:10:16.229-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26905993, 26905993, 0 committed:false 2016-04-07T02:10:16.23-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26905993, 26905993, 0 committed:false) SnapCreateDur 39.727µs SnapOpenDur 1.284288ms 2016-04-07T02:10:16.235-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:16.239-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:16.243-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:16.247-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:16.251-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:16.312-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34200000 2016-04-07T02:10:16.491-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26910000 QueuedCount 48595 2016-04-07T02:10:16.529-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34210000 2016-04-07T02:10:16.748-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34220000 2016-04-07T02:10:16.863-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26920000 QueuedCount 38595 2016-04-07T02:10:16.962-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34230000 2016-04-07T02:10:17.185-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34240000 2016-04-07T02:10:17.24-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26930000 QueuedCount 28595 2016-04-07T02:10:17.409-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34250000 2016-04-07T02:10:17.636-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26940000 QueuedCount 18595 2016-04-07T02:10:17.643-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34260000 2016-04-07T02:10:17.869-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34270000 2016-04-07T02:10:18.04-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26950000 QueuedCount 8595 2016-04-07T02:10:18.095-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34280000 2016-04-07T02:10:18.335-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34290000 2016-04-07T02:10:18.386-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26958595, 26958595, 0 committed:false 2016-04-07T02:10:18.387-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26958595, 26958595, 0 committed:false) SnapCreateDur 29.917µs SnapOpenDur 670.574µs 2016-04-07T02:10:18.57-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34300000 2016-04-07T02:10:18.643-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26960000 QueuedCount 7388 2016-04-07T02:10:18.793-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34310000 2016-04-07T02:10:19.02-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34320000 2016-04-07T02:10:19.343-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34330000 2016-04-07T02:10:19.404-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.638µs CommitTime 502.694798ms TotalFlushTime 19m47.674709195s TotalCommitTime 2m17.477423746s 2016-04-07T02:10:19.404-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 26967388, 26967388, 211 committed:true 2016-04-07T02:10:19.404-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 26967388, 26967388, 211 committed:true) SnapCreateDur 512.251036ms SnapOpenDur 42.369µs 2016-04-07T02:10:19.408-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:19.412-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:19.416-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:19.42-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:19.424-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:19.588-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26970000 QueuedCount 49279 2016-04-07T02:10:19.59-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34340000 2016-04-07T02:10:19.788-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34350000 2016-04-07T02:10:19.951-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26980000 QueuedCount 39279 2016-04-07T02:10:20.018-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34360000 2016-04-07T02:10:20.239-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34370000 2016-04-07T02:10:20.324-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 26990000 QueuedCount 29279 2016-04-07T02:10:20.46-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34380000 2016-04-07T02:10:20.639-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27000000 QueuedCount 19279 2016-04-07T02:10:20.691-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34390000 2016-04-07T02:10:20.912-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34400000 2016-04-07T02:10:21.017-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27010000 QueuedCount 9279 2016-04-07T02:10:21.129-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34410000 2016-04-07T02:10:21.35-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34420000 2016-04-07T02:10:21.425-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27019279, 27019279, 0 committed:false 2016-04-07T02:10:21.426-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27019279, 27019279, 0 committed:false) SnapCreateDur 27.579µs SnapOpenDur 621.515µs 2016-04-07T02:10:21.43-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:21.434-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:21.438-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:21.442-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:21.442-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27020000 QueuedCount 34399 2016-04-07T02:10:21.446-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:21.578-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34430000 2016-04-07T02:10:21.802-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34440000 2016-04-07T02:10:21.926-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27030000 QueuedCount 41222 2016-04-07T02:10:22.025-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34450000 2016-04-07T02:10:22.247-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34460000 2016-04-07T02:10:22.306-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27040000 QueuedCount 31222 2016-04-07T02:10:22.468-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34470000 2016-04-07T02:10:22.689-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27050000 QueuedCount 21222 2016-04-07T02:10:22.691-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34480000 2016-04-07T02:10:22.915-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34490000 2016-04-07T02:10:23.082-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27060000 QueuedCount 11222 2016-04-07T02:10:23.14-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34500000 2016-04-07T02:10:23.36-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34510000 2016-04-07T02:10:23.478-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27070000 QueuedCount 1222 2016-04-07T02:10:23.492-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27071222, 27071222, 0 committed:false 2016-04-07T02:10:23.493-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27071222, 27071222, 0 committed:false) SnapCreateDur 32.683µs SnapOpenDur 1.264601ms 2016-04-07T02:10:23.588-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34520000 2016-04-07T02:10:23.808-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34530000 2016-04-07T02:10:24.038-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34540000 2016-04-07T02:10:24.261-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34550000 2016-04-07T02:10:24.487-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34560000 2016-04-07T02:10:24.542-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 33.075µs CommitTime 523.918342ms TotalFlushTime 19m52.199084075s TotalCommitTime 2m18.001342088s 2016-04-07T02:10:24.542-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27079732, 27079732, 212 committed:true 2016-04-07T02:10:24.542-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27079732, 27079732, 212 committed:true) SnapCreateDur 534.667981ms SnapOpenDur 72.254µs 2016-04-07T02:10:24.546-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:24.549-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27080000 QueuedCount 15884 2016-04-07T02:10:24.55-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:24.554-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:24.558-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:24.562-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:24.708-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34570000 2016-04-07T02:10:24.929-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34580000 2016-04-07T02:10:25.083-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27090000 QueuedCount 43506 2016-04-07T02:10:25.148-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34590000 2016-04-07T02:10:25.379-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34600000 2016-04-07T02:10:25.387-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27100000 QueuedCount 33506 2016-04-07T02:10:25.595-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34610000 2016-04-07T02:10:25.737-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27110000 QueuedCount 23506 2016-04-07T02:10:25.823-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34620000 2016-04-07T02:10:26.041-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34630000 2016-04-07T02:10:26.118-07:00 [Info] Indexer::ReadMemstats Time Taken 14.323267ms 2016-04-07T02:10:26.118-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27120000 QueuedCount 13506 2016-04-07T02:10:26.264-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34640000 2016-04-07T02:10:26.492-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34650000 2016-04-07T02:10:26.496-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27130000 QueuedCount 3506 2016-04-07T02:10:26.645-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27133506, 27133506, 0 committed:false 2016-04-07T02:10:26.645-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27133506, 27133506, 0 committed:false) SnapCreateDur 28.246µs SnapOpenDur 548.237µs 2016-04-07T02:10:26.649-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:26.652-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:26.656-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:26.66-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:26.663-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:26.714-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34660000 2016-04-07T02:10:26.934-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34670000 2016-04-07T02:10:27.003-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27140000 QueuedCount 44692 2016-04-07T02:10:27.159-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34680000 2016-04-07T02:10:27.388-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34690000 2016-04-07T02:10:27.389-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27150000 QueuedCount 34692 2016-04-07T02:10:27.608-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34700000 2016-04-07T02:10:27.785-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27160000 QueuedCount 24692 2016-04-07T02:10:27.831-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34710000 2016-04-07T02:10:28.061-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34720000 2016-04-07T02:10:28.186-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27170000 QueuedCount 14692 2016-04-07T02:10:28.283-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34730000 2016-04-07T02:10:28.507-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34740000 2016-04-07T02:10:28.527-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27180000 QueuedCount 4692 2016-04-07T02:10:28.735-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34750000 2016-04-07T02:10:28.758-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27184692, 27184692, 0 committed:false 2016-04-07T02:10:28.759-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27184692, 27184692, 0 committed:false) SnapCreateDur 26.164µs SnapOpenDur 417.017µs 2016-04-07T02:10:28.961-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34760000 2016-04-07T02:10:29.03-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27190000 QueuedCount 3605 2016-04-07T02:10:29.185-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34770000 2016-04-07T02:10:29.409-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34780000 2016-04-07T02:10:29.668-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34790000 2016-04-07T02:10:29.716-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.177µs CommitTime 523.006118ms TotalFlushTime 19m56.769560663s TotalCommitTime 2m18.524348206s 2016-04-07T02:10:29.716-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27193605, 27193605, 213 committed:true 2016-04-07T02:10:29.716-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27193605, 27193605, 213 committed:true) SnapCreateDur 532.987624ms SnapOpenDur 44.315µs 2016-04-07T02:10:29.72-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:29.724-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:29.727-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:29.73-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:29.734-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:29.9-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34800000 2016-04-07T02:10:30.115-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27200000 QueuedCount 46082 2016-04-07T02:10:30.134-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34810000 2016-04-07T02:10:30.357-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34820000 2016-04-07T02:10:30.472-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27210000 QueuedCount 36082 2016-04-07T02:10:30.573-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34830000 2016-04-07T02:10:30.777-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27220000 QueuedCount 26082 2016-04-07T02:10:30.798-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34840000 2016-04-07T02:10:31.025-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34850000 2016-04-07T02:10:31.147-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27230000 QueuedCount 16082 2016-04-07T02:10:31.246-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34860000 2016-04-07T02:10:31.469-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34870000 2016-04-07T02:10:31.524-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27240000 QueuedCount 6082 2016-04-07T02:10:31.695-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34880000 2016-04-07T02:10:31.768-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27246082, 27246082, 0 committed:false 2016-04-07T02:10:31.769-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27246082, 27246082, 0 committed:false) SnapCreateDur 28.891µs SnapOpenDur 1.16824ms 2016-04-07T02:10:31.773-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:31.777-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:31.781-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:31.784-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:31.788-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:31.918-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34890000 2016-04-07T02:10:32.024-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27250000 QueuedCount 48121 2016-04-07T02:10:32.15-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34900000 2016-04-07T02:10:32.379-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34910000 2016-04-07T02:10:32.409-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27260000 QueuedCount 38121 2016-04-07T02:10:32.604-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34920000 2016-04-07T02:10:32.789-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27270000 QueuedCount 28121 2016-04-07T02:10:32.824-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34930000 2016-04-07T02:10:33.051-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34940000 2016-04-07T02:10:33.177-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27280000 QueuedCount 18121 2016-04-07T02:10:33.272-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34950000 2016-04-07T02:10:33.493-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34960000 2016-04-07T02:10:33.568-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27290000 QueuedCount 8121 2016-04-07T02:10:33.712-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34970000 2016-04-07T02:10:33.89-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27298121, 27298121, 0 committed:false 2016-04-07T02:10:33.89-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27298121, 27298121, 0 committed:false) SnapCreateDur 28.386µs SnapOpenDur 126.443µs 2016-04-07T02:10:33.91-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27300000 QueuedCount 6666 2016-04-07T02:10:33.948-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34980000 2016-04-07T02:10:34.175-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 34990000 2016-04-07T02:10:34.403-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35000000 2016-04-07T02:10:34.632-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35010000 2016-04-07T02:10:34.855-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35020000 2016-04-07T02:10:34.911-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.878µs CommitTime 485.961724ms TotalFlushTime 20m1.395348s TotalCommitTime 2m19.01030993s 2016-04-07T02:10:34.911-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27306666, 27306666, 214 committed:true 2016-04-07T02:10:34.911-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27306666, 27306666, 214 committed:true) SnapCreateDur 496.882611ms SnapOpenDur 38.25µs 2016-04-07T02:10:34.915-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:34.918-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:34.923-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:34.926-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:34.93-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:35.079-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35030000 2016-04-07T02:10:35.175-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27310000 QueuedCount 50000 2016-04-07T02:10:35.306-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35040000 2016-04-07T02:10:35.528-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35050000 2016-04-07T02:10:35.54-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27320000 QueuedCount 40142 2016-04-07T02:10:35.744-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35060000 2016-04-07T02:10:35.922-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27330000 QueuedCount 30142 2016-04-07T02:10:35.975-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35070000 2016-04-07T02:10:36.2-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35080000 2016-04-07T02:10:36.327-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27340000 QueuedCount 20142 2016-04-07T02:10:36.425-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35090000 2016-04-07T02:10:36.647-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35100000 2016-04-07T02:10:36.719-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27350000 QueuedCount 10142 2016-04-07T02:10:36.875-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35110000 2016-04-07T02:10:37.098-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35120000 2016-04-07T02:10:37.11-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27360000 QueuedCount 142 2016-04-07T02:10:37.117-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27360142, 27360142, 0 committed:false 2016-04-07T02:10:37.117-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27360142, 27360142, 0 committed:false) SnapCreateDur 25.4µs SnapOpenDur 260.92µs 2016-04-07T02:10:37.121-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:37.125-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:37.128-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:37.132-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:37.135-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:37.321-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35130000 2016-04-07T02:10:37.506-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27370000 QueuedCount 43896 2016-04-07T02:10:37.555-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35140000 2016-04-07T02:10:37.772-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35150000 2016-04-07T02:10:37.892-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27380000 QueuedCount 33896 2016-04-07T02:10:38.002-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35160000 2016-04-07T02:10:38.23-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35170000 2016-04-07T02:10:38.29-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27390000 QueuedCount 23896 2016-04-07T02:10:38.457-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35180000 2016-04-07T02:10:38.679-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35190000 2016-04-07T02:10:38.699-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27400000 QueuedCount 13896 2016-04-07T02:10:38.909-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35200000 2016-04-07T02:10:39.121-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27410000 QueuedCount 3896 2016-04-07T02:10:39.135-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35210000 2016-04-07T02:10:39.299-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27413896, 27413896, 0 committed:false 2016-04-07T02:10:39.299-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27413896, 27413896, 0 committed:false) SnapCreateDur 26.97µs SnapOpenDur 563.376µs 2016-04-07T02:10:39.364-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35220000 2016-04-07T02:10:39.59-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35230000 2016-04-07T02:10:39.699-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27420000 QueuedCount 2467 2016-04-07T02:10:39.819-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35240000 2016-04-07T02:10:39.965-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:7352160634, Disk:20360749056, ExtraSnapshotData:0, Fragmentation:63.8905198734158%, mode : circular 2016-04-07T02:10:39.965-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:10:40.051-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35250000 2016-04-07T02:10:40.273-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35260000 2016-04-07T02:10:40.291-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.18µs CommitTime 497.936851ms TotalFlushTime 20m6.186929123s TotalCommitTime 2m19.508246781s 2016-04-07T02:10:40.291-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27422467, 27422467, 215 committed:true 2016-04-07T02:10:40.291-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27422467, 27422467, 215 committed:true) SnapCreateDur 507.460742ms SnapOpenDur 44µs 2016-04-07T02:10:40.295-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:40.298-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:40.302-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:40.305-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:40.309-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:40.924-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35270000 2016-04-07T02:10:41.114-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35280000 2016-04-07T02:10:41.185-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27430000 QueuedCount 45721 2016-04-07T02:10:41.344-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35290000 2016-04-07T02:10:41.799-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35300000 2016-04-07T02:10:41.813-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35310000 2016-04-07T02:10:41.828-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35320000 2016-04-07T02:10:41.843-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35330000 2016-04-07T02:10:41.948-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27440000 QueuedCount 35721 2016-04-07T02:10:42.059-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35340000 2016-04-07T02:10:42.275-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35350000 2016-04-07T02:10:42.312-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27450000 QueuedCount 25721 2016-04-07T02:10:42.493-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35360000 2016-04-07T02:10:42.678-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27460000 QueuedCount 15721 2016-04-07T02:10:42.718-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35370000 2016-04-07T02:10:42.938-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35380000 2016-04-07T02:10:43.047-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27470000 QueuedCount 5721 2016-04-07T02:10:43.155-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35390000 2016-04-07T02:10:43.219-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27475721, 27475721, 0 committed:false 2016-04-07T02:10:43.221-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27475721, 27475721, 0 committed:false) SnapCreateDur 29.721µs SnapOpenDur 1.986273ms 2016-04-07T02:10:43.226-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:43.23-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:43.235-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:43.239-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:43.244-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:43.381-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35400000 2016-04-07T02:10:43.485-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27480000 QueuedCount 49121 2016-04-07T02:10:43.604-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35410000 2016-04-07T02:10:43.796-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27490000 QueuedCount 39121 2016-04-07T02:10:43.824-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35420000 2016-04-07T02:10:44.047-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35430000 2016-04-07T02:10:44.173-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27500000 QueuedCount 29121 2016-04-07T02:10:44.273-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35440000 2016-04-07T02:10:44.494-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35450000 2016-04-07T02:10:44.562-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27510000 QueuedCount 19121 2016-04-07T02:10:44.722-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35460000 2016-04-07T02:10:44.938-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27520000 QueuedCount 9121 2016-04-07T02:10:44.948-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35470000 2016-04-07T02:10:45.17-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35480000 2016-04-07T02:10:45.315-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27529121, 27529121, 0 committed:false 2016-04-07T02:10:45.315-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27529121, 27529121, 0 committed:false) SnapCreateDur 29.602µs SnapOpenDur 154.877µs 2016-04-07T02:10:45.326-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27530000 QueuedCount 7615 2016-04-07T02:10:45.399-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35490000 2016-04-07T02:10:45.625-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35500000 2016-04-07T02:10:45.853-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35510000 2016-04-07T02:10:46.09-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35520000 2016-04-07T02:10:46.287-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.625µs CommitTime 487.790647ms TotalFlushTime 20m11.60500957s TotalCommitTime 2m19.996037428s 2016-04-07T02:10:46.287-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27537615, 27537615, 216 committed:true 2016-04-07T02:10:46.287-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27537615, 27537615, 216 committed:true) SnapCreateDur 497.471025ms SnapOpenDur 44.04µs 2016-04-07T02:10:46.292-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:46.298-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:46.303-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:46.309-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:46.313-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:46.329-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35530000 2016-04-07T02:10:46.435-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27540000 QueuedCount 50000 2016-04-07T02:10:46.554-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35540000 2016-04-07T02:10:46.735-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27550000 QueuedCount 40722 2016-04-07T02:10:46.782-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35550000 2016-04-07T02:10:47.001-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27560000 QueuedCount 30722 2016-04-07T02:10:47.009-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35560000 2016-04-07T02:10:47.236-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35570000 2016-04-07T02:10:47.322-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27570000 QueuedCount 20722 2016-04-07T02:10:47.462-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35580000 2016-04-07T02:10:47.651-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27580000 QueuedCount 10722 2016-04-07T02:10:47.698-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35590000 2016-04-07T02:10:47.922-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35600000 2016-04-07T02:10:47.999-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27590000 QueuedCount 722 2016-04-07T02:10:48.013-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27590722, 27590722, 0 committed:false 2016-04-07T02:10:48.015-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27590722, 27590722, 0 committed:false) SnapCreateDur 33.225µs SnapOpenDur 2.249892ms 2016-04-07T02:10:48.02-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:48.025-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:48.03-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:48.035-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:48.04-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:48.15-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35610000 2016-04-07T02:10:48.378-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35620000 2016-04-07T02:10:48.425-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27600000 QueuedCount 42884 2016-04-07T02:10:48.618-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35630000 2016-04-07T02:10:48.803-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27610000 QueuedCount 32884 2016-04-07T02:10:48.851-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35640000 2016-04-07T02:10:49.083-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35650000 2016-04-07T02:10:49.181-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27620000 QueuedCount 22884 2016-04-07T02:10:49.32-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35660000 2016-04-07T02:10:49.547-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35670000 2016-04-07T02:10:49.56-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27630000 QueuedCount 12884 2016-04-07T02:10:49.779-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35680000 2016-04-07T02:10:49.887-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27640000 QueuedCount 2884 2016-04-07T02:10:50.014-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35690000 2016-04-07T02:10:50.034-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27642884, 27642884, 0 committed:false 2016-04-07T02:10:50.035-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27642884, 27642884, 0 committed:false) SnapCreateDur 40.668µs SnapOpenDur 657.472µs 2016-04-07T02:10:50.04-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:50.045-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:50.049-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:50.054-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:50.058-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:50.24-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35700000 2016-04-07T02:10:50.467-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35710000 2016-04-07T02:10:50.502-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27650000 QueuedCount 46750 2016-04-07T02:10:50.707-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35720000 2016-04-07T02:10:50.9-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27660000 QueuedCount 36750 2016-04-07T02:10:50.927-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35730000 2016-04-07T02:10:51.145-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35740000 2016-04-07T02:10:51.295-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27670000 QueuedCount 26750 2016-04-07T02:10:51.372-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35750000 2016-04-07T02:10:51.602-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35760000 2016-04-07T02:10:51.698-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27680000 QueuedCount 16750 2016-04-07T02:10:51.823-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35770000 2016-04-07T02:10:52.045-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35780000 2016-04-07T02:10:52.099-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27690000 QueuedCount 6750 2016-04-07T02:10:52.272-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35790000 2016-04-07T02:10:52.356-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27696750, 27696750, 0 committed:false 2016-04-07T02:10:52.357-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27696750, 27696750, 0 committed:false) SnapCreateDur 26.622µs SnapOpenDur 1.217984ms 2016-04-07T02:10:52.494-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35800000 2016-04-07T02:10:52.682-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27700000 QueuedCount 5765 2016-04-07T02:10:52.73-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35810000 2016-04-07T02:10:52.957-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35820000 2016-04-07T02:10:53.188-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35830000 2016-04-07T02:10:53.413-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35840000 2016-04-07T02:10:53.53-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 22.904µs CommitTime 578.242911ms TotalFlushTime 20m18.152088953s TotalCommitTime 2m20.574280339s 2016-04-07T02:10:53.53-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27705765, 27705765, 217 committed:true 2016-04-07T02:10:53.53-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27705765, 27705765, 217 committed:true) SnapCreateDur 588.683399ms SnapOpenDur 34.811µs 2016-04-07T02:10:53.535-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:53.54-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:53.544-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:53.548-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:53.553-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:53.643-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35850000 2016-04-07T02:10:53.868-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35860000 2016-04-07T02:10:53.881-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27710000 QueuedCount 49664 2016-04-07T02:10:54.089-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35870000 2016-04-07T02:10:54.179-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27720000 QueuedCount 39664 2016-04-07T02:10:54.315-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35880000 2016-04-07T02:10:54.546-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27730000 QueuedCount 29664 2016-04-07T02:10:54.548-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35890000 2016-04-07T02:10:54.768-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35900000 2016-04-07T02:10:54.909-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27740000 QueuedCount 19664 2016-04-07T02:10:54.993-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35910000 2016-04-07T02:10:55.218-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35920000 2016-04-07T02:10:55.279-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27750000 QueuedCount 9664 2016-04-07T02:10:55.439-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35930000 2016-04-07T02:10:55.661-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27759664, 27759664, 0 committed:false 2016-04-07T02:10:55.662-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27759664, 27759664, 0 committed:false) SnapCreateDur 62.165µs SnapOpenDur 772.74µs 2016-04-07T02:10:55.666-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:55.666-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35940000 2016-04-07T02:10:55.67-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27760000 QueuedCount 16409 2016-04-07T02:10:55.67-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:55.675-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:55.679-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:55.683-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:55.884-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35950000 2016-04-07T02:10:56.113-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35960000 2016-04-07T02:10:56.156-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27770000 QueuedCount 43810 2016-04-07T02:10:56.338-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35970000 2016-04-07T02:10:56.538-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27780000 QueuedCount 33810 2016-04-07T02:10:56.558-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35980000 2016-04-07T02:10:56.781-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 35990000 2016-04-07T02:10:56.914-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27790000 QueuedCount 23810 2016-04-07T02:10:57.008-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36000000 2016-04-07T02:10:57.24-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36010000 2016-04-07T02:10:57.296-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27800000 QueuedCount 13810 2016-04-07T02:10:57.458-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36020000 2016-04-07T02:10:57.633-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27810000 QueuedCount 3810 2016-04-07T02:10:57.678-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36030000 2016-04-07T02:10:57.805-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27813810, 27813810, 0 committed:false 2016-04-07T02:10:57.807-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27813810, 27813810, 0 committed:false) SnapCreateDur 26.116µs SnapOpenDur 1.320407ms 2016-04-07T02:10:57.903-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36040000 2016-04-07T02:10:58.13-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36050000 2016-04-07T02:10:58.297-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27820000 QueuedCount 2140 2016-04-07T02:10:58.352-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36060000 2016-04-07T02:10:58.585-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36070000 2016-04-07T02:10:58.809-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36080000 2016-04-07T02:10:58.987-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 23.331µs CommitTime 586.677106ms TotalFlushTime 20m22.922539969s TotalCommitTime 2m21.160957445s 2016-04-07T02:10:58.987-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27822140, 27822140, 218 committed:true 2016-04-07T02:10:58.987-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27822140, 27822140, 218 committed:true) SnapCreateDur 596.182876ms SnapOpenDur 49.955µs 2016-04-07T02:10:58.991-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:58.995-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:58.999-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:59.003-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:59.007-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:10:59.148-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36090000 2016-04-07T02:10:59.382-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36100000 2016-04-07T02:10:59.39-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27830000 QueuedCount 46160 2016-04-07T02:10:59.594-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36110000 2016-04-07T02:10:59.718-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27840000 QueuedCount 36160 2016-04-07T02:10:59.932-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36120000 2016-04-07T02:11:00.079-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27850000 QueuedCount 26160 2016-04-07T02:11:00.159-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36130000 2016-04-07T02:11:00.395-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36140000 2016-04-07T02:11:00.443-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27860000 QueuedCount 16160 2016-04-07T02:11:00.618-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36150000 2016-04-07T02:11:00.814-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27870000 QueuedCount 6160 2016-04-07T02:11:00.844-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36160000 2016-04-07T02:11:01.059-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27876160, 27876160, 0 committed:false 2016-04-07T02:11:01.06-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27876160, 27876160, 0 committed:false) SnapCreateDur 27.053µs SnapOpenDur 880.691µs 2016-04-07T02:11:01.064-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:01.068-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:01.072-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:01.076-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:01.08-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36170000 2016-04-07T02:11:01.08-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:01.273-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27880000 QueuedCount 49597 2016-04-07T02:11:01.306-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36180000 2016-04-07T02:11:01.528-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36190000 2016-04-07T02:11:01.674-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27890000 QueuedCount 39597 2016-04-07T02:11:01.751-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36200000 2016-04-07T02:11:01.979-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36210000 2016-04-07T02:11:02.075-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27900000 QueuedCount 29597 2016-04-07T02:11:02.204-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36220000 2016-04-07T02:11:02.422-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27910000 QueuedCount 19597 2016-04-07T02:11:02.424-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36230000 2016-04-07T02:11:02.657-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36240000 2016-04-07T02:11:02.833-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27920000 QueuedCount 9597 2016-04-07T02:11:02.879-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36250000 2016-04-07T02:11:03.111-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36260000 2016-04-07T02:11:03.269-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27929597, 27929597, 0 committed:false 2016-04-07T02:11:03.27-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27929597, 27929597, 0 committed:false) SnapCreateDur 27.855µs SnapOpenDur 899.789µs 2016-04-07T02:11:03.276-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27930000 QueuedCount 8115 2016-04-07T02:11:03.334-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36270000 2016-04-07T02:11:03.559-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36280000 2016-04-07T02:11:03.793-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36290000 2016-04-07T02:11:04.016-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36300000 2016-04-07T02:11:04.197-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 33.669µs CommitTime 484.175487ms TotalFlushTime 20m27.558026941s TotalCommitTime 2m21.645132932s 2016-04-07T02:11:04.197-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27938115, 27938115, 219 committed:true 2016-04-07T02:11:04.197-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27938115, 27938115, 219 committed:true) SnapCreateDur 492.984022ms SnapOpenDur 55.162µs 2016-04-07T02:11:04.202-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:04.206-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:04.209-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:04.214-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:04.218-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:04.229-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27940000 QueuedCount 50000 2016-04-07T02:11:04.242-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36310000 2016-04-07T02:11:04.463-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36320000 2016-04-07T02:11:04.694-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36330000 2016-04-07T02:11:04.709-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27950000 QueuedCount 42004 2016-04-07T02:11:04.929-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36340000 2016-04-07T02:11:05.058-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27960000 QueuedCount 32004 2016-04-07T02:11:05.16-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36350000 2016-04-07T02:11:05.383-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36360000 2016-04-07T02:11:05.409-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27970000 QueuedCount 22004 2016-04-07T02:11:05.608-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36370000 2016-04-07T02:11:05.772-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27980000 QueuedCount 12004 2016-04-07T02:11:05.838-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36380000 2016-04-07T02:11:06.059-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36390000 2016-04-07T02:11:06.164-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 27990000 QueuedCount 2004 2016-04-07T02:11:06.25-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 27992004, 27992004, 0 committed:false 2016-04-07T02:11:06.25-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 27992004, 27992004, 0 committed:false) SnapCreateDur 44.584µs SnapOpenDur 545.112µs 2016-04-07T02:11:06.254-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:06.256-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:06.26-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:06.264-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:06.266-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:06.284-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36400000 2016-04-07T02:11:06.51-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36410000 2016-04-07T02:11:06.605-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28000000 QueuedCount 35811 2016-04-07T02:11:06.733-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36420000 2016-04-07T02:11:06.96-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36430000 2016-04-07T02:11:06.991-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28010000 QueuedCount 25811 2016-04-07T02:11:07.183-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36440000 2016-04-07T02:11:07.345-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28020000 QueuedCount 15811 2016-04-07T02:11:07.412-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36450000 2016-04-07T02:11:07.634-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36460000 2016-04-07T02:11:07.758-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28030000 QueuedCount 5811 2016-04-07T02:11:07.858-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36470000 2016-04-07T02:11:08-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28035811, 28035811, 0 committed:false 2016-04-07T02:11:08.001-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28035811, 28035811, 0 committed:false) SnapCreateDur 56.995µs SnapOpenDur 1.258314ms 2016-04-07T02:11:08.005-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:08.009-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:08.012-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:08.016-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:08.019-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:08.081-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36480000 2016-04-07T02:11:08.28-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28040000 QueuedCount 49686 2016-04-07T02:11:08.303-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36490000 2016-04-07T02:11:08.53-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36500000 2016-04-07T02:11:08.675-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28050000 QueuedCount 39686 2016-04-07T02:11:08.751-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36510000 2016-04-07T02:11:09.057-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28060000 QueuedCount 29686 2016-04-07T02:11:09.13-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36520000 2016-04-07T02:11:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:11:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:11:09.374-07:00 [Info] client load stats {} 2016-04-07T02:11:09.442-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28070000 QueuedCount 19686 2016-04-07T02:11:09.449-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36530000 2016-04-07T02:11:09.655-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":8449328,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":1757451529,"bucket-1:myindex:avg_ts_items_count":42347,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":7510894653,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":20360749056,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":14322,"bucket-1:myindex:frag_percent":63,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":2942100000,"bucket-1:myindex:items_count":27938115,"bucket-1:myindex:num_commits":219,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":28075364,"bucket-1:myindex:num_docs_pending":1181,"bucket-1:myindex:num_docs_processed":28028035,"bucket-1:myindex:num_docs_queued":8418073,"bucket-1:myindex:num_flush_queued":28089686,"bucket-1:myindex:num_items_flushed":28075364,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":380,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":1751176625,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"219 141645132932 6386851202501449228","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"28075365 156599383294 1163483912720028662","bucket-1:myindex:timings/storage_info":"1199 8883151 127738527125","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"218 77225525 30650976183967","bucket-1:myindex:timings/storage_meta_set":"219 77624143 30125493076521","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"219 9140416 423059825492","bucket-1:myindex:timings/storage_set":"56150728 1061473996543 -5388131205714514057","bucket-1:myindex:timings/storage_snapshot_close":"160 116437571 251473043019907","bucket-1:myindex:timings/storage_snapshot_create":"161 141037969 177874922756399","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":36539014,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"368 634049295 27772132624842177","bucket-1:ts_queue_size":954,"indexer_state":"Active","memory_quota":31457280000,"memory_used":24369440768,"memory_used_storage":18874368000,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"4309 774811972 344617159011406","uptime":"42m1.347920282s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:11:09.677-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36540000 2016-04-07T02:11:09.68-07:00 [Info] memstats {"Alloc":5173999568, "TotalAlloc":38817991472, "Sys":9854296392, "Lookups":9030, "Mallocs":1105928867,"Frees":978382207, "HeapAlloc":5173999568, "HeapSys":9382264832, "HeapIdle":4184637440, "HeapInuse":5197627392,"HeapReleased":4182458368, "HeapObjects":127546660,"GCSys":297445376, "LastGC":1460020174499664624,"PauseTotalNs":52850105, "PauseNs":[1771149], "NumGC":40} 2016-04-07T02:11:09.847-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28080000 QueuedCount 9686 2016-04-07T02:11:09.93-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36550000 2016-04-07T02:11:09.965-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:7527768889, Disk:20360749056, ExtraSnapshotData:0, Fragmentation:63.02803561747311%, mode : circular 2016-04-07T02:11:09.965-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:11:10.187-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28089686, 28089686, 0 committed:false 2016-04-07T02:11:10.187-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36560000 2016-04-07T02:11:10.189-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28089686, 28089686, 0 committed:false) SnapCreateDur 28.497µs SnapOpenDur 2.372903ms 2016-04-07T02:11:10.41-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36570000 2016-04-07T02:11:10.504-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28090000 QueuedCount 8685 2016-04-07T02:11:10.633-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36580000 2016-04-07T02:11:10.854-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36590000 2016-04-07T02:11:11.073-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36600000 2016-04-07T02:11:11.299-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36610000 2016-04-07T02:11:11.468-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.289µs CommitTime 615.872089ms TotalFlushTime 20m34.089065426s TotalCommitTime 2m22.261005021s 2016-04-07T02:11:11.468-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28098685, 28098685, 220 committed:true 2016-04-07T02:11:11.468-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28098685, 28098685, 220 committed:true) SnapCreateDur 625.514746ms SnapOpenDur 39.933µs 2016-04-07T02:11:11.472-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:11.476-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:11.479-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:11.483-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:11.486-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:11.49-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28100000 QueuedCount 50000 2016-04-07T02:11:11.522-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36620000 2016-04-07T02:11:11.752-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36630000 2016-04-07T02:11:11.906-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28110000 QueuedCount 42764 2016-04-07T02:11:11.982-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36640000 2016-04-07T02:11:12.205-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36650000 2016-04-07T02:11:12.253-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28120000 QueuedCount 32764 2016-04-07T02:11:12.438-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36660000 2016-04-07T02:11:12.607-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28130000 QueuedCount 22764 2016-04-07T02:11:12.653-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36670000 2016-04-07T02:11:12.874-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36680000 2016-04-07T02:11:12.976-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28140000 QueuedCount 12764 2016-04-07T02:11:13.097-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36690000 2016-04-07T02:11:13.313-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36700000 2016-04-07T02:11:13.359-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28150000 QueuedCount 2764 2016-04-07T02:11:13.447-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28152764, 28152764, 0 committed:false 2016-04-07T02:11:13.448-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28152764, 28152764, 0 committed:false) SnapCreateDur 37.686µs SnapOpenDur 764.051µs 2016-04-07T02:11:13.452-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:13.456-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:13.46-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:13.464-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:13.468-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:13.528-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36710000 2016-04-07T02:11:13.744-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36720000 2016-04-07T02:11:13.753-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28160000 QueuedCount 45899 2016-04-07T02:11:13.97-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36730000 2016-04-07T02:11:14.146-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28170000 QueuedCount 35899 2016-04-07T02:11:14.193-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36740000 2016-04-07T02:11:14.402-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36750000 2016-04-07T02:11:14.527-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28180000 QueuedCount 25899 2016-04-07T02:11:14.622-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36760000 2016-04-07T02:11:14.842-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36770000 2016-04-07T02:11:14.907-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28190000 QueuedCount 15899 2016-04-07T02:11:15.063-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36780000 2016-04-07T02:11:15.295-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36790000 2016-04-07T02:11:15.302-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28200000 QueuedCount 5899 2016-04-07T02:11:15.52-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36800000 2016-04-07T02:11:15.554-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28205899, 28205899, 0 committed:false 2016-04-07T02:11:15.554-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28205899, 28205899, 0 committed:false) SnapCreateDur 31.333µs SnapOpenDur 676.13µs 2016-04-07T02:11:15.753-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36810000 2016-04-07T02:11:15.817-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28210000 QueuedCount 4911 2016-04-07T02:11:15.98-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36820000 2016-04-07T02:11:16.2-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36830000 2016-04-07T02:11:16.422-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36840000 2016-04-07T02:11:16.545-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.417µs CommitTime 547.236173ms TotalFlushTime 20m38.536282594s TotalCommitTime 2m22.808241194s 2016-04-07T02:11:16.545-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28214911, 28214911, 221 committed:true 2016-04-07T02:11:16.545-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28214911, 28214911, 221 committed:true) SnapCreateDur 556.985931ms SnapOpenDur 43.114µs 2016-04-07T02:11:16.55-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:16.554-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:16.558-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:16.562-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:16.566-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:16.645-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36850000 2016-04-07T02:11:16.763-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28220000 QueuedCount 48823 2016-04-07T02:11:16.87-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36860000 2016-04-07T02:11:17.093-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36870000 2016-04-07T02:11:17.108-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28230000 QueuedCount 38823 2016-04-07T02:11:17.317-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36880000 2016-04-07T02:11:17.476-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28240000 QueuedCount 28823 2016-04-07T02:11:17.543-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36890000 2016-04-07T02:11:17.769-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36900000 2016-04-07T02:11:17.858-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28250000 QueuedCount 18823 2016-04-07T02:11:17.989-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36910000 2016-04-07T02:11:18.214-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36920000 2016-04-07T02:11:18.238-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28260000 QueuedCount 8823 2016-04-07T02:11:18.438-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36930000 2016-04-07T02:11:18.554-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28268823, 28268823, 0 committed:false 2016-04-07T02:11:18.554-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28268823, 28268823, 0 committed:false) SnapCreateDur 26.815µs SnapOpenDur 517.559µs 2016-04-07T02:11:18.559-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:18.563-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:18.568-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:18.572-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:18.576-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:18.579-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28270000 QueuedCount 49073 2016-04-07T02:11:18.658-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36940000 2016-04-07T02:11:18.887-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36950000 2016-04-07T02:11:19.054-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28280000 QueuedCount 42314 2016-04-07T02:11:19.115-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36960000 2016-04-07T02:11:19.34-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36970000 2016-04-07T02:11:19.437-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28290000 QueuedCount 32314 2016-04-07T02:11:19.572-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36980000 2016-04-07T02:11:19.813-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 36990000 2016-04-07T02:11:19.832-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28300000 QueuedCount 22314 2016-04-07T02:11:20.045-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37000000 2016-04-07T02:11:20.226-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28310000 QueuedCount 12314 2016-04-07T02:11:20.262-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37010000 2016-04-07T02:11:20.489-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37020000 2016-04-07T02:11:20.64-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28320000 QueuedCount 2314 2016-04-07T02:11:20.714-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37030000 2016-04-07T02:11:20.738-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28322314, 28322314, 0 committed:false 2016-04-07T02:11:20.739-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28322314, 28322314, 0 committed:false) SnapCreateDur 31.338µs SnapOpenDur 841.561µs 2016-04-07T02:11:20.944-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37040000 2016-04-07T02:11:21.179-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37050000 2016-04-07T02:11:21.256-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28330000 QueuedCount 1062 2016-04-07T02:11:21.403-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37060000 2016-04-07T02:11:21.625-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37070000 2016-04-07T02:11:21.797-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.511µs CommitTime 514.12491ms TotalFlushTime 20m43.181834265s TotalCommitTime 2m23.322366104s 2016-04-07T02:11:21.797-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28331062, 28331062, 222 committed:true 2016-04-07T02:11:21.797-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28331062, 28331062, 222 committed:true) SnapCreateDur 523.341062ms SnapOpenDur 42.609µs 2016-04-07T02:11:21.802-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:21.806-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:21.81-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:21.815-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:21.819-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:21.86-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37080000 2016-04-07T02:11:22.083-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37090000 2016-04-07T02:11:22.299-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28340000 QueuedCount 43725 2016-04-07T02:11:22.302-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37100000 2016-04-07T02:11:22.521-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37110000 2016-04-07T02:11:22.644-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28350000 QueuedCount 33725 2016-04-07T02:11:22.749-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37120000 2016-04-07T02:11:22.973-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37130000 2016-04-07T02:11:22.996-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28360000 QueuedCount 23725 2016-04-07T02:11:23.194-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37140000 2016-04-07T02:11:23.356-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28370000 QueuedCount 13725 2016-04-07T02:11:23.417-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37150000 2016-04-07T02:11:23.645-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37160000 2016-04-07T02:11:23.674-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28380000 QueuedCount 3725 2016-04-07T02:11:23.829-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28383725, 28383725, 0 committed:false 2016-04-07T02:11:23.83-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28383725, 28383725, 0 committed:false) SnapCreateDur 29.17µs SnapOpenDur 1.502036ms 2016-04-07T02:11:23.835-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:23.839-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:23.843-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:23.846-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:23.85-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:23.868-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37170000 2016-04-07T02:11:24.093-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37180000 2016-04-07T02:11:24.118-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28390000 QueuedCount 46854 2016-04-07T02:11:24.318-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37190000 2016-04-07T02:11:24.482-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28400000 QueuedCount 36854 2016-04-07T02:11:24.542-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37200000 2016-04-07T02:11:24.767-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37210000 2016-04-07T02:11:24.87-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28410000 QueuedCount 26854 2016-04-07T02:11:24.992-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37220000 2016-04-07T02:11:25.205-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37230000 2016-04-07T02:11:25.245-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28420000 QueuedCount 16854 2016-04-07T02:11:25.429-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37240000 2016-04-07T02:11:25.63-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28430000 QueuedCount 6854 2016-04-07T02:11:25.652-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37250000 2016-04-07T02:11:25.873-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37260000 2016-04-07T02:11:25.884-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28436854, 28436854, 0 committed:false 2016-04-07T02:11:25.885-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28436854, 28436854, 0 committed:false) SnapCreateDur 41.669µs SnapOpenDur 1.121009ms 2016-04-07T02:11:26.089-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37270000 2016-04-07T02:11:26.285-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28440000 QueuedCount 5332 2016-04-07T02:11:26.311-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37280000 2016-04-07T02:11:26.528-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37290000 2016-04-07T02:11:26.748-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37300000 2016-04-07T02:11:26.97-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37310000 2016-04-07T02:11:27.038-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 16.79µs CommitTime 569.516183ms TotalFlushTime 20m47.768654969s TotalCommitTime 2m23.891882287s 2016-04-07T02:11:27.039-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28445332, 28445332, 223 committed:true 2016-04-07T02:11:27.04-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28445332, 28445332, 223 committed:true) SnapCreateDur 580.193173ms SnapOpenDur 68.64µs 2016-04-07T02:11:27.044-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:27.048-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:27.052-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:27.056-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:27.058-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:27.193-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37320000 2016-04-07T02:11:27.246-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28450000 QueuedCount 44697 2016-04-07T02:11:27.418-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37330000 2016-04-07T02:11:27.493-07:00 [Info] Indexer::ReadMemstats Time Taken 15.066801ms 2016-04-07T02:11:27.59-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28460000 QueuedCount 34697 2016-04-07T02:11:27.654-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37340000 2016-04-07T02:11:27.884-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37350000 2016-04-07T02:11:27.956-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28470000 QueuedCount 24697 2016-04-07T02:11:28.109-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37360000 2016-04-07T02:11:28.263-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28480000 QueuedCount 14697 2016-04-07T02:11:28.334-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37370000 2016-04-07T02:11:28.555-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37380000 2016-04-07T02:11:28.624-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28490000 QueuedCount 4697 2016-04-07T02:11:28.778-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37390000 2016-04-07T02:11:28.843-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28494697, 28494697, 0 committed:false 2016-04-07T02:11:28.843-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28494697, 28494697, 0 committed:false) SnapCreateDur 30.243µs SnapOpenDur 271.883µs 2016-04-07T02:11:28.847-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:28.851-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:28.855-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:28.859-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:28.863-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:29.003-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37400000 2016-04-07T02:11:29.062-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28500000 QueuedCount 47496 2016-04-07T02:11:29.234-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37410000 2016-04-07T02:11:29.418-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28510000 QueuedCount 37496 2016-04-07T02:11:29.459-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37420000 2016-04-07T02:11:29.701-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37430000 2016-04-07T02:11:29.794-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28520000 QueuedCount 27496 2016-04-07T02:11:29.95-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37440000 2016-04-07T02:11:30.173-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28530000 QueuedCount 17496 2016-04-07T02:11:30.19-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37450000 2016-04-07T02:11:30.404-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37460000 2016-04-07T02:11:30.547-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28540000 QueuedCount 7496 2016-04-07T02:11:30.634-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37470000 2016-04-07T02:11:30.799-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28547496, 28547496, 0 committed:false 2016-04-07T02:11:30.8-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28547496, 28547496, 0 committed:false) SnapCreateDur 27.519µs SnapOpenDur 1.674886ms 2016-04-07T02:11:30.804-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:30.808-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:30.812-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:30.816-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:30.82-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:30.859-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37480000 2016-04-07T02:11:31.046-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28550000 QueuedCount 49935 2016-04-07T02:11:31.082-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37490000 2016-04-07T02:11:31.313-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37500000 2016-04-07T02:11:31.381-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28560000 QueuedCount 39935 2016-04-07T02:11:31.533-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37510000 2016-04-07T02:11:31.755-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37520000 2016-04-07T02:11:31.78-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28570000 QueuedCount 29935 2016-04-07T02:11:31.982-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37530000 2016-04-07T02:11:32.198-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28580000 QueuedCount 19935 2016-04-07T02:11:32.21-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37540000 2016-04-07T02:11:32.434-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37550000 2016-04-07T02:11:32.603-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28590000 QueuedCount 9935 2016-04-07T02:11:32.66-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37560000 2016-04-07T02:11:32.893-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37570000 2016-04-07T02:11:33.005-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28599935, 28599935, 0 committed:false 2016-04-07T02:11:33.006-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28599935, 28599935, 0 committed:false) SnapCreateDur 30.878µs SnapOpenDur 773.356µs 2016-04-07T02:11:33.008-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28600000 QueuedCount 3093 2016-04-07T02:11:33.114-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37580000 2016-04-07T02:11:33.339-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37590000 2016-04-07T02:11:33.565-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37600000 2016-04-07T02:11:33.795-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37610000 2016-04-07T02:11:34.028-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37620000 2016-04-07T02:11:34.172-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.77µs CommitTime 602.374453ms TotalFlushTime 20m54.175996138s TotalCommitTime 2m24.49425674s 2016-04-07T02:11:34.172-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28608919, 28608919, 224 committed:true 2016-04-07T02:11:34.172-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28608919, 28608919, 224 committed:true) SnapCreateDur 611.828857ms SnapOpenDur 32.754µs 2016-04-07T02:11:34.176-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:34.18-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:34.185-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:34.189-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:34.193-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:34.194-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28610000 QueuedCount 44746 2016-04-07T02:11:34.259-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37630000 2016-04-07T02:11:34.483-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37640000 2016-04-07T02:11:34.669-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28620000 QueuedCount 42119 2016-04-07T02:11:34.701-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37650000 2016-04-07T02:11:34.939-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37660000 2016-04-07T02:11:35.037-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28630000 QueuedCount 32119 2016-04-07T02:11:35.164-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37670000 2016-04-07T02:11:35.389-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37680000 2016-04-07T02:11:35.409-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28640000 QueuedCount 22119 2016-04-07T02:11:35.618-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37690000 2016-04-07T02:11:35.786-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28650000 QueuedCount 12119 2016-04-07T02:11:35.849-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37700000 2016-04-07T02:11:36.072-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37710000 2016-04-07T02:11:36.126-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28660000 QueuedCount 2119 2016-04-07T02:11:36.218-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28662119, 28662119, 0 committed:false 2016-04-07T02:11:36.22-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28662119, 28662119, 0 committed:false) SnapCreateDur 29.659µs SnapOpenDur 2.112549ms 2016-04-07T02:11:36.225-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:36.228-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:36.232-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:36.235-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:36.239-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:36.304-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37720000 2016-04-07T02:11:36.532-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37730000 2016-04-07T02:11:36.648-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28670000 QueuedCount 45569 2016-04-07T02:11:36.759-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37740000 2016-04-07T02:11:36.978-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37750000 2016-04-07T02:11:37.057-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28680000 QueuedCount 35569 2016-04-07T02:11:37.204-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37760000 2016-04-07T02:11:37.43-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37770000 2016-04-07T02:11:37.469-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28690000 QueuedCount 25569 2016-04-07T02:11:37.655-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37780000 2016-04-07T02:11:37.878-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37790000 2016-04-07T02:11:37.883-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28700000 QueuedCount 15569 2016-04-07T02:11:38.108-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37800000 2016-04-07T02:11:38.3-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28710000 QueuedCount 5569 2016-04-07T02:11:38.338-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37810000 2016-04-07T02:11:38.558-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28715569, 28715569, 0 committed:false 2016-04-07T02:11:38.558-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28715569, 28715569, 0 committed:false) SnapCreateDur 30.107µs SnapOpenDur 182.928µs 2016-04-07T02:11:38.563-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37820000 2016-04-07T02:11:38.794-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37830000 2016-04-07T02:11:38.937-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28720000 QueuedCount 4801 2016-04-07T02:11:39.023-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37840000 2016-04-07T02:11:39.245-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37850000 2016-04-07T02:11:39.473-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37860000 2016-04-07T02:11:39.636-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.49µs CommitTime 513.528294ms TotalFlushTime 20m59.034832636s TotalCommitTime 2m25.007785034s 2016-04-07T02:11:39.636-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28724801, 28724801, 225 committed:true 2016-04-07T02:11:39.636-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28724801, 28724801, 225 committed:true) SnapCreateDur 522.910876ms SnapOpenDur 52.279µs 2016-04-07T02:11:39.64-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:39.645-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:39.649-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:39.654-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:39.658-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:39.694-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37870000 2016-04-07T02:11:39.893-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28730000 QueuedCount 48842 2016-04-07T02:11:39.917-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37880000 2016-04-07T02:11:39.966-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:7700958242, Disk:20360749056, ExtraSnapshotData:0, Fragmentation:62.17743158260356%, mode : circular 2016-04-07T02:11:39.966-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:11:40.195-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37890000 2016-04-07T02:11:40.234-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28740000 QueuedCount 38842 2016-04-07T02:11:40.463-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37900000 2016-04-07T02:11:40.592-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28750000 QueuedCount 28842 2016-04-07T02:11:40.684-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37910000 2016-04-07T02:11:40.917-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37920000 2016-04-07T02:11:40.941-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28760000 QueuedCount 18842 2016-04-07T02:11:41.139-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37930000 2016-04-07T02:11:41.302-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28770000 QueuedCount 8842 2016-04-07T02:11:41.368-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37940000 2016-04-07T02:11:41.593-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37950000 2016-04-07T02:11:41.625-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28778842, 28778842, 0 committed:false 2016-04-07T02:11:41.626-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28778842, 28778842, 0 committed:false) SnapCreateDur 30.391µs SnapOpenDur 647.976µs 2016-04-07T02:11:41.63-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:41.635-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:41.639-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:41.643-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:41.647-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:41.651-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28780000 QueuedCount 50000 2016-04-07T02:11:41.822-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37960000 2016-04-07T02:11:42.048-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37970000 2016-04-07T02:11:42.103-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28790000 QueuedCount 41923 2016-04-07T02:11:42.273-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37980000 2016-04-07T02:11:42.503-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 37990000 2016-04-07T02:11:42.506-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28800000 QueuedCount 31923 2016-04-07T02:11:42.727-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38000000 2016-04-07T02:11:42.904-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28810000 QueuedCount 21923 2016-04-07T02:11:42.956-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38010000 2016-04-07T02:11:43.187-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38020000 2016-04-07T02:11:43.311-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28820000 QueuedCount 11923 2016-04-07T02:11:43.415-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38030000 2016-04-07T02:11:43.635-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38040000 2016-04-07T02:11:43.723-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28830000 QueuedCount 1923 2016-04-07T02:11:43.816-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28831923, 28831923, 0 committed:false 2016-04-07T02:11:43.817-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28831923, 28831923, 0 committed:false) SnapCreateDur 48.443µs SnapOpenDur 626.804µs 2016-04-07T02:11:43.86-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38050000 2016-04-07T02:11:44.074-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38060000 2016-04-07T02:11:44.3-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38070000 2016-04-07T02:11:44.338-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28840000 QueuedCount 948 2016-04-07T02:11:44.525-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38080000 2016-04-07T02:11:44.752-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38090000 2016-04-07T02:11:44.911-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.586µs CommitTime 550.21757ms TotalFlushTime 21m3.665964092s TotalCommitTime 2m25.558002604s 2016-04-07T02:11:44.911-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28840948, 28840948, 226 committed:true 2016-04-07T02:11:44.912-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28840948, 28840948, 226 committed:true) SnapCreateDur 560.305019ms SnapOpenDur 44.816µs 2016-04-07T02:11:44.916-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:44.92-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:44.924-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:44.928-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:44.932-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:44.978-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38100000 2016-04-07T02:11:45.202-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38110000 2016-04-07T02:11:45.347-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28850000 QueuedCount 44007 2016-04-07T02:11:45.429-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38120000 2016-04-07T02:11:45.652-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38130000 2016-04-07T02:11:45.685-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28860000 QueuedCount 34007 2016-04-07T02:11:45.88-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38140000 2016-04-07T02:11:46.048-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28870000 QueuedCount 24007 2016-04-07T02:11:46.104-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38150000 2016-04-07T02:11:46.332-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38160000 2016-04-07T02:11:46.422-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28880000 QueuedCount 14007 2016-04-07T02:11:46.555-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38170000 2016-04-07T02:11:46.748-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28890000 QueuedCount 4007 2016-04-07T02:11:46.788-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38180000 2016-04-07T02:11:46.912-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28894007, 28894007, 0 committed:false 2016-04-07T02:11:46.913-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28894007, 28894007, 0 committed:false) SnapCreateDur 83.673µs SnapOpenDur 1.598628ms 2016-04-07T02:11:46.918-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:46.922-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:46.927-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:46.931-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:46.935-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:47.01-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38190000 2016-04-07T02:11:47.238-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38200000 2016-04-07T02:11:47.251-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28900000 QueuedCount 44156 2016-04-07T02:11:47.468-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38210000 2016-04-07T02:11:47.627-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28910000 QueuedCount 34156 2016-04-07T02:11:47.687-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38220000 2016-04-07T02:11:47.913-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38230000 2016-04-07T02:11:48.014-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28920000 QueuedCount 24156 2016-04-07T02:11:48.133-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38240000 2016-04-07T02:11:48.362-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38250000 2016-04-07T02:11:48.412-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28930000 QueuedCount 14156 2016-04-07T02:11:48.6-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38260000 2016-04-07T02:11:48.823-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28940000 QueuedCount 4156 2016-04-07T02:11:48.826-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38270000 2016-04-07T02:11:48.988-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28944156, 28944156, 0 committed:false 2016-04-07T02:11:48.989-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28944156, 28944156, 0 committed:false) SnapCreateDur 51.656µs SnapOpenDur 931.531µs 2016-04-07T02:11:49.073-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38280000 2016-04-07T02:11:49.305-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38290000 2016-04-07T02:11:49.331-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28950000 QueuedCount 2507 2016-04-07T02:11:49.53-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38300000 2016-04-07T02:11:49.757-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38310000 2016-04-07T02:11:49.925-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 30.243µs CommitTime 491.361394ms TotalFlushTime 21m8.099914123s TotalCommitTime 2m26.049363998s 2016-04-07T02:11:49.925-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 28952507, 28952507, 227 committed:true 2016-04-07T02:11:49.925-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 28952507, 28952507, 227 committed:true) SnapCreateDur 501.623476ms SnapOpenDur 48.429µs 2016-04-07T02:11:49.93-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:49.935-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:49.94-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:49.944-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:49.949-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:49.987-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38320000 2016-04-07T02:11:50.236-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38330000 2016-04-07T02:11:50.301-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28960000 QueuedCount 46457 2016-04-07T02:11:50.457-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38340000 2016-04-07T02:11:50.629-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28970000 QueuedCount 36457 2016-04-07T02:11:50.714-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38350000 2016-04-07T02:11:50.944-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38360000 2016-04-07T02:11:50.961-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28980000 QueuedCount 26457 2016-04-07T02:11:51.172-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38370000 2016-04-07T02:11:51.324-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 28990000 QueuedCount 16457 2016-04-07T02:11:51.4-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38380000 2016-04-07T02:11:51.624-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38390000 2016-04-07T02:11:51.692-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29000000 QueuedCount 6457 2016-04-07T02:11:51.851-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38400000 2016-04-07T02:11:51.933-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29006457, 29006457, 0 committed:false 2016-04-07T02:11:51.933-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29006457, 29006457, 0 committed:false) SnapCreateDur 34.135µs SnapOpenDur 647.603µs 2016-04-07T02:11:51.938-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:51.943-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:51.947-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:51.952-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:51.956-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:52.084-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38410000 2016-04-07T02:11:52.22-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29010000 QueuedCount 49194 2016-04-07T02:11:52.31-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38420000 2016-04-07T02:11:52.538-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38430000 2016-04-07T02:11:52.558-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29020000 QueuedCount 39194 2016-04-07T02:11:52.756-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38440000 2016-04-07T02:11:52.949-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29030000 QueuedCount 29194 2016-04-07T02:11:52.971-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38450000 2016-04-07T02:11:53.199-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38460000 2016-04-07T02:11:53.349-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29040000 QueuedCount 19194 2016-04-07T02:11:53.413-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38470000 2016-04-07T02:11:53.634-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38480000 2016-04-07T02:11:53.774-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29050000 QueuedCount 9194 2016-04-07T02:11:53.854-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38490000 2016-04-07T02:11:54.072-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38500000 2016-04-07T02:11:54.201-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29059194, 29059194, 0 committed:false 2016-04-07T02:11:54.201-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29059194, 29059194, 0 committed:false) SnapCreateDur 31.18µs SnapOpenDur 191.437µs 2016-04-07T02:11:54.211-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29060000 QueuedCount 8065 2016-04-07T02:11:54.292-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38510000 2016-04-07T02:11:54.513-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38520000 2016-04-07T02:11:54.737-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38530000 2016-04-07T02:11:54.965-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38540000 2016-04-07T02:11:55.189-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38550000 2016-04-07T02:11:55.335-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.006µs CommitTime 598.503105ms TotalFlushTime 21m12.805965467s TotalCommitTime 2m26.647867103s 2016-04-07T02:11:55.335-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29068065, 29068065, 228 committed:true 2016-04-07T02:11:55.335-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29068065, 29068065, 228 committed:true) SnapCreateDur 608.325524ms SnapOpenDur 44.429µs 2016-04-07T02:11:55.34-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:55.345-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:55.349-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:55.354-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:55.358-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:55.365-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29070000 QueuedCount 50000 2016-04-07T02:11:55.413-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38560000 2016-04-07T02:11:55.635-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38570000 2016-04-07T02:11:55.86-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38580000 2016-04-07T02:11:55.908-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29080000 QueuedCount 41353 2016-04-07T02:11:56.087-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38590000 2016-04-07T02:11:56.264-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29090000 QueuedCount 31353 2016-04-07T02:11:56.305-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38600000 2016-04-07T02:11:56.527-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38610000 2016-04-07T02:11:56.64-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29100000 QueuedCount 21353 2016-04-07T02:11:56.751-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38620000 2016-04-07T02:11:56.975-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38630000 2016-04-07T02:11:57.019-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29110000 QueuedCount 11353 2016-04-07T02:11:57.199-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38640000 2016-04-07T02:11:57.409-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29120000 QueuedCount 1353 2016-04-07T02:11:57.423-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38650000 2016-04-07T02:11:57.483-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29121353, 29121353, 0 committed:false 2016-04-07T02:11:57.484-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29121353, 29121353, 0 committed:false) SnapCreateDur 33.89µs SnapOpenDur 116.694µs 2016-04-07T02:11:57.488-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:57.492-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:57.497-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:57.501-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:57.504-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:11:57.649-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38660000 2016-04-07T02:11:57.882-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38670000 2016-04-07T02:11:57.905-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29130000 QueuedCount 45175 2016-04-07T02:11:58.109-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38680000 2016-04-07T02:11:58.286-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29140000 QueuedCount 35175 2016-04-07T02:11:58.333-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38690000 2016-04-07T02:11:58.559-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38700000 2016-04-07T02:11:58.615-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29150000 QueuedCount 25175 2016-04-07T02:11:58.79-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38710000 2016-04-07T02:11:59.01-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29160000 QueuedCount 15175 2016-04-07T02:11:59.019-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38720000 2016-04-07T02:11:59.239-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38730000 2016-04-07T02:11:59.394-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29170000 QueuedCount 5175 2016-04-07T02:11:59.466-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38740000 2016-04-07T02:11:59.631-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29175175, 29175175, 0 committed:false 2016-04-07T02:11:59.632-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29175175, 29175175, 0 committed:false) SnapCreateDur 38.859µs SnapOpenDur 481.52µs 2016-04-07T02:11:59.691-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38750000 2016-04-07T02:11:59.913-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38760000 2016-04-07T02:12:00.044-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29180000 QueuedCount 3999 2016-04-07T02:12:00.149-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38770000 2016-04-07T02:12:00.38-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38780000 2016-04-07T02:12:00.602-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38790000 2016-04-07T02:12:00.805-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.03µs CommitTime 578.79587ms TotalFlushTime 21m17.609726557s TotalCommitTime 2m27.226662973s 2016-04-07T02:12:00.807-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29183999, 29183999, 229 committed:true 2016-04-07T02:12:00.807-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29183999, 29183999, 229 committed:true) SnapCreateDur 590.237684ms SnapOpenDur 33.847µs 2016-04-07T02:12:00.811-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:00.815-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:00.819-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:00.824-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:00.828-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:00.869-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38800000 2016-04-07T02:12:01.123-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38810000 2016-04-07T02:12:01.151-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29190000 QueuedCount 47460 2016-04-07T02:12:01.357-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38820000 2016-04-07T02:12:01.482-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29200000 QueuedCount 37460 2016-04-07T02:12:01.575-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38830000 2016-04-07T02:12:01.795-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38840000 2016-04-07T02:12:01.832-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29210000 QueuedCount 27460 2016-04-07T02:12:02.066-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38850000 2016-04-07T02:12:02.275-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38860000 2016-04-07T02:12:02.34-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29220000 QueuedCount 17460 2016-04-07T02:12:02.482-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38870000 2016-04-07T02:12:02.711-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38880000 2016-04-07T02:12:02.827-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29230000 QueuedCount 7460 2016-04-07T02:12:02.933-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38890000 2016-04-07T02:12:03.176-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38900000 2016-04-07T02:12:03.235-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29237460, 29237460, 0 committed:false 2016-04-07T02:12:03.235-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29237460, 29237460, 0 committed:false) SnapCreateDur 46.148µs SnapOpenDur 414.008µs 2016-04-07T02:12:03.665-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:03.671-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:03.677-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:03.683-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:03.688-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38910000 2016-04-07T02:12:03.689-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:03.703-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38920000 2016-04-07T02:12:03.816-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29240000 QueuedCount 50000 2016-04-07T02:12:03.849-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38930000 2016-04-07T02:12:04.073-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38940000 2016-04-07T02:12:04.185-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29250000 QueuedCount 40770 2016-04-07T02:12:04.297-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38950000 2016-04-07T02:12:04.527-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38960000 2016-04-07T02:12:04.56-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29260000 QueuedCount 30770 2016-04-07T02:12:04.749-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38970000 2016-04-07T02:12:04.885-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29270000 QueuedCount 20770 2016-04-07T02:12:04.978-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38980000 2016-04-07T02:12:05.207-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 38990000 2016-04-07T02:12:05.278-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29280000 QueuedCount 10770 2016-04-07T02:12:05.432-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39000000 2016-04-07T02:12:05.655-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39010000 2016-04-07T02:12:05.688-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29290000 QueuedCount 770 2016-04-07T02:12:05.757-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29290770, 29290770, 0 committed:false 2016-04-07T02:12:05.758-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29290770, 29290770, 0 committed:false) SnapCreateDur 41.538µs SnapOpenDur 273.016µs 2016-04-07T02:12:05.872-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39020000 2016-04-07T02:12:06.096-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39030000 2016-04-07T02:12:06.318-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39040000 2016-04-07T02:12:06.554-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39050000 2016-04-07T02:12:06.779-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39060000 2016-04-07T02:12:06.896-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 31.691µs CommitTime 553.838116ms TotalFlushTime 21m22.988746264s TotalCommitTime 2m27.780501089s 2016-04-07T02:12:06.896-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29299574, 29299574, 230 committed:true 2016-04-07T02:12:06.896-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29299574, 29299574, 230 committed:true) SnapCreateDur 563.320732ms SnapOpenDur 46.991µs 2016-04-07T02:12:06.902-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:06.905-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29300000 QueuedCount 16230 2016-04-07T02:12:06.906-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:06.911-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:06.915-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:06.919-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:07.005-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39070000 2016-04-07T02:12:07.232-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39080000 2016-04-07T02:12:07.309-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29310000 QueuedCount 43532 2016-04-07T02:12:07.455-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39090000 2016-04-07T02:12:07.596-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29320000 QueuedCount 33532 2016-04-07T02:12:07.671-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39100000 2016-04-07T02:12:07.896-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39110000 2016-04-07T02:12:07.94-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29330000 QueuedCount 23532 2016-04-07T02:12:08.126-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39120000 2016-04-07T02:12:08.295-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29340000 QueuedCount 13532 2016-04-07T02:12:08.345-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39130000 2016-04-07T02:12:08.57-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39140000 2016-04-07T02:12:08.661-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29350000 QueuedCount 3532 2016-04-07T02:12:08.788-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39150000 2016-04-07T02:12:08.81-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29353532, 29353532, 0 committed:false 2016-04-07T02:12:08.811-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29353532, 29353532, 0 committed:false) SnapCreateDur 52.518µs SnapOpenDur 679.763µs 2016-04-07T02:12:08.816-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:08.82-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:08.825-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:08.829-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:08.833-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:09.021-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39160000 2016-04-07T02:12:09.101-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29360000 QueuedCount 42874 2016-04-07T02:12:09.248-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39170000 2016-04-07T02:12:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:12:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:12:09.374-07:00 [Info] client load stats {} 2016-04-07T02:12:09.472-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39180000 2016-04-07T02:12:09.485-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29370000 QueuedCount 32874 2016-04-07T02:12:09.656-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":9784957,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":1808265804,"bucket-1:myindex:avg_ts_items_count":40934,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":7859812639,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":20360749056,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":27891,"bucket-1:myindex:frag_percent":61,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":3079251420,"bucket-1:myindex:items_count":29299574,"bucket-1:myindex:num_commits":230,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":29374982,"bucket-1:myindex:num_docs_pending":883,"bucket-1:myindex:num_docs_processed":29344577,"bucket-1:myindex:num_docs_queued":9761393,"bucket-1:myindex:num_flush_queued":29402874,"bucket-1:myindex:num_items_flushed":29374983,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":414,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":1914383380,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"230 147780501089 -8621764937603739787","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"29374986 162461564505 1320104433942668829","bucket-1:myindex:timings/storage_info":"1289 9503755 136516985475","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"229 80747897 31858650011105","bucket-1:myindex:timings/storage_meta_set":"230 81013568 31225628389942","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"230 9479103 434419996085","bucket-1:myindex:timings/storage_set":"58749972 1107778053638 -2157507571229462814","bucket-1:myindex:timings/storage_snapshot_close":"183 129541892 274764487609672","bucket-1:myindex:timings/storage_snapshot_create":"184 159083651 200506910621255","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":39187831,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"380 669426511 28210782763316341","bucket-1:ts_queue_size":1104,"indexer_state":"Active","memory_quota":31457280000,"memory_used":24821743616,"memory_used_storage":18874355712,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"4393 793709479 349259785707065","uptime":"43m1.349055548s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:12:09.68-07:00 [Info] memstats {"Alloc":5292598872, "TotalAlloc":42423635408, "Sys":9855607112, "Lookups":9112, "Mallocs":1210695064,"Frees":1080696166, "HeapAlloc":5292598872, "HeapSys":9382199296, "HeapIdle":3733520384, "HeapInuse":5648678912,"HeapReleased":3283763200, "HeapObjects":129998898,"GCSys":298708992, "LastGC":1460020241388974489,"PauseTotalNs":54649797, "PauseNs":[1799692], "NumGC":41} 2016-04-07T02:12:09.703-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39190000 2016-04-07T02:12:09.882-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29380000 QueuedCount 22874 2016-04-07T02:12:09.929-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39200000 2016-04-07T02:12:09.966-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:7874906265, Disk:20360749056, ExtraSnapshotData:0, Fragmentation:61.323101407807066%, mode : circular 2016-04-07T02:12:09.966-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:12:10.154-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39210000 2016-04-07T02:12:10.263-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29390000 QueuedCount 12874 2016-04-07T02:12:10.41-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39220000 2016-04-07T02:12:10.601-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39230000 2016-04-07T02:12:10.646-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29400000 QueuedCount 2874 2016-04-07T02:12:10.737-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29402874, 29402874, 0 committed:false 2016-04-07T02:12:10.737-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29402874, 29402874, 0 committed:false) SnapCreateDur 39.841µs SnapOpenDur 798.155µs 2016-04-07T02:12:10.742-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:10.745-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:10.749-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:10.753-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:10.758-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:10.85-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39240000 2016-04-07T02:12:11.079-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39250000 2016-04-07T02:12:11.112-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29410000 QueuedCount 45491 2016-04-07T02:12:11.379-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39260000 2016-04-07T02:12:11.515-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29420000 QueuedCount 35491 2016-04-07T02:12:11.568-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39270000 2016-04-07T02:12:11.782-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39280000 2016-04-07T02:12:11.935-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29430000 QueuedCount 25491 2016-04-07T02:12:12.004-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39290000 2016-04-07T02:12:12.229-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39300000 2016-04-07T02:12:12.353-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29440000 QueuedCount 15491 2016-04-07T02:12:12.452-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39310000 2016-04-07T02:12:12.679-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39320000 2016-04-07T02:12:12.777-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29450000 QueuedCount 5491 2016-04-07T02:12:12.898-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39330000 2016-04-07T02:12:13.042-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29455491, 29455491, 0 committed:false 2016-04-07T02:12:13.044-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29455491, 29455491, 0 committed:false) SnapCreateDur 27.804µs SnapOpenDur 1.722188ms 2016-04-07T02:12:13.126-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39340000 2016-04-07T02:12:13.361-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39350000 2016-04-07T02:12:13.405-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29460000 QueuedCount 4198 2016-04-07T02:12:13.582-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39360000 2016-04-07T02:12:13.8-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39370000 2016-04-07T02:12:14.026-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39380000 2016-04-07T02:12:14.203-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.802µs CommitTime 614.558913ms TotalFlushTime 21m29.560981165s TotalCommitTime 2m28.395060002s 2016-04-07T02:12:14.203-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29464198, 29464198, 231 committed:true 2016-04-07T02:12:14.203-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29464198, 29464198, 231 committed:true) SnapCreateDur 623.99077ms SnapOpenDur 45.001µs 2016-04-07T02:12:14.207-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:14.211-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:14.215-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:14.219-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:14.223-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:14.252-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39390000 2016-04-07T02:12:14.477-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39400000 2016-04-07T02:12:14.521-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29470000 QueuedCount 47596 2016-04-07T02:12:14.699-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39410000 2016-04-07T02:12:14.862-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29480000 QueuedCount 37596 2016-04-07T02:12:14.925-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39420000 2016-04-07T02:12:15.149-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39430000 2016-04-07T02:12:15.228-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29490000 QueuedCount 27596 2016-04-07T02:12:15.37-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39440000 2016-04-07T02:12:15.594-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39450000 2016-04-07T02:12:15.594-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29500000 QueuedCount 17596 2016-04-07T02:12:15.817-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39460000 2016-04-07T02:12:15.957-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29510000 QueuedCount 7596 2016-04-07T02:12:16.034-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39470000 2016-04-07T02:12:16.261-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29517596, 29517596, 0 committed:false 2016-04-07T02:12:16.261-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29517596, 29517596, 0 committed:false) SnapCreateDur 32.524µs SnapOpenDur 193.947µs 2016-04-07T02:12:16.265-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:16.268-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39480000 2016-04-07T02:12:16.269-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:16.273-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:16.277-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:16.281-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:16.421-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29520000 QueuedCount 50000 2016-04-07T02:12:16.479-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39490000 2016-04-07T02:12:16.7-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39500000 2016-04-07T02:12:16.815-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29530000 QueuedCount 41670 2016-04-07T02:12:16.927-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39510000 2016-04-07T02:12:17.145-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39520000 2016-04-07T02:12:17.209-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29540000 QueuedCount 31670 2016-04-07T02:12:17.367-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39530000 2016-04-07T02:12:17.565-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29550000 QueuedCount 21670 2016-04-07T02:12:17.595-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39540000 2016-04-07T02:12:17.81-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39550000 2016-04-07T02:12:17.982-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29560000 QueuedCount 11670 2016-04-07T02:12:18.03-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39560000 2016-04-07T02:12:18.247-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39570000 2016-04-07T02:12:18.389-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29570000 QueuedCount 1670 2016-04-07T02:12:18.467-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39580000 2016-04-07T02:12:18.469-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29571670, 29571670, 0 committed:false 2016-04-07T02:12:18.47-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29571670, 29571670, 0 committed:false) SnapCreateDur 29.22µs SnapOpenDur 838.539µs 2016-04-07T02:12:18.687-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39590000 2016-04-07T02:12:18.91-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39600000 2016-04-07T02:12:18.925-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29580000 QueuedCount 698 2016-04-07T02:12:19.143-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39610000 2016-04-07T02:12:19.359-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39620000 2016-04-07T02:12:19.501-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.27µs CommitTime 557.140651ms TotalFlushTime 21m34.222498007s TotalCommitTime 2m28.952200653s 2016-04-07T02:12:19.501-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29580698, 29580698, 232 committed:true 2016-04-07T02:12:19.501-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29580698, 29580698, 232 committed:true) SnapCreateDur 567.07268ms SnapOpenDur 43.451µs 2016-04-07T02:12:19.505-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:19.508-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:19.512-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:19.515-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:19.519-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:19.58-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39630000 2016-04-07T02:12:19.814-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39640000 2016-04-07T02:12:19.884-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29590000 QueuedCount 43823 2016-04-07T02:12:20.045-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39650000 2016-04-07T02:12:20.237-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29600000 QueuedCount 33823 2016-04-07T02:12:20.267-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39660000 2016-04-07T02:12:20.499-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39670000 2016-04-07T02:12:20.587-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29610000 QueuedCount 23823 2016-04-07T02:12:20.714-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39680000 2016-04-07T02:12:20.939-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39690000 2016-04-07T02:12:20.947-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29620000 QueuedCount 13823 2016-04-07T02:12:21.19-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39700000 2016-04-07T02:12:21.309-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29630000 QueuedCount 3823 2016-04-07T02:12:21.4-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29633823, 29633823, 0 committed:false 2016-04-07T02:12:21.402-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29633823, 29633823, 0 committed:false) SnapCreateDur 42.385µs SnapOpenDur 1.761197ms 2016-04-07T02:12:21.406-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:21.409-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:21.413-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:21.416-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:21.42-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:21.463-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39710000 2016-04-07T02:12:21.687-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39720000 2016-04-07T02:12:21.701-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29640000 QueuedCount 47169 2016-04-07T02:12:21.913-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39730000 2016-04-07T02:12:22.099-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29650000 QueuedCount 37169 2016-04-07T02:12:22.14-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39740000 2016-04-07T02:12:22.355-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39750000 2016-04-07T02:12:22.501-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29660000 QueuedCount 27169 2016-04-07T02:12:22.574-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39760000 2016-04-07T02:12:22.797-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39770000 2016-04-07T02:12:22.899-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29670000 QueuedCount 17169 2016-04-07T02:12:23.028-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39780000 2016-04-07T02:12:23.253-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39790000 2016-04-07T02:12:23.3-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29680000 QueuedCount 7169 2016-04-07T02:12:23.479-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39800000 2016-04-07T02:12:23.581-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29687169, 29687169, 0 committed:false 2016-04-07T02:12:23.583-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29687169, 29687169, 0 committed:false) SnapCreateDur 58.213µs SnapOpenDur 2.050949ms 2016-04-07T02:12:23.708-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39810000 2016-04-07T02:12:23.915-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29690000 QueuedCount 5766 2016-04-07T02:12:23.93-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39820000 2016-04-07T02:12:24.152-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39830000 2016-04-07T02:12:24.374-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39840000 2016-04-07T02:12:24.599-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39850000 2016-04-07T02:12:24.706-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.788µs CommitTime 539.59861ms TotalFlushTime 21m38.811393885s TotalCommitTime 2m29.491799263s 2016-04-07T02:12:24.707-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29695766, 29695766, 233 committed:true 2016-04-07T02:12:24.707-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29695766, 29695766, 233 committed:true) SnapCreateDur 549.289542ms SnapOpenDur 56.957µs 2016-04-07T02:12:24.71-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:24.714-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:24.717-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:24.721-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:24.726-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:24.823-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39860000 2016-04-07T02:12:25.012-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29700000 QueuedCount 49606 2016-04-07T02:12:25.069-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39870000 2016-04-07T02:12:25.295-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39880000 2016-04-07T02:12:25.331-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29710000 QueuedCount 39606 2016-04-07T02:12:25.534-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39890000 2016-04-07T02:12:25.724-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29720000 QueuedCount 29606 2016-04-07T02:12:25.759-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39900000 2016-04-07T02:12:25.984-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39910000 2016-04-07T02:12:26.105-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29730000 QueuedCount 19606 2016-04-07T02:12:26.211-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39920000 2016-04-07T02:12:26.435-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39930000 2016-04-07T02:12:26.511-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29740000 QueuedCount 9606 2016-04-07T02:12:26.655-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39940000 2016-04-07T02:12:26.88-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39950000 2016-04-07T02:12:26.897-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29749606, 29749606, 0 committed:false 2016-04-07T02:12:26.898-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29749606, 29749606, 0 committed:false) SnapCreateDur 22.885µs SnapOpenDur 528.084µs 2016-04-07T02:12:26.902-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:26.905-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:26.906-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29750000 QueuedCount 17787 2016-04-07T02:12:26.909-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:26.912-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:26.916-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:27.097-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39960000 2016-04-07T02:12:27.319-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39970000 2016-04-07T02:12:27.349-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29760000 QueuedCount 42329 2016-04-07T02:12:27.538-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39980000 2016-04-07T02:12:27.728-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29770000 QueuedCount 32329 2016-04-07T02:12:27.764-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 39990000 2016-04-07T02:12:28.04-07:00 [Info] logReaderStat:: MAINT_STREAM MutationCount 40000000 2016-04-07T02:12:28.088-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29780000 QueuedCount 22329 2016-04-07T02:12:28.439-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29790000 QueuedCount 12329 2016-04-07T02:12:28.746-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29800000 QueuedCount 2329 2016-04-07T02:12:28.883-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29802329, 29802329, 0 committed:false 2016-04-07T02:12:28.884-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29802329, 29802329, 0 committed:false) SnapCreateDur 33.019µs SnapOpenDur 158.295µs 2016-04-07T02:12:29.344-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29810000 QueuedCount 1407 2016-04-07T02:12:29.99-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.56µs CommitTime 562.987939ms TotalFlushTime 21m43.449393622s TotalCommitTime 2m30.054787202s 2016-04-07T02:12:29.99-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29811407, 29811407, 234 committed:true 2016-04-07T02:12:29.99-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29811407, 29811407, 234 committed:true) SnapCreateDur 571.75025ms SnapOpenDur 58.004µs 2016-04-07T02:12:29.994-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:29.998-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:30.002-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:30.006-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:30.01-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:30.336-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29820000 QueuedCount 44648 2016-04-07T02:12:30.663-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29830000 QueuedCount 34648 2016-04-07T02:12:30.942-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29840000 QueuedCount 24648 2016-04-07T02:12:31.057-07:00 [Info] Indexer::ReadMemstats Time Taken 17.575496ms 2016-04-07T02:12:31.282-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29850000 QueuedCount 14648 2016-04-07T02:12:31.631-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29860000 QueuedCount 4648 2016-04-07T02:12:31.795-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29864648, 29864648, 0 committed:false 2016-04-07T02:12:31.797-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29864648, 29864648, 0 committed:false) SnapCreateDur 27.292µs SnapOpenDur 1.959738ms 2016-04-07T02:12:31.801-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:31.805-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:31.809-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:31.812-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:31.816-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:32.059-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29870000 QueuedCount 41734 2016-04-07T02:12:32.394-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29880000 QueuedCount 31734 2016-04-07T02:12:32.738-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29890000 QueuedCount 21734 2016-04-07T02:12:33.099-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29900000 QueuedCount 11734 2016-04-07T02:12:33.455-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29910000 QueuedCount 1734 2016-04-07T02:12:33.479-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29911734, 29911734, 0 committed:false 2016-04-07T02:12:33.481-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29911734, 29911734, 0 committed:false) SnapCreateDur 55.414µs SnapOpenDur 1.958042ms 2016-04-07T02:12:33.482-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:33.494-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:33.499-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:33.712-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:34.039-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29920000 QueuedCount 50000 2016-04-07T02:12:34.047-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:34.414-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29930000 QueuedCount 49523 2016-04-07T02:12:34.81-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29940000 QueuedCount 39523 2016-04-07T02:12:35.191-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29950000 QueuedCount 29523 2016-04-07T02:12:35.566-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29960000 QueuedCount 19523 2016-04-07T02:12:35.943-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29970000 QueuedCount 9523 2016-04-07T02:12:36.904-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 38.174µs CommitTime 576.096725ms TotalFlushTime 21m49.679177712s TotalCommitTime 2m30.630883927s 2016-04-07T02:12:36.904-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 29979523, 29979523, 235 committed:true 2016-04-07T02:12:36.904-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 29979523, 29979523, 235 committed:true) SnapCreateDur 585.143685ms SnapOpenDur 53.386µs 2016-04-07T02:12:36.909-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:36.913-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29980000 QueuedCount 16227 2016-04-07T02:12:36.914-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:36.918-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:36.922-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:36.927-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:37.325-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 29990000 QueuedCount 43344 2016-04-07T02:12:37.603-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30000000 QueuedCount 33344 2016-04-07T02:12:37.932-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30010000 QueuedCount 23344 2016-04-07T02:12:38.269-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30020000 QueuedCount 13344 2016-04-07T02:12:38.62-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30030000 QueuedCount 3344 2016-04-07T02:12:38.76-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30033344, 30033344, 0 committed:false 2016-04-07T02:12:38.761-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30033344, 30033344, 0 committed:false) SnapCreateDur 40.358µs SnapOpenDur 498.525µs 2016-04-07T02:12:38.765-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:38.769-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:38.773-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:38.777-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:38.781-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:39.047-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30040000 QueuedCount 45517 2016-04-07T02:12:39.394-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30050000 QueuedCount 35517 2016-04-07T02:12:39.763-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30060000 QueuedCount 25517 2016-04-07T02:12:39.967-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:8056702489, Disk:20360749056, ExtraSnapshotData:0, Fragmentation:60.43022549494163%, mode : circular 2016-04-07T02:12:39.967-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:12:40.112-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30070000 QueuedCount 15517 2016-04-07T02:12:40.47-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30080000 QueuedCount 5517 2016-04-07T02:12:40.641-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30085517, 30085517, 0 committed:false 2016-04-07T02:12:40.642-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30085517, 30085517, 0 committed:false) SnapCreateDur 57.203µs SnapOpenDur 1.43715ms 2016-04-07T02:12:40.954-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30090000 QueuedCount 3902 2016-04-07T02:12:41.594-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 14.757µs CommitTime 488.267958ms TotalFlushTime 21m53.793907653s TotalCommitTime 2m31.119151885s 2016-04-07T02:12:41.594-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30093902, 30093902, 236 committed:true 2016-04-07T02:12:41.594-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30093902, 30093902, 236 committed:true) SnapCreateDur 497.044552ms SnapOpenDur 50.532µs 2016-04-07T02:12:41.599-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:41.603-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:41.607-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:41.611-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:41.615-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:41.807-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30100000 QueuedCount 45466 2016-04-07T02:12:42.122-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30110000 QueuedCount 35466 2016-04-07T02:12:42.446-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30120000 QueuedCount 25466 2016-04-07T02:12:42.776-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30130000 QueuedCount 15466 2016-04-07T02:12:43.114-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30140000 QueuedCount 5466 2016-04-07T02:12:43.321-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30145466, 30145466, 0 committed:false 2016-04-07T02:12:43.321-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30145466, 30145466, 0 committed:false) SnapCreateDur 52.04µs SnapOpenDur 478.347µs 2016-04-07T02:12:43.326-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:43.33-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:43.334-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:43.338-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:43.342-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:43.593-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30150000 QueuedCount 47584 2016-04-07T02:12:43.97-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30160000 QueuedCount 37584 2016-04-07T02:12:44.346-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30170000 QueuedCount 27584 2016-04-07T02:12:44.71-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30180000 QueuedCount 17584 2016-04-07T02:12:45.01-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30190000 QueuedCount 7584 2016-04-07T02:12:45.315-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30197584, 30197584, 0 committed:false 2016-04-07T02:12:45.316-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30197584, 30197584, 0 committed:false) SnapCreateDur 27.55µs SnapOpenDur 961.521µs 2016-04-07T02:12:45.321-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:45.325-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:45.33-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:45.334-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:45.338-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:45.595-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30200000 QueuedCount 50000 2016-04-07T02:12:45.955-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30210000 QueuedCount 40349 2016-04-07T02:12:46.32-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30220000 QueuedCount 30349 2016-04-07T02:12:46.684-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30230000 QueuedCount 20349 2016-04-07T02:12:47.059-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30240000 QueuedCount 10349 2016-04-07T02:12:47.447-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30250000 QueuedCount 349 2016-04-07T02:12:47.46-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30250349, 30250349, 0 committed:false 2016-04-07T02:12:47.46-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30250349, 30250349, 0 committed:false) SnapCreateDur 30.734µs SnapOpenDur 600.323µs 2016-04-07T02:12:48.758-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 28.722µs CommitTime 634.090523ms TotalFlushTime 22m0.199019915s TotalCommitTime 2m31.753242408s 2016-04-07T02:12:48.758-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30259581, 30259581, 237 committed:true 2016-04-07T02:12:48.758-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30259581, 30259581, 237 committed:true) SnapCreateDur 642.925733ms SnapOpenDur 62.571µs 2016-04-07T02:12:48.762-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:48.766-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30260000 QueuedCount 16060 2016-04-07T02:12:48.767-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:48.771-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:48.775-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:48.78-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:49.174-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30270000 QueuedCount 41869 2016-04-07T02:12:49.453-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30280000 QueuedCount 31869 2016-04-07T02:12:49.771-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30290000 QueuedCount 21869 2016-04-07T02:12:50.098-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30300000 QueuedCount 11869 2016-04-07T02:12:50.433-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30310000 QueuedCount 1869 2016-04-07T02:12:50.51-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30311869, 30311869, 0 committed:false 2016-04-07T02:12:50.511-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30311869, 30311869, 0 committed:false) SnapCreateDur 27.936µs SnapOpenDur 976.387µs 2016-04-07T02:12:50.516-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:50.52-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:50.524-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:50.529-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:50.533-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:50.937-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30320000 QueuedCount 44006 2016-04-07T02:12:51.288-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30330000 QueuedCount 34006 2016-04-07T02:12:51.64-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30340000 QueuedCount 24006 2016-04-07T02:12:52.016-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30350000 QueuedCount 14006 2016-04-07T02:12:52.393-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30360000 QueuedCount 4006 2016-04-07T02:12:52.497-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30364006, 30364006, 0 committed:false 2016-04-07T02:12:52.499-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30364006, 30364006, 0 committed:false) SnapCreateDur 52.801µs SnapOpenDur 1.908582ms 2016-04-07T02:12:52.899-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30370000 QueuedCount 2653 2016-04-07T02:12:53.56-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 41.682µs CommitTime 518.438123ms TotalFlushTime 22m4.396704581s TotalCommitTime 2m32.271680531s 2016-04-07T02:12:53.56-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30372653, 30372653, 238 committed:true 2016-04-07T02:12:53.56-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30372653, 30372653, 238 committed:true) SnapCreateDur 527.032571ms SnapOpenDur 56.573µs 2016-04-07T02:12:53.565-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:53.568-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:53.572-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:53.576-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:53.58-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:53.856-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30380000 QueuedCount 45333 2016-04-07T02:12:54.165-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30390000 QueuedCount 35333 2016-04-07T02:12:54.481-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30400000 QueuedCount 25333 2016-04-07T02:12:54.806-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30410000 QueuedCount 15333 2016-04-07T02:12:55.142-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30420000 QueuedCount 5333 2016-04-07T02:12:55.298-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30425333, 30425333, 0 committed:false 2016-04-07T02:12:55.299-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30425333, 30425333, 0 committed:false) SnapCreateDur 33.882µs SnapOpenDur 1.236908ms 2016-04-07T02:12:55.303-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:55.308-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:55.311-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:55.316-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:55.319-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:55.518-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30430000 QueuedCount 48813 2016-04-07T02:12:55.89-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30440000 QueuedCount 38813 2016-04-07T02:12:56.261-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30450000 QueuedCount 28813 2016-04-07T02:12:56.615-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30460000 QueuedCount 18813 2016-04-07T02:12:56.988-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30470000 QueuedCount 8813 2016-04-07T02:12:57.315-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30478813, 30478813, 0 committed:false 2016-04-07T02:12:57.317-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30478813, 30478813, 0 committed:false) SnapCreateDur 32.068µs SnapOpenDur 1.78341ms 2016-04-07T02:12:57.321-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:57.324-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:57.328-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:57.332-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:57.336-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:12:57.511-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30480000 QueuedCount 50000 2016-04-07T02:12:57.893-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30490000 QueuedCount 42957 2016-04-07T02:12:58.272-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30500000 QueuedCount 32957 2016-04-07T02:12:58.657-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30510000 QueuedCount 22957 2016-04-07T02:12:59.001-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30520000 QueuedCount 12957 2016-04-07T02:12:59.397-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30530000 QueuedCount 2957 2016-04-07T02:12:59.561-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30532957, 30532957, 0 committed:false 2016-04-07T02:12:59.562-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30532957, 30532957, 0 committed:false) SnapCreateDur 30.399µs SnapOpenDur 566.606µs 2016-04-07T02:13:00.025-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30540000 QueuedCount 1999 2016-04-07T02:13:00.699-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.891µs CommitTime 573.370105ms TotalFlushTime 22m10.85244866s TotalCommitTime 2m32.845050636s 2016-04-07T02:13:00.699-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30541999, 30541999, 239 committed:true 2016-04-07T02:13:00.699-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30541999, 30541999, 239 committed:true) SnapCreateDur 583.088083ms SnapOpenDur 51.808µs 2016-04-07T02:13:00.705-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:00.709-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:00.714-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:00.719-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:00.723-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:01.034-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30550000 QueuedCount 45911 2016-04-07T02:13:01.367-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30560000 QueuedCount 35911 2016-04-07T02:13:01.705-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30570000 QueuedCount 25911 2016-04-07T02:13:02.059-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30580000 QueuedCount 15911 2016-04-07T02:13:02.396-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30590000 QueuedCount 5911 2016-04-07T02:13:02.609-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30595911, 30595911, 0 committed:false 2016-04-07T02:13:02.61-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30595911, 30595911, 0 committed:false) SnapCreateDur 51.375µs SnapOpenDur 570.478µs 2016-04-07T02:13:02.615-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:02.619-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:02.623-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:02.628-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:02.632-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:02.89-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30600000 QueuedCount 48225 2016-04-07T02:13:03.253-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30610000 QueuedCount 38225 2016-04-07T02:13:03.618-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30620000 QueuedCount 28225 2016-04-07T02:13:03.932-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30630000 QueuedCount 18225 2016-04-07T02:13:04.323-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30640000 QueuedCount 8225 2016-04-07T02:13:04.639-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30648225, 30648225, 0 committed:false 2016-04-07T02:13:04.641-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30648225, 30648225, 0 committed:false) SnapCreateDur 53.082µs SnapOpenDur 1.572152ms 2016-04-07T02:13:04.84-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30650000 QueuedCount 7455 2016-04-07T02:13:05.647-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 36.239µs CommitTime 512.222371ms TotalFlushTime 22m15.19251013s TotalCommitTime 2m33.357273007s 2016-04-07T02:13:05.647-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30657455, 30657455, 240 committed:true 2016-04-07T02:13:05.647-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30657455, 30657455, 240 committed:true) SnapCreateDur 521.244632ms SnapOpenDur 70.764µs 2016-04-07T02:13:05.652-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:05.656-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:05.661-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:05.664-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:05.668-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:05.824-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30660000 QueuedCount 44903 2016-04-07T02:13:06.14-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30670000 QueuedCount 34903 2016-04-07T02:13:06.47-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30680000 QueuedCount 24903 2016-04-07T02:13:06.76-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30690000 QueuedCount 14903 2016-04-07T02:13:07.115-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30700000 QueuedCount 4903 2016-04-07T02:13:07.342-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30704903, 30704903, 0 committed:false 2016-04-07T02:13:07.342-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30704903, 30704903, 0 committed:false) SnapCreateDur 53.962µs SnapOpenDur 353.113µs 2016-04-07T02:13:07.345-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:07.348-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:07.352-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:07.357-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:07.359-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:07.558-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30710000 QueuedCount 38656 2016-04-07T02:13:07.894-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30720000 QueuedCount 28656 2016-04-07T02:13:08.236-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30730000 QueuedCount 18656 2016-04-07T02:13:08.587-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30740000 QueuedCount 8656 2016-04-07T02:13:08.884-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30748656, 30748656, 0 committed:false 2016-04-07T02:13:08.885-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30748656, 30748656, 0 committed:false) SnapCreateDur 31.944µs SnapOpenDur 1.022997ms 2016-04-07T02:13:08.89-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:08.894-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:08.898-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:08.902-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:08.907-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:09.12-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30750000 QueuedCount 50000 2016-04-07T02:13:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:13:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:13:09.374-07:00 [Info] client load stats {} 2016-04-07T02:13:09.501-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30760000 QueuedCount 42086 2016-04-07T02:13:09.658-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":9197914,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":1557092211,"bucket-1:myindex:avg_ts_items_count":40776,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":8217573413,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":20360749056,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":36718,"bucket-1:myindex:frag_percent":59,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":3220538370,"bucket-1:myindex:items_count":30657455,"bucket-1:myindex:num_commits":240,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":30765366,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":30697453,"bucket-1:myindex:num_docs_queued":9302547,"bucket-1:myindex:num_flush_queued":30802086,"bucket-1:myindex:num_items_flushed":30765368,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":448,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":1543209703,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"240 153357273007 -5493287546921929883","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"30765370 167946346272 1320128607144405646","bucket-1:myindex:timings/storage_info":"1377 10114714 144334154202","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"239 84423139 33297174537613","bucket-1:myindex:timings/storage_meta_set":"240 84423298 32421113544782","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"240 9816436 446178234798","bucket-1:myindex:timings/storage_set":"61530740 1155076342600 902723347545193692","bucket-1:myindex:timings/storage_snapshot_close":"207 151194901 323105935412985","bucket-1:myindex:timings/storage_snapshot_create":"208 182002961 231428888732177","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":40000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"391 680543116 28223247527176026","bucket-1:ts_queue_size":1038,"indexer_state":"Active","memory_quota":31457280000,"memory_used":25135681536,"memory_used_storage":18874335232,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"4518 819718402 354852052417772","uptime":"44m1.350272693s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:13:09.68-07:00 [Info] memstats {"Alloc":4638391544, "TotalAlloc":46009240584, "Sys":9855607112, "Lookups":9197, "Mallocs":1315675266,"Frees":1201503124, "HeapAlloc":4638391544, "HeapSys":9382395904, "HeapIdle":3419758592, "HeapInuse":5962637312,"HeapReleased":1959968768, "HeapObjects":114172142,"GCSys":298708992, "LastGC":1460020323271333686,"PauseTotalNs":56154724, "PauseNs":[1504927], "NumGC":42} 2016-04-07T02:13:09.803-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30770000 QueuedCount 32086 2016-04-07T02:13:09.967-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:8244795958, Disk:20360749056, ExtraSnapshotData:0, Fragmentation:59.506421225841954%, mode : circular 2016-04-07T02:13:09.967-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:13:10.161-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30780000 QueuedCount 22086 2016-04-07T02:13:10.528-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30790000 QueuedCount 12086 2016-04-07T02:13:10.907-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30800000 QueuedCount 2086 2016-04-07T02:13:10.989-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30802086, 30802086, 0 committed:false 2016-04-07T02:13:10.99-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30802086, 30802086, 0 committed:false) SnapCreateDur 29.095µs SnapOpenDur 1.080536ms 2016-04-07T02:13:11.485-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30810000 QueuedCount 1526 2016-04-07T02:13:12.128-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 23.591µs CommitTime 554.227065ms TotalFlushTime 22m21.010335338s TotalCommitTime 2m33.911500072s 2016-04-07T02:13:12.128-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30811526, 30811526, 241 committed:true 2016-04-07T02:13:12.128-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30811526, 30811526, 241 committed:true) SnapCreateDur 563.102388ms SnapOpenDur 71.453µs 2016-04-07T02:13:12.132-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:12.136-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:12.14-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:12.144-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:12.148-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:12.526-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30820000 QueuedCount 45211 2016-04-07T02:13:12.879-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30830000 QueuedCount 35211 2016-04-07T02:13:13.184-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30840000 QueuedCount 25211 2016-04-07T02:13:13.552-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30850000 QueuedCount 15211 2016-04-07T02:13:13.914-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30860000 QueuedCount 5211 2016-04-07T02:13:14.129-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30865211, 30865211, 0 committed:false 2016-04-07T02:13:14.129-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30865211, 30865211, 0 committed:false) SnapCreateDur 30.385µs SnapOpenDur 395.496µs 2016-04-07T02:13:14.133-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:14.137-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:14.141-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:14.145-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:14.149-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:14.356-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30870000 QueuedCount 48538 2016-04-07T02:13:14.712-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30880000 QueuedCount 38538 2016-04-07T02:13:15.077-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30890000 QueuedCount 28538 2016-04-07T02:13:15.454-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30900000 QueuedCount 18538 2016-04-07T02:13:15.84-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30910000 QueuedCount 8538 2016-04-07T02:13:16.161-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30918538, 30918538, 0 committed:false 2016-04-07T02:13:16.162-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30918538, 30918538, 0 committed:false) SnapCreateDur 44.712µs SnapOpenDur 568.531µs 2016-04-07T02:13:16.177-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30920000 QueuedCount 7582 2016-04-07T02:13:17.17-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 22.271µs CommitTime 585.378963ms TotalFlushTime 22m25.386625234s TotalCommitTime 2m34.496879035s 2016-04-07T02:13:17.171-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30927582, 30927582, 242 committed:true 2016-04-07T02:13:17.171-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30927582, 30927582, 242 committed:true) SnapCreateDur 593.951461ms SnapOpenDur 77.151µs 2016-04-07T02:13:17.175-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:17.18-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:17.185-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:17.189-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:17.194-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:17.321-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30930000 QueuedCount 50000 2016-04-07T02:13:17.653-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30940000 QueuedCount 40577 2016-04-07T02:13:17.948-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30950000 QueuedCount 30577 2016-04-07T02:13:18.292-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30960000 QueuedCount 20577 2016-04-07T02:13:18.652-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30970000 QueuedCount 10577 2016-04-07T02:13:19.017-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30980000 QueuedCount 577 2016-04-07T02:13:19.027-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 30980577, 30980577, 0 committed:false 2016-04-07T02:13:19.029-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 30980577, 30980577, 0 committed:false) SnapCreateDur 40.336µs SnapOpenDur 1.857605ms 2016-04-07T02:13:19.033-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:19.037-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:19.041-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:19.045-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:19.048-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:19.45-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 30990000 QueuedCount 43811 2016-04-07T02:13:19.81-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31000000 QueuedCount 33811 2016-04-07T02:13:20.176-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31010000 QueuedCount 23811 2016-04-07T02:13:20.547-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31020000 QueuedCount 13811 2016-04-07T02:13:20.871-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31030000 QueuedCount 3811 2016-04-07T02:13:21.034-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31033811, 31033811, 0 committed:false 2016-04-07T02:13:21.036-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31033811, 31033811, 0 committed:false) SnapCreateDur 37.53µs SnapOpenDur 1.957824ms 2016-04-07T02:13:21.041-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:21.045-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:21.049-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:21.053-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:21.058-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:21.496-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31040000 QueuedCount 47774 2016-04-07T02:13:21.878-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31050000 QueuedCount 37774 2016-04-07T02:13:22.274-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31060000 QueuedCount 27774 2016-04-07T02:13:22.65-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31070000 QueuedCount 17774 2016-04-07T02:13:23.032-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31080000 QueuedCount 7774 2016-04-07T02:13:23.352-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31087774, 31087774, 0 committed:false 2016-04-07T02:13:23.353-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31087774, 31087774, 0 committed:false) SnapCreateDur 39.71µs SnapOpenDur 468.811µs 2016-04-07T02:13:23.678-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31090000 QueuedCount 6290 2016-04-07T02:13:24.641-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.071µs CommitTime 714.532795ms TotalFlushTime 22m32.020812418s TotalCommitTime 2m35.21141183s 2016-04-07T02:13:24.641-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31096290, 31096290, 243 committed:true 2016-04-07T02:13:24.641-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31096290, 31096290, 243 committed:true) SnapCreateDur 723.363164ms SnapOpenDur 69.116µs 2016-04-07T02:13:24.645-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:24.65-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:24.655-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:24.66-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:24.665-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:24.938-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31100000 QueuedCount 47136 2016-04-07T02:13:25.31-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31110000 QueuedCount 37136 2016-04-07T02:13:25.687-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31120000 QueuedCount 27136 2016-04-07T02:13:26.06-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31130000 QueuedCount 17136 2016-04-07T02:13:26.437-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31140000 QueuedCount 7136 2016-04-07T02:13:26.678-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31147136, 31147136, 0 committed:false 2016-04-07T02:13:26.679-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31147136, 31147136, 0 committed:false) SnapCreateDur 40.806µs SnapOpenDur 1.509939ms 2016-04-07T02:13:26.684-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:26.688-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:26.691-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:26.695-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:26.699-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:26.882-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31150000 QueuedCount 44426 2016-04-07T02:13:27.25-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31160000 QueuedCount 34426 2016-04-07T02:13:27.571-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31170000 QueuedCount 24426 2016-04-07T02:13:27.937-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31180000 QueuedCount 14426 2016-04-07T02:13:28.304-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31190000 QueuedCount 4426 2016-04-07T02:13:28.463-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31194426, 31194426, 0 committed:false 2016-04-07T02:13:28.465-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31194426, 31194426, 0 committed:false) SnapCreateDur 40.799µs SnapOpenDur 1.741135ms 2016-04-07T02:13:28.928-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31200000 QueuedCount 3538 2016-04-07T02:13:29.733-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.744µs CommitTime 634.270555ms TotalFlushTime 22m36.390811436s TotalCommitTime 2m35.845682385s 2016-04-07T02:13:29.735-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31203538, 31203538, 244 committed:true 2016-04-07T02:13:29.735-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31203538, 31203538, 244 committed:true) SnapCreateDur 644.669923ms SnapOpenDur 67.532µs 2016-04-07T02:13:29.739-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:29.744-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:29.748-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:29.752-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:29.757-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:30.048-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31210000 QueuedCount 46663 2016-04-07T02:13:30.417-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31220000 QueuedCount 36663 2016-04-07T02:13:30.727-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31230000 QueuedCount 26663 2016-04-07T02:13:31.103-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31240000 QueuedCount 16663 2016-04-07T02:13:31.286-07:00 [Info] Indexer::ReadMemstats Time Taken 17.885573ms 2016-04-07T02:13:31.531-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31250000 QueuedCount 6663 2016-04-07T02:13:31.771-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31256663, 31256663, 0 committed:false 2016-04-07T02:13:31.773-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31256663, 31256663, 0 committed:false) SnapCreateDur 31.942µs SnapOpenDur 1.773992ms 2016-04-07T02:13:31.777-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:31.782-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:31.786-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:31.791-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:31.795-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:32.033-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31260000 QueuedCount 50000 2016-04-07T02:13:32.431-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31270000 QueuedCount 41355 2016-04-07T02:13:32.812-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31280000 QueuedCount 31355 2016-04-07T02:13:33.188-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31290000 QueuedCount 21355 2016-04-07T02:13:33.574-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31300000 QueuedCount 11355 2016-04-07T02:13:33.9-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31310000 QueuedCount 1355 2016-04-07T02:13:33.982-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31311355, 31311355, 0 committed:false 2016-04-07T02:13:33.983-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31311355, 31311355, 0 committed:false) SnapCreateDur 29.712µs SnapOpenDur 1.19562ms 2016-04-07T02:13:34.468-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31320000 QueuedCount 502 2016-04-07T02:13:35.096-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 33.35µs CommitTime 549.360234ms TotalFlushTime 22m41.107828955s TotalCommitTime 2m36.395042619s 2016-04-07T02:13:35.096-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31320502, 31320502, 245 committed:true 2016-04-07T02:13:35.096-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31320502, 31320502, 245 committed:true) SnapCreateDur 558.102265ms SnapOpenDur 51.245µs 2016-04-07T02:13:35.101-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:35.105-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:35.109-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:35.113-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:35.118-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:35.579-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31330000 QueuedCount 43583 2016-04-07T02:13:35.924-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31340000 QueuedCount 33583 2016-04-07T02:13:36.277-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31350000 QueuedCount 23583 2016-04-07T02:13:36.64-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31360000 QueuedCount 13583 2016-04-07T02:13:37.01-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31370000 QueuedCount 3583 2016-04-07T02:13:37.109-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31373583, 31373583, 0 committed:false 2016-04-07T02:13:37.111-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31373583, 31373583, 0 committed:false) SnapCreateDur 40.024µs SnapOpenDur 1.850484ms 2016-04-07T02:13:37.118-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:37.123-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:37.127-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:37.132-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:37.137-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:37.483-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31380000 QueuedCount 48706 2016-04-07T02:13:37.822-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31390000 QueuedCount 38706 2016-04-07T02:13:38.235-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31400000 QueuedCount 28706 2016-04-07T02:13:38.621-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31410000 QueuedCount 18706 2016-04-07T02:13:39.007-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31420000 QueuedCount 8706 2016-04-07T02:13:39.34-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31428706, 31428706, 0 committed:false 2016-04-07T02:13:39.342-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31428706, 31428706, 0 committed:false) SnapCreateDur 29.212µs SnapOpenDur 2.763265ms 2016-04-07T02:13:39.527-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31430000 QueuedCount 8054 2016-04-07T02:13:39.968-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:8421329790, Disk:21569781760, ExtraSnapshotData:0, Fragmentation:60.95774225394852%, mode : circular 2016-04-07T02:13:39.968-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:13:40.409-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 24.025µs CommitTime 561.883197ms TotalFlushTime 22m45.770319148s TotalCommitTime 2m36.956925816s 2016-04-07T02:13:40.409-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31438054, 31438054, 246 committed:true 2016-04-07T02:13:40.409-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31438054, 31438054, 246 committed:true) SnapCreateDur 571.09974ms SnapOpenDur 74.458µs 2016-04-07T02:13:40.414-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:40.419-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:40.423-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:40.427-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:40.432-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:40.443-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31440000 QueuedCount 50000 2016-04-07T02:13:40.936-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31450000 QueuedCount 41113 2016-04-07T02:13:41.312-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31460000 QueuedCount 31113 2016-04-07T02:13:41.7-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31470000 QueuedCount 21113 2016-04-07T02:13:42.081-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31480000 QueuedCount 11113 2016-04-07T02:13:42.466-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31490000 QueuedCount 1113 2016-04-07T02:13:42.481-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31491113, 31491113, 0 committed:false 2016-04-07T02:13:42.483-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31491113, 31491113, 0 committed:false) SnapCreateDur 31.005µs SnapOpenDur 1.62352ms 2016-04-07T02:13:42.488-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:42.493-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:42.497-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:42.502-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:42.507-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:42.986-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31500000 QueuedCount 44994 2016-04-07T02:13:43.393-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31510000 QueuedCount 34994 2016-04-07T02:13:43.78-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31520000 QueuedCount 24994 2016-04-07T02:13:44.111-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31530000 QueuedCount 14994 2016-04-07T02:13:44.51-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31540000 QueuedCount 4994 2016-04-07T02:13:44.756-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31544994, 31544994, 0 committed:false 2016-04-07T02:13:44.756-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31544994, 31544994, 0 committed:false) SnapCreateDur 31.966µs SnapOpenDur 400.522µs 2016-04-07T02:13:45.105-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31550000 QueuedCount 3903 2016-04-07T02:13:45.845-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.201µs CommitTime 574.576716ms TotalFlushTime 22m50.53819239s TotalCommitTime 2m37.531502532s 2016-04-07T02:13:45.845-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31553903, 31553903, 247 committed:true 2016-04-07T02:13:45.845-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31553903, 31553903, 247 committed:true) SnapCreateDur 583.290548ms SnapOpenDur 77.871µs 2016-04-07T02:13:45.85-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:45.855-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:45.86-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:45.865-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:45.87-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:46.191-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31560000 QueuedCount 46970 2016-04-07T02:13:46.531-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31570000 QueuedCount 36970 2016-04-07T02:13:46.882-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31580000 QueuedCount 26970 2016-04-07T02:13:47.249-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31590000 QueuedCount 16970 2016-04-07T02:13:47.613-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31600000 QueuedCount 6970 2016-04-07T02:13:47.854-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31606970, 31606970, 0 committed:false 2016-04-07T02:13:47.856-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31606970, 31606970, 0 committed:false) SnapCreateDur 24.299µs SnapOpenDur 2.045886ms 2016-04-07T02:13:47.86-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:47.866-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:47.871-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:47.876-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:47.881-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:48.06-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31610000 QueuedCount 50000 2016-04-07T02:13:48.459-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31620000 QueuedCount 40526 2016-04-07T02:13:48.849-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31630000 QueuedCount 30526 2016-04-07T02:13:49.261-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31640000 QueuedCount 20526 2016-04-07T02:13:49.585-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31650000 QueuedCount 10526 2016-04-07T02:13:49.97-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31660000 QueuedCount 526 2016-04-07T02:13:50.036-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31660526, 31660526, 0 committed:false 2016-04-07T02:13:50.037-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31660526, 31660526, 0 committed:false) SnapCreateDur 32.34µs SnapOpenDur 178.513µs 2016-04-07T02:13:51.078-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 31.124µs CommitTime 597.636772ms TotalFlushTime 22m55.08488538s TotalCommitTime 2m38.129139304s 2016-04-07T02:13:51.078-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31669697, 31669697, 248 committed:true 2016-04-07T02:13:51.079-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31669697, 31669697, 248 committed:true) SnapCreateDur 606.275156ms SnapOpenDur 69.985µs 2016-04-07T02:13:51.084-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:51.084-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31670000 QueuedCount 8699 2016-04-07T02:13:51.088-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:51.093-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:51.097-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:51.102-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:51.514-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31680000 QueuedCount 43348 2016-04-07T02:13:51.833-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31690000 QueuedCount 33348 2016-04-07T02:13:52.192-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31700000 QueuedCount 23348 2016-04-07T02:13:52.558-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31710000 QueuedCount 13348 2016-04-07T02:13:52.933-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31720000 QueuedCount 3348 2016-04-07T02:13:53.085-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31723348, 31723348, 0 committed:false 2016-04-07T02:13:53.085-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31723348, 31723348, 0 committed:false) SnapCreateDur 26.798µs SnapOpenDur 444.422µs 2016-04-07T02:13:53.091-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:53.097-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:53.102-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:53.107-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:53.112-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:53.462-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31730000 QueuedCount 47798 2016-04-07T02:13:53.861-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31740000 QueuedCount 37798 2016-04-07T02:13:54.242-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31750000 QueuedCount 27798 2016-04-07T02:13:54.626-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31760000 QueuedCount 17798 2016-04-07T02:13:54.946-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31770000 QueuedCount 7798 2016-04-07T02:13:55.259-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31777798, 31777798, 0 committed:false 2016-04-07T02:13:55.26-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31777798, 31777798, 0 committed:false) SnapCreateDur 27.059µs SnapOpenDur 1.365698ms 2016-04-07T02:13:55.517-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31780000 QueuedCount 6708 2016-04-07T02:13:56.465-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 32.324µs CommitTime 630.326421ms TotalFlushTime 22m59.737962276s TotalCommitTime 2m38.759465725s 2016-04-07T02:13:56.465-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31786708, 31786708, 249 committed:true 2016-04-07T02:13:56.465-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31786708, 31786708, 249 committed:true) SnapCreateDur 639.340983ms SnapOpenDur 74.331µs 2016-04-07T02:13:56.47-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:56.476-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:56.481-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:56.486-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:56.491-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:56.716-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31790000 QueuedCount 50000 2016-04-07T02:13:57.084-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31800000 QueuedCount 40503 2016-04-07T02:13:57.482-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31810000 QueuedCount 30503 2016-04-07T02:13:57.871-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31820000 QueuedCount 20503 2016-04-07T02:13:58.277-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31830000 QueuedCount 10503 2016-04-07T02:13:58.659-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31840000 QueuedCount 503 2016-04-07T02:13:58.672-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31840503, 31840503, 0 committed:false 2016-04-07T02:13:58.672-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31840503, 31840503, 0 committed:false) SnapCreateDur 29.474µs SnapOpenDur 494.072µs 2016-04-07T02:13:58.677-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:58.681-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:58.686-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:58.69-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:58.695-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:13:59.12-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31850000 QueuedCount 44545 2016-04-07T02:13:59.495-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31860000 QueuedCount 34545 2016-04-07T02:13:59.878-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31870000 QueuedCount 24545 2016-04-07T02:14:00.265-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31880000 QueuedCount 14545 2016-04-07T02:14:00.652-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31890000 QueuedCount 4545 2016-04-07T02:14:00.826-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31894545, 31894545, 0 committed:false 2016-04-07T02:14:00.827-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31894545, 31894545, 0 committed:false) SnapCreateDur 33.12µs SnapOpenDur 1.217292ms 2016-04-07T02:14:01.24-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31900000 QueuedCount 3573 2016-04-07T02:14:01.958-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 15.883µs CommitTime 547.135466ms TotalFlushTime 23m4.574733859s TotalCommitTime 2m39.306601191s 2016-04-07T02:14:01.958-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31903573, 31903573, 250 committed:true 2016-04-07T02:14:01.958-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31903573, 31903573, 250 committed:true) SnapCreateDur 556.353466ms SnapOpenDur 79.805µs 2016-04-07T02:14:01.962-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:01.966-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:01.97-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:01.974-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:01.978-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:02.29-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31910000 QueuedCount 47336 2016-04-07T02:14:02.64-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31920000 QueuedCount 37336 2016-04-07T02:14:02.945-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31930000 QueuedCount 27336 2016-04-07T02:14:03.351-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31940000 QueuedCount 17336 2016-04-07T02:14:04.011-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31950000 QueuedCount 7336 2016-04-07T02:14:04.942-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 31957336, 31957336, 0 committed:false 2016-04-07T02:14:04.944-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 31957336, 31957336, 0 committed:false) SnapCreateDur 57.652µs SnapOpenDur 1.788804ms 2016-04-07T02:14:04.949-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:04.954-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:04.96-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:04.965-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:04.97-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:05.103-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31960000 QueuedCount 50000 2016-04-07T02:14:05.498-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31970000 QueuedCount 40872 2016-04-07T02:14:05.883-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31980000 QueuedCount 30872 2016-04-07T02:14:06.261-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 31990000 QueuedCount 20872 2016-04-07T02:14:06.646-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32000000 QueuedCount 10872 2016-04-07T02:14:06.97-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32010000 QueuedCount 872 2016-04-07T02:14:07.043-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32010872, 32010872, 0 committed:false 2016-04-07T02:14:07.043-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32010872, 32010872, 0 committed:false) SnapCreateDur 41.608µs SnapOpenDur 591.714µs 2016-04-07T02:14:08.1-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 31.34µs CommitTime 602.603793ms TotalFlushTime 23m9.999287264s TotalCommitTime 2m39.909204984s 2016-04-07T02:14:08.1-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32019606, 32019606, 251 committed:true 2016-04-07T02:14:08.1-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32019606, 32019606, 251 committed:true) SnapCreateDur 611.636107ms SnapOpenDur 77.139µs 2016-04-07T02:14:08.106-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:08.11-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32020000 QueuedCount 15201 2016-04-07T02:14:08.111-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:08.117-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:08.122-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:08.127-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:08.566-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32030000 QueuedCount 43171 2016-04-07T02:14:08.884-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32040000 QueuedCount 33171 2016-04-07T02:14:09.26-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32050000 QueuedCount 23171 2016-04-07T02:14:09.373-07:00 [Info] connected with 1 indexers 2016-04-07T02:14:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:14:09.374-07:00 [Info] client load stats {} 2016-04-07T02:14:09.632-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32060000 QueuedCount 13171 2016-04-07T02:14:09.659-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":7926829,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":1623942934,"bucket-1:myindex:avg_ts_items_count":28007,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":8571155099,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":23016693760,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":12596,"bucket-1:myindex:frag_percent":62,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":3360629747,"bucket-1:myindex:items_count":31903573,"bucket-1:myindex:num_commits":251,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":32060575,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":32001749,"bucket-1:myindex:num_docs_queued":7998251,"bucket-1:myindex:num_flush_queued":32073171,"bucket-1:myindex:num_items_flushed":32060575,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":481,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":1057090712,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"251 159909204984 -1565977670858846788","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"32060576 173481513726 1588470189106035442","bucket-1:myindex:timings/storage_info":"1465 10698932 151308393842","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"250 88713077 35091899212459","bucket-1:myindex:timings/storage_meta_set":"251 89211526 34578301280454","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"251 10423493 480380864217","bucket-1:myindex:timings/storage_set":"64121151 1201323976676 4050625115502957646","bucket-1:myindex:timings/storage_snapshot_close":"230 172391924 362335205179754","bucket-1:myindex:timings/storage_snapshot_create":"230 205615110 265169162041448","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":40000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"403 691032606 28232445670948826","bucket-1:ts_queue_size":895,"indexer_state":"Active","memory_quota":31457280000,"memory_used":25349099520,"memory_used_storage":18874368000,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"4661 850842676 362109229036344","uptime":"45m1.351483908s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:14:09.68-07:00 [Info] memstats {"Alloc":5010795288, "TotalAlloc":46381644328, "Sys":9855607112, "Lookups":9396, "Mallocs":1328366610,"Frees":1204581429, "HeapAlloc":5010795288, "HeapSys":9382166528, "HeapIdle":3206144000, "HeapInuse":6176022528,"HeapReleased":1959968768, "HeapObjects":123785181,"GCSys":298708992, "LastGC":1460020323271333686,"PauseTotalNs":56154724, "PauseNs":[], "NumGC":42} 2016-04-07T02:14:09.968-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:8587789615, Disk:23306592256, ExtraSnapshotData:0, Fragmentation:63.15295895396643%, mode : circular 2016-04-07T02:14:09.968-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:14:10.011-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32070000 QueuedCount 3171 2016-04-07T02:14:10.169-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32073171, 32073171, 0 committed:false 2016-04-07T02:14:10.169-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32073171, 32073171, 0 committed:false) SnapCreateDur 86.805µs SnapOpenDur 321.166µs 2016-04-07T02:14:10.174-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:10.18-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:10.185-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:10.19-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:10.195-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:10.551-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32080000 QueuedCount 46872 2016-04-07T02:14:10.93-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32090000 QueuedCount 36872 2016-04-07T02:14:11.309-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32100000 QueuedCount 26872 2016-04-07T02:14:11.708-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32110000 QueuedCount 16872 2016-04-07T02:14:12.113-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32120000 QueuedCount 6872 2016-04-07T02:14:12.373-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32126872, 32126872, 0 committed:false 2016-04-07T02:14:12.374-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32126872, 32126872, 0 committed:false) SnapCreateDur 36.325µs SnapOpenDur 834.972µs 2016-04-07T02:14:12.606-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32130000 QueuedCount 6014 2016-04-07T02:14:13.659-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 40.287µs CommitTime 800.631025ms TotalFlushTime 23m14.650629705s TotalCommitTime 2m40.709836009s 2016-04-07T02:14:13.66-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32136014, 32136014, 252 committed:true 2016-04-07T02:14:13.66-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32136014, 32136014, 252 committed:true) SnapCreateDur 809.682693ms SnapOpenDur 71.949µs 2016-04-07T02:14:13.665-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:13.671-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:13.676-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:13.682-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:13.687-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:13.924-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32140000 QueuedCount 50000 2016-04-07T02:14:14.285-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32150000 QueuedCount 41078 2016-04-07T02:14:14.657-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32160000 QueuedCount 31078 2016-04-07T02:14:15.032-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32170000 QueuedCount 21078 2016-04-07T02:14:15.412-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32180000 QueuedCount 11078 2016-04-07T02:14:15.797-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32190000 QueuedCount 1078 2016-04-07T02:14:15.808-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32191078, 32191078, 0 committed:false 2016-04-07T02:14:15.81-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32191078, 32191078, 0 committed:false) SnapCreateDur 28.239µs SnapOpenDur 1.84111ms 2016-04-07T02:14:15.815-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:15.82-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:15.825-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:15.83-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:15.834-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:16.279-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32200000 QueuedCount 44235 2016-04-07T02:14:16.66-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32210000 QueuedCount 34235 2016-04-07T02:14:16.989-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32220000 QueuedCount 24235 2016-04-07T02:14:17.402-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32230000 QueuedCount 14235 2016-04-07T02:14:17.817-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32240000 QueuedCount 4235 2016-04-07T02:14:17.992-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32244235, 32244235, 0 committed:false 2016-04-07T02:14:17.994-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32244235, 32244235, 0 committed:false) SnapCreateDur 27.741µs SnapOpenDur 1.941936ms 2016-04-07T02:14:18.484-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32250000 QueuedCount 3336 2016-04-07T02:14:19.289-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 29.954µs CommitTime 640.476593ms TotalFlushTime 23m19.547842598s TotalCommitTime 2m41.350312602s 2016-04-07T02:14:19.289-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32253336, 32253336, 253 committed:true 2016-04-07T02:14:19.289-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32253336, 32253336, 253 committed:true) SnapCreateDur 649.169717ms SnapOpenDur 68.12µs 2016-04-07T02:14:19.294-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:19.3-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:19.304-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:19.309-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:19.314-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:19.618-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32260000 QueuedCount 45442 2016-04-07T02:14:19.997-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32270000 QueuedCount 35442 2016-04-07T02:14:20.369-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32280000 QueuedCount 25442 2016-04-07T02:14:20.687-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32290000 QueuedCount 15442 2016-04-07T02:14:21.075-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32300000 QueuedCount 5442 2016-04-07T02:14:21.309-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32305442, 32305442, 0 committed:false 2016-04-07T02:14:21.31-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32305442, 32305442, 0 committed:false) SnapCreateDur 29.256µs SnapOpenDur 943.633µs 2016-04-07T02:14:21.313-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:21.314-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:21.318-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:21.323-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:21.549-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32310000 QueuedCount 50000 2016-04-07T02:14:21.552-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:21.98-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32320000 QueuedCount 49004 2016-04-07T02:14:22.389-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32330000 QueuedCount 39004 2016-04-07T02:14:22.772-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32340000 QueuedCount 29004 2016-04-07T02:14:23.181-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32350000 QueuedCount 19004 2016-04-07T02:14:23.617-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32360000 QueuedCount 9004 2016-04-07T02:14:23.961-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32369004, 32369004, 0 committed:false 2016-04-07T02:14:23.961-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32369004, 32369004, 0 committed:false) SnapCreateDur 69.401µs SnapOpenDur 801.744µs 2016-04-07T02:14:23.973-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32370000 QueuedCount 7741 2016-04-07T02:14:25.217-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.211µs CommitTime 662.884727ms TotalFlushTime 23m24.710291227s TotalCommitTime 2m42.013197329s 2016-04-07T02:14:25.217-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32377741, 32377741, 254 committed:true 2016-04-07T02:14:25.217-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32377741, 32377741, 254 committed:true) SnapCreateDur 671.625515ms SnapOpenDur 63.934µs 2016-04-07T02:14:25.222-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:25.226-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:25.23-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:25.234-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:25.238-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:25.384-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32380000 QueuedCount 50000 2016-04-07T02:14:25.696-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32390000 QueuedCount 41309 2016-04-07T02:14:26.061-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32400000 QueuedCount 31309 2016-04-07T02:14:26.43-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32410000 QueuedCount 21309 2016-04-07T02:14:26.799-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32420000 QueuedCount 11309 2016-04-07T02:14:27.172-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32430000 QueuedCount 1309 2016-04-07T02:14:27.248-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32431309, 32431309, 0 committed:false 2016-04-07T02:14:27.248-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32431309, 32431309, 0 committed:false) SnapCreateDur 30.187µs SnapOpenDur 287.438µs 2016-04-07T02:14:27.252-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:27.257-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:27.262-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:27.268-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:27.273-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:27.657-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32440000 QueuedCount 44843 2016-04-07T02:14:28.047-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32450000 QueuedCount 34843 2016-04-07T02:14:28.438-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32460000 QueuedCount 24843 2016-04-07T02:14:28.834-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32470000 QueuedCount 14843 2016-04-07T02:14:29.173-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32480000 QueuedCount 4843 2016-04-07T02:14:29.412-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32484843, 32484843, 0 committed:false 2016-04-07T02:14:29.412-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32484843, 32484843, 0 committed:false) SnapCreateDur 50.59µs SnapOpenDur 448.139µs 2016-04-07T02:14:29.788-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32490000 QueuedCount 3680 2016-04-07T02:14:30.636-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 14.569µs CommitTime 667.788785ms TotalFlushTime 23m29.368602692s TotalCommitTime 2m42.680986114s 2016-04-07T02:14:30.636-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32493680, 32493680, 255 committed:true 2016-04-07T02:14:30.636-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32493680, 32493680, 255 committed:true) SnapCreateDur 677.615437ms SnapOpenDur 62.087µs 2016-04-07T02:14:30.642-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:30.648-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:30.653-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:30.659-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:30.665-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:31.008-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32500000 QueuedCount 45294 2016-04-07T02:14:31.326-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32510000 QueuedCount 35294 2016-04-07T02:14:31.709-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32520000 QueuedCount 25294 2016-04-07T02:14:31.981-07:00 [Info] Indexer::ReadMemstats Time Taken 15.864895ms 2016-04-07T02:14:32.112-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32530000 QueuedCount 15294 2016-04-07T02:14:32.507-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32540000 QueuedCount 5294 2016-04-07T02:14:32.742-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32545294, 32545294, 0 committed:false 2016-04-07T02:14:32.742-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32545294, 32545294, 0 committed:false) SnapCreateDur 57.246µs SnapOpenDur 557.654µs 2016-04-07T02:14:32.748-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:32.754-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:32.76-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:32.766-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:32.772-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:32.999-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32550000 QueuedCount 48498 2016-04-07T02:14:33.366-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32560000 QueuedCount 38498 2016-04-07T02:14:33.749-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32570000 QueuedCount 28498 2016-04-07T02:14:34.129-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32580000 QueuedCount 18498 2016-04-07T02:14:34.514-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32590000 QueuedCount 8498 2016-04-07T02:14:34.839-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32598498, 32598498, 0 committed:false 2016-04-07T02:14:34.839-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32598498, 32598498, 0 committed:false) SnapCreateDur 43.92µs SnapOpenDur 307.09µs 2016-04-07T02:14:34.856-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32600000 QueuedCount 7446 2016-04-07T02:14:35.929-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 22.403µs CommitTime 584.550254ms TotalFlushTime 23m33.96548316s TotalCommitTime 2m43.265536368s 2016-04-07T02:14:35.929-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32607446, 32607446, 256 committed:true 2016-04-07T02:14:35.929-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32607446, 32607446, 256 committed:true) SnapCreateDur 594.114266ms SnapOpenDur 70.6µs 2016-04-07T02:14:35.935-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:35.94-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:35.945-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:35.95-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:35.956-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:36.087-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32610000 QueuedCount 50000 2016-04-07T02:14:36.439-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32620000 QueuedCount 41592 2016-04-07T02:14:36.794-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32630000 QueuedCount 31592 2016-04-07T02:14:37.096-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32640000 QueuedCount 21592 2016-04-07T02:14:37.463-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32650000 QueuedCount 11592 2016-04-07T02:14:37.846-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32660000 QueuedCount 1592 2016-04-07T02:14:37.928-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32661592, 32661592, 0 committed:false 2016-04-07T02:14:37.929-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32661592, 32661592, 0 committed:false) SnapCreateDur 32.351µs SnapOpenDur 722.999µs 2016-04-07T02:14:37.934-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:37.939-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:37.944-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:37.95-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:37.955-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:38.311-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32670000 QueuedCount 45032 2016-04-07T02:14:38.697-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32680000 QueuedCount 35032 2016-04-07T02:14:39.088-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32690000 QueuedCount 25032 2016-04-07T02:14:39.466-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32700000 QueuedCount 15032 2016-04-07T02:14:39.859-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32710000 QueuedCount 5032 2016-04-07T02:14:39.969-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:8758510989, Disk:24766775296, ExtraSnapshotData:0, Fragmentation:64.63604613712243%, mode : circular 2016-04-07T02:14:39.969-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:14:40.033-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32715032, 32715032, 0 committed:false 2016-04-07T02:14:40.034-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32715032, 32715032, 0 committed:false) SnapCreateDur 28.2µs SnapOpenDur 853.213µs 2016-04-07T02:14:40.41-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32720000 QueuedCount 3314 2016-04-07T02:14:41.181-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.926µs CommitTime 613.039812ms TotalFlushTime 23m38.506385302s TotalCommitTime 2m43.87857618s 2016-04-07T02:14:41.181-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32723314, 32723314, 257 committed:true 2016-04-07T02:14:41.181-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32723314, 32723314, 257 committed:true) SnapCreateDur 621.833788ms SnapOpenDur 68.609µs 2016-04-07T02:14:41.186-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:41.192-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:41.197-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:41.202-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:41.207-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:41.499-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32730000 QueuedCount 46560 2016-04-07T02:14:41.863-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32740000 QueuedCount 36560 2016-04-07T02:14:42.237-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32750000 QueuedCount 26560 2016-04-07T02:14:42.561-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32760000 QueuedCount 16560 2016-04-07T02:14:42.945-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32770000 QueuedCount 6560 2016-04-07T02:14:43.199-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32776560, 32776560, 0 committed:false 2016-04-07T02:14:43.201-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32776560, 32776560, 0 committed:false) SnapCreateDur 31.139µs SnapOpenDur 1.991678ms 2016-04-07T02:14:43.207-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:43.212-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:43.218-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:43.223-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:43.228-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:43.42-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32780000 QueuedCount 50000 2016-04-07T02:14:43.801-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32790000 QueuedCount 40097 2016-04-07T02:14:44.195-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32800000 QueuedCount 30097 2016-04-07T02:14:44.583-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32810000 QueuedCount 20097 2016-04-07T02:14:44.978-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32820000 QueuedCount 10097 2016-04-07T02:14:45.379-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32830000 QueuedCount 97 2016-04-07T02:14:45.382-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32830097, 32830097, 0 committed:false 2016-04-07T02:14:45.382-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32830097, 32830097, 0 committed:false) SnapCreateDur 31.607µs SnapOpenDur 256.245µs 2016-04-07T02:14:46.573-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.679µs CommitTime 616.639108ms TotalFlushTime 23m43.197389013s TotalCommitTime 2m44.495215288s 2016-04-07T02:14:46.573-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32838805, 32838805, 258 committed:true 2016-04-07T02:14:46.573-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32838805, 32838805, 258 committed:true) SnapCreateDur 625.192619ms SnapOpenDur 79.177µs 2016-04-07T02:14:46.579-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:46.585-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:46.59-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:46.596-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:46.598-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32840000 QueuedCount 38266 2016-04-07T02:14:46.601-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:47.104-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32850000 QueuedCount 42780 2016-04-07T02:14:47.458-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32860000 QueuedCount 32780 2016-04-07T02:14:47.822-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32870000 QueuedCount 22780 2016-04-07T02:14:48.187-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32880000 QueuedCount 12780 2016-04-07T02:14:48.496-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32890000 QueuedCount 2780 2016-04-07T02:14:48.643-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32892780, 32892780, 0 committed:false 2016-04-07T02:14:48.644-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32892780, 32892780, 0 committed:false) SnapCreateDur 28.925µs SnapOpenDur 685.61µs 2016-04-07T02:14:48.649-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:48.654-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:48.659-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:48.664-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:48.668-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:48.952-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32900000 QueuedCount 47339 2016-04-07T02:14:49.337-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32910000 QueuedCount 37339 2016-04-07T02:14:49.735-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32920000 QueuedCount 27339 2016-04-07T02:14:50.142-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32930000 QueuedCount 17339 2016-04-07T02:14:50.533-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32940000 QueuedCount 7339 2016-04-07T02:14:50.79-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32947339, 32947339, 0 committed:false 2016-04-07T02:14:50.792-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32947339, 32947339, 0 committed:false) SnapCreateDur 53.723µs SnapOpenDur 2.203432ms 2016-04-07T02:14:51.053-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32950000 QueuedCount 6313 2016-04-07T02:14:51.884-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 22.406µs CommitTime 578.943114ms TotalFlushTime 23m47.826396847s TotalCommitTime 2m45.074158402s 2016-04-07T02:14:51.884-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 32956313, 32956313, 259 committed:true 2016-04-07T02:14:51.884-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 32956313, 32956313, 259 committed:true) SnapCreateDur 587.59116ms SnapOpenDur 60.696µs 2016-04-07T02:14:51.89-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:51.894-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:51.899-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:51.904-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:51.909-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:52.128-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32960000 QueuedCount 49212 2016-04-07T02:14:52.52-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32970000 QueuedCount 39212 2016-04-07T02:14:52.894-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32980000 QueuedCount 29212 2016-04-07T02:14:53.27-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 32990000 QueuedCount 19212 2016-04-07T02:14:53.654-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33000000 QueuedCount 9212 2016-04-07T02:14:53.991-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33009212, 33009212, 0 committed:false 2016-04-07T02:14:53.992-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33009212, 33009212, 0 committed:false) SnapCreateDur 39.105µs SnapOpenDur 1.450346ms 2016-04-07T02:14:53.998-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:54.003-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:54.008-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:54.013-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:54.018-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:54.126-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33010000 QueuedCount 50000 2016-04-07T02:14:54.506-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33020000 QueuedCount 42341 2016-04-07T02:14:54.822-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33030000 QueuedCount 32341 2016-04-07T02:14:55.198-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33040000 QueuedCount 22341 2016-04-07T02:14:55.581-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33050000 QueuedCount 12341 2016-04-07T02:14:55.975-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33060000 QueuedCount 2341 2016-04-07T02:14:56.07-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33062341, 33062341, 0 committed:false 2016-04-07T02:14:56.071-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33062341, 33062341, 0 committed:false) SnapCreateDur 37.435µs SnapOpenDur 1.332577ms 2016-04-07T02:14:56.611-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33070000 QueuedCount 951 2016-04-07T02:14:57.439-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 46.587µs CommitTime 801.547425ms TotalFlushTime 23m52.486242309s TotalCommitTime 2m45.875705827s 2016-04-07T02:14:57.439-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33070951, 33070951, 260 committed:true 2016-04-07T02:14:57.439-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33070951, 33070951, 260 committed:true) SnapCreateDur 812.676916ms SnapOpenDur 54.105µs 2016-04-07T02:14:57.445-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:57.45-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:57.456-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:57.461-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:57.466-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:57.82-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33080000 QueuedCount 44615 2016-04-07T02:14:58.169-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33090000 QueuedCount 34615 2016-04-07T02:14:58.524-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33100000 QueuedCount 24615 2016-04-07T02:14:58.893-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33110000 QueuedCount 14615 2016-04-07T02:14:59.224-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33120000 QueuedCount 4615 2016-04-07T02:14:59.442-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33124615, 33124615, 0 committed:false 2016-04-07T02:14:59.442-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33124615, 33124615, 0 committed:false) SnapCreateDur 33µs SnapOpenDur 374.578µs 2016-04-07T02:14:59.448-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:59.453-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:59.459-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:59.464-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:59.469-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:14:59.749-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33130000 QueuedCount 48215 2016-04-07T02:15:00.138-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33140000 QueuedCount 38215 2016-04-07T02:15:00.525-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33150000 QueuedCount 28215 2016-04-07T02:15:00.91-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33160000 QueuedCount 18215 2016-04-07T02:15:01.299-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33170000 QueuedCount 8215 2016-04-07T02:15:01.617-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33178215, 33178215, 0 committed:false 2016-04-07T02:15:01.618-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33178215, 33178215, 0 committed:false) SnapCreateDur 29.617µs SnapOpenDur 677.582µs 2016-04-07T02:15:01.81-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33180000 QueuedCount 6947 2016-04-07T02:15:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:15:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:15:09.374-07:00 [Info] client load stats {} 2016-04-07T02:15:09.66-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":6813053,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":2003539395,"bucket-1:myindex:avg_ts_items_count":47152,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":8883902437,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":26222219264,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":0,"bucket-1:myindex:frag_percent":66,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":3484629435,"bucket-1:myindex:items_count":33070951,"bucket-1:myindex:num_commits":260,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":33186947,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":33178215,"bucket-1:myindex:num_docs_queued":6821785,"bucket-1:myindex:num_flush_queued":33186947,"bucket-1:myindex:num_items_flushed":33186947,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":510,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":2175885628,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"260 165875705827 2446014660011875385","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"33186947 177843580487 1588489122730137629","bucket-1:myindex:timings/storage_info":"1544 11220058 157696061618","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"260 92549029 36663296974481","bucket-1:myindex:timings/storage_meta_set":"261 93587408 36554102864230","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"260 10889209 504917482511","bucket-1:myindex:timings/storage_set":"66373894 1241946152877 6858203503904292549","bucket-1:myindex:timings/storage_snapshot_close":"249 189136716 403673791952118","bucket-1:myindex:timings/storage_snapshot_create":"250 223280273 288299599316447","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":40000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"415 701471731 28241559377869951","bucket-1:ts_queue_size":771,"indexer_state":"Active","memory_quota":31457280000,"memory_used":24268419072,"memory_used_storage":18874368000,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"4805 881454770 368785586460718","uptime":"46m1.352553758s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:15:09.68-07:00 [Info] memstats {"Alloc":2784989112, "TotalAlloc":46730220672, "Sys":9855607112, "Lookups":9595, "Mallocs":1340136057,"Frees":1270241814, "HeapAlloc":2784989112, "HeapSys":9382297600, "HeapIdle":4286955520, "HeapInuse":5095342080,"HeapReleased":1959968768, "HeapObjects":69894243,"GCSys":298708992, "LastGC":1460020444345319334,"PauseTotalNs":58052255, "PauseNs":[1897531], "NumGC":43} 2016-04-07T02:15:09.97-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:8883902437, Disk:26222219264, ExtraSnapshotData:0, Fragmentation:66.12070722329538%, mode : circular 2016-04-07T02:15:09.97-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:15:12.683-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 31.318µs CommitTime 10.620692955s TotalFlushTime 23m57.015238634s TotalCommitTime 2m56.496398782s 2016-04-07T02:15:12.683-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33186947, 33186947, 261 committed:true 2016-04-07T02:15:12.683-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33186947, 33186947, 261 committed:true) SnapCreateDur 10.629604074s SnapOpenDur 89.278µs 2016-04-07T02:15:12.904-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33190000 QueuedCount 5808 2016-04-07T02:15:13.182-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.28µs CommitTime 84.739682ms TotalFlushTime 23m57.410097923s TotalCommitTime 2m56.581138464s 2016-04-07T02:15:13.182-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33195808, 33195808, 262 committed:true 2016-04-07T02:15:13.182-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33195808, 33195808, 262 committed:true) SnapCreateDur 93.464306ms SnapOpenDur 63.477µs 2016-04-07T02:15:13.187-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:13.191-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:13.195-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:13.2-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:13.204-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:13.356-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33200000 QueuedCount 49459 2016-04-07T02:15:13.676-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33210000 QueuedCount 39459 2016-04-07T02:15:14.072-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33220000 QueuedCount 29459 2016-04-07T02:15:14.458-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33230000 QueuedCount 19459 2016-04-07T02:15:14.843-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33240000 QueuedCount 9459 2016-04-07T02:15:15.233-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33249459, 33249459, 0 committed:false 2016-04-07T02:15:15.233-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33249459, 33249459, 0 committed:false) SnapCreateDur 32.032µs SnapOpenDur 369.031µs 2016-04-07T02:15:15.238-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:15.243-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:15.244-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33250000 QueuedCount 20368 2016-04-07T02:15:15.247-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:15.252-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:15.256-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:15.757-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33260000 QueuedCount 42917 2016-04-07T02:15:16.158-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33270000 QueuedCount 32917 2016-04-07T02:15:16.56-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33280000 QueuedCount 22917 2016-04-07T02:15:16.897-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33290000 QueuedCount 12917 2016-04-07T02:15:17.296-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33300000 QueuedCount 2917 2016-04-07T02:15:17.456-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33302917, 33302917, 0 committed:false 2016-04-07T02:15:17.456-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33302917, 33302917, 0 committed:false) SnapCreateDur 76.932µs SnapOpenDur 571.191µs 2016-04-07T02:15:17.461-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:17.465-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:17.47-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:17.474-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:17.479-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:17.919-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33310000 QueuedCount 46765 2016-04-07T02:15:18.324-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33320000 QueuedCount 36765 2016-04-07T02:15:18.719-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33330000 QueuedCount 26765 2016-04-07T02:15:19.12-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33340000 QueuedCount 16765 2016-04-07T02:15:19.544-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33350000 QueuedCount 6765 2016-04-07T02:15:19.801-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33356765, 33356765, 0 committed:false 2016-04-07T02:15:19.803-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33356765, 33356765, 0 committed:false) SnapCreateDur 32.123µs SnapOpenDur 2.608404ms 2016-04-07T02:15:20.164-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33360000 QueuedCount 5567 2016-04-07T02:15:20.993-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.618µs CommitTime 634.521322ms TotalFlushTime 24m4.452525052s TotalCommitTime 2m57.215659786s 2016-04-07T02:15:20.993-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33365567, 33365567, 263 committed:true 2016-04-07T02:15:20.993-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33365567, 33365567, 263 committed:true) SnapCreateDur 643.410084ms SnapOpenDur 55.074µs 2016-04-07T02:15:20.998-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:21.003-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:21.008-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:21.013-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:21.018-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:21.265-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33370000 QueuedCount 48405 2016-04-07T02:15:21.643-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33380000 QueuedCount 38405 2016-04-07T02:15:21.994-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33390000 QueuedCount 28405 2016-04-07T02:15:22.398-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33400000 QueuedCount 18405 2016-04-07T02:15:22.803-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33410000 QueuedCount 8405 2016-04-07T02:15:23.121-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33418405, 33418405, 0 committed:false 2016-04-07T02:15:23.123-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33418405, 33418405, 0 committed:false) SnapCreateDur 43.413µs SnapOpenDur 1.542737ms 2016-04-07T02:15:23.128-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:23.132-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:23.136-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:23.14-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:23.144-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:23.302-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33420000 QueuedCount 50000 2016-04-07T02:15:23.696-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33430000 QueuedCount 41590 2016-04-07T02:15:24.087-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33440000 QueuedCount 31590 2016-04-07T02:15:24.479-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33450000 QueuedCount 21590 2016-04-07T02:15:24.91-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33460000 QueuedCount 11590 2016-04-07T02:15:25.268-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33470000 QueuedCount 1590 2016-04-07T02:15:25.351-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33471590, 33471590, 0 committed:false 2016-04-07T02:15:25.352-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33471590, 33471590, 0 committed:false) SnapCreateDur 32.868µs SnapOpenDur 1.393059ms 2016-04-07T02:15:25.911-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33480000 QueuedCount 33 2016-04-07T02:15:26.539-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 36.11µs CommitTime 613.558228ms TotalFlushTime 24m9.301032408s TotalCommitTime 2m57.829218014s 2016-04-07T02:15:26.539-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33480033, 33480033, 264 committed:true 2016-04-07T02:15:26.539-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33480033, 33480033, 264 committed:true) SnapCreateDur 622.811643ms SnapOpenDur 60.775µs 2016-04-07T02:15:26.544-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:26.548-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:26.552-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:26.556-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:26.56-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:26.965-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33490000 QueuedCount 43890 2016-04-07T02:15:27.339-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33500000 QueuedCount 33890 2016-04-07T02:15:27.71-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33510000 QueuedCount 23890 2016-04-07T02:15:28.087-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33520000 QueuedCount 13890 2016-04-07T02:15:28.469-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33530000 QueuedCount 3890 2016-04-07T02:15:28.633-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33533890, 33533890, 0 committed:false 2016-04-07T02:15:28.633-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33533890, 33533890, 0 committed:false) SnapCreateDur 35.572µs SnapOpenDur 502.981µs 2016-04-07T02:15:28.637-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:28.64-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:28.646-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:28.65-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:28.654-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:29.014-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33540000 QueuedCount 46679 2016-04-07T02:15:29.411-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33550000 QueuedCount 36679 2016-04-07T02:15:29.809-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33560000 QueuedCount 26679 2016-04-07T02:15:30.154-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33570000 QueuedCount 16679 2016-04-07T02:15:30.549-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33580000 QueuedCount 6679 2016-04-07T02:15:30.862-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33586679, 33586679, 0 committed:false 2016-04-07T02:15:30.862-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33586679, 33586679, 0 committed:false) SnapCreateDur 66.525µs SnapOpenDur 188.53µs 2016-04-07T02:15:31.072-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33590000 QueuedCount 5630 2016-04-07T02:15:31.852-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.364µs CommitTime 527.27337ms TotalFlushTime 24m13.99508001s TotalCommitTime 2m58.356491384s 2016-04-07T02:15:31.852-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33595630, 33595630, 265 committed:true 2016-04-07T02:15:31.852-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33595630, 33595630, 265 committed:true) SnapCreateDur 535.829295ms SnapOpenDur 63.27µs 2016-04-07T02:15:31.857-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:31.861-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:31.866-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:31.87-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:31.875-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:32.196-07:00 [Info] Indexer::ReadMemstats Time Taken 17.244039ms 2016-04-07T02:15:32.199-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33600000 QueuedCount 48946 2016-04-07T02:15:32.579-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33610000 QueuedCount 38946 2016-04-07T02:15:32.913-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33620000 QueuedCount 28946 2016-04-07T02:15:33.299-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33630000 QueuedCount 18946 2016-04-07T02:15:33.686-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33640000 QueuedCount 8946 2016-04-07T02:15:34.096-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33648946, 33648946, 0 committed:false 2016-04-07T02:15:34.096-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33648946, 33648946, 0 committed:false) SnapCreateDur 32.881µs SnapOpenDur 129.34µs 2016-04-07T02:15:34.101-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:34.106-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:34.11-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:34.115-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:34.118-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33650000 QueuedCount 39992 2016-04-07T02:15:34.119-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:34.553-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33660000 QueuedCount 41454 2016-04-07T02:15:34.94-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33670000 QueuedCount 31454 2016-04-07T02:15:35.33-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33680000 QueuedCount 21454 2016-04-07T02:15:35.712-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33690000 QueuedCount 11454 2016-04-07T02:15:36.055-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33700000 QueuedCount 1454 2016-04-07T02:15:36.137-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33701454, 33701454, 0 committed:false 2016-04-07T02:15:36.139-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33701454, 33701454, 0 committed:false) SnapCreateDur 42.15µs SnapOpenDur 1.099578ms 2016-04-07T02:15:36.629-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33710000 QueuedCount 64 2016-04-07T02:15:37.247-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.712µs CommitTime 605.255872ms TotalFlushTime 24m18.688267902s TotalCommitTime 2m58.961747256s 2016-04-07T02:15:37.247-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33710064, 33710064, 266 committed:true 2016-04-07T02:15:37.247-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33710064, 33710064, 266 committed:true) SnapCreateDur 614.257113ms SnapOpenDur 60.887µs 2016-04-07T02:15:37.252-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:37.256-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:37.26-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:37.264-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:37.268-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:37.644-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33720000 QueuedCount 43825 2016-04-07T02:15:37.996-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33730000 QueuedCount 33825 2016-04-07T02:15:38.352-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33740000 QueuedCount 23825 2016-04-07T02:15:38.717-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33750000 QueuedCount 13825 2016-04-07T02:15:39.089-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33760000 QueuedCount 3825 2016-04-07T02:15:39.24-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33763825, 33763825, 0 committed:false 2016-04-07T02:15:39.24-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33763825, 33763825, 0 committed:false) SnapCreateDur 55.083µs SnapOpenDur 581.151µs 2016-04-07T02:15:39.244-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:39.248-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:39.252-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:39.256-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:39.261-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:39.63-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33770000 QueuedCount 47303 2016-04-07T02:15:39.971-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:9039842518, Disk:26274299904, ExtraSnapshotData:0, Fragmentation:65.59435436518035%, mode : circular 2016-04-07T02:15:39.971-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:15:40.018-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33780000 QueuedCount 37303 2016-04-07T02:15:40.404-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33790000 QueuedCount 27303 2016-04-07T02:15:40.73-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33800000 QueuedCount 17303 2016-04-07T02:15:41.126-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33810000 QueuedCount 7303 2016-04-07T02:15:41.45-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33817303, 33817303, 0 committed:false 2016-04-07T02:15:41.451-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33817303, 33817303, 0 committed:false) SnapCreateDur 34.85µs SnapOpenDur 760.114µs 2016-04-07T02:15:41.724-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33820000 QueuedCount 6047 2016-04-07T02:15:42.581-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.941µs CommitTime 585.480379ms TotalFlushTime 24m23.355794781s TotalCommitTime 2m59.547227635s 2016-04-07T02:15:42.581-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33826047, 33826047, 267 committed:true 2016-04-07T02:15:42.581-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33826047, 33826047, 267 committed:true) SnapCreateDur 595.176573ms SnapOpenDur 57.896µs 2016-04-07T02:15:42.585-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:42.591-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:42.597-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:42.602-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:42.608-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:42.864-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33830000 QueuedCount 49616 2016-04-07T02:15:43.239-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33840000 QueuedCount 39616 2016-04-07T02:15:43.615-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33850000 QueuedCount 29616 2016-04-07T02:15:44-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33860000 QueuedCount 19616 2016-04-07T02:15:44.382-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33870000 QueuedCount 9616 2016-04-07T02:15:44.77-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33879616, 33879616, 0 committed:false 2016-04-07T02:15:44.771-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33879616, 33879616, 0 committed:false) SnapCreateDur 28.323µs SnapOpenDur 392.352µs 2016-04-07T02:15:44.776-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:44.779-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33880000 QueuedCount 12149 2016-04-07T02:15:44.782-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:44.787-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:44.793-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:44.798-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:45.315-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33890000 QueuedCount 42995 2016-04-07T02:15:45.707-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33900000 QueuedCount 32995 2016-04-07T02:15:46.062-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33910000 QueuedCount 22995 2016-04-07T02:15:46.474-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33920000 QueuedCount 12995 2016-04-07T02:15:46.872-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33930000 QueuedCount 2995 2016-04-07T02:15:47.027-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33932995, 33932995, 0 committed:false 2016-04-07T02:15:47.027-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33932995, 33932995, 0 committed:false) SnapCreateDur 31.507µs SnapOpenDur 417.351µs 2016-04-07T02:15:47.531-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33940000 QueuedCount 2011 2016-04-07T02:15:48.284-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 22.966µs CommitTime 649.812355ms TotalFlushTime 24m28.301094212s TotalCommitTime 3m0.19703999s 2016-04-07T02:15:48.284-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33942011, 33942011, 268 committed:true 2016-04-07T02:15:48.284-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33942011, 33942011, 268 committed:true) SnapCreateDur 660.187784ms SnapOpenDur 39.668µs 2016-04-07T02:15:48.291-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:48.296-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:48.303-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:48.309-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:48.315-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:48.725-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33950000 QueuedCount 44676 2016-04-07T02:15:49.111-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33960000 QueuedCount 34676 2016-04-07T02:15:49.5-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33970000 QueuedCount 24676 2016-04-07T02:15:49.895-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33980000 QueuedCount 14676 2016-04-07T02:15:50.3-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 33990000 QueuedCount 4676 2016-04-07T02:15:50.488-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 33994676, 33994676, 0 committed:false 2016-04-07T02:15:50.489-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 33994676, 33994676, 0 committed:false) SnapCreateDur 32.622µs SnapOpenDur 1.216751ms 2016-04-07T02:15:50.496-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:50.502-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:50.508-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:50.514-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:50.519-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:50.86-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34000000 QueuedCount 47769 2016-04-07T02:15:51.269-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34010000 QueuedCount 37769 2016-04-07T02:15:51.673-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34020000 QueuedCount 27769 2016-04-07T02:15:52.006-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34030000 QueuedCount 17769 2016-04-07T02:15:52.395-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34040000 QueuedCount 7769 2016-04-07T02:15:52.725-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34047769, 34047769, 0 committed:false 2016-04-07T02:15:52.727-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34047769, 34047769, 0 committed:false) SnapCreateDur 39.263µs SnapOpenDur 1.500436ms 2016-04-07T02:15:52.995-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34050000 QueuedCount 6260 2016-04-07T02:15:53.793-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.086µs CommitTime 550.595393ms TotalFlushTime 24m33.154988552s TotalCommitTime 3m0.747635383s 2016-04-07T02:15:53.793-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34056260, 34056260, 269 committed:true 2016-04-07T02:15:53.793-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34056260, 34056260, 269 committed:true) SnapCreateDur 559.68428ms SnapOpenDur 60.278µs 2016-04-07T02:15:53.799-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:53.806-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:53.812-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:53.818-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:53.824-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:54.081-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34060000 QueuedCount 49957 2016-04-07T02:15:54.46-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34070000 QueuedCount 39957 2016-04-07T02:15:54.838-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34080000 QueuedCount 29957 2016-04-07T02:15:55.215-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34090000 QueuedCount 19957 2016-04-07T02:15:55.601-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34100000 QueuedCount 9957 2016-04-07T02:15:56.001-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34109957, 34109957, 0 committed:false 2016-04-07T02:15:56.001-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34109957, 34109957, 0 committed:false) SnapCreateDur 57.013µs SnapOpenDur 530.733µs 2016-04-07T02:15:56.003-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34110000 QueuedCount 1310 2016-04-07T02:15:56.008-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:56.014-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:56.02-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:56.026-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:56.032-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:56.489-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34120000 QueuedCount 42641 2016-04-07T02:15:56.823-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34130000 QueuedCount 32641 2016-04-07T02:15:57.21-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34140000 QueuedCount 22641 2016-04-07T02:15:57.608-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34150000 QueuedCount 12641 2016-04-07T02:15:58.032-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34160000 QueuedCount 2641 2016-04-07T02:15:58.129-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34162641, 34162641, 0 committed:false 2016-04-07T02:15:58.131-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34162641, 34162641, 0 committed:false) SnapCreateDur 62.972µs SnapOpenDur 1.663371ms 2016-04-07T02:15:58.613-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34170000 QueuedCount 1690 2016-04-07T02:15:59.249-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.848µs CommitTime 542.362192ms TotalFlushTime 24m37.957088236s TotalCommitTime 3m1.289997575s 2016-04-07T02:15:59.249-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34171690, 34171690, 270 committed:true 2016-04-07T02:15:59.249-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34171690, 34171690, 270 committed:true) SnapCreateDur 551.396827ms SnapOpenDur 44.048µs 2016-04-07T02:15:59.255-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:59.261-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:59.267-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:59.273-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:59.279-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:15:59.726-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34180000 QueuedCount 46294 2016-04-07T02:16:00.069-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34190000 QueuedCount 36294 2016-04-07T02:16:00.46-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34200000 QueuedCount 26294 2016-04-07T02:16:00.844-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34210000 QueuedCount 16294 2016-04-07T02:16:01.245-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34220000 QueuedCount 6294 2016-04-07T02:16:01.492-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34226294, 34226294, 0 committed:false 2016-04-07T02:16:01.493-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34226294, 34226294, 0 committed:false) SnapCreateDur 77.526µs SnapOpenDur 1.440711ms 2016-04-07T02:16:01.499-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:01.505-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:01.51-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:01.515-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:01.52-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:01.792-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34230000 QueuedCount 49544 2016-04-07T02:16:02.217-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34240000 QueuedCount 39544 2016-04-07T02:16:02.61-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34250000 QueuedCount 29544 2016-04-07T02:16:03.007-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34260000 QueuedCount 19544 2016-04-07T02:16:03.341-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34270000 QueuedCount 9544 2016-04-07T02:16:03.738-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34279544, 34279544, 0 committed:false 2016-04-07T02:16:03.74-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34279544, 34279544, 0 committed:false) SnapCreateDur 54.76µs SnapOpenDur 1.467706ms 2016-04-07T02:16:03.747-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34280000 QueuedCount 8003 2016-04-07T02:16:05.045-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 22.163µs CommitTime 771.504802ms TotalFlushTime 24m42.87239927s TotalCommitTime 3m2.061502377s 2016-04-07T02:16:05.045-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34288003, 34288003, 271 committed:true 2016-04-07T02:16:05.045-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34288003, 34288003, 271 committed:true) SnapCreateDur 780.432312ms SnapOpenDur 75.487µs 2016-04-07T02:16:05.062-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:05.072-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:05.083-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:05.092-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:05.094-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34290000 QueuedCount 32734 2016-04-07T02:16:05.1-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:06.011-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34300000 QueuedCount 33748 2016-04-07T02:16:06.371-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34310000 QueuedCount 23748 2016-04-07T02:16:06.762-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34320000 QueuedCount 13748 2016-04-07T02:16:07.141-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34330000 QueuedCount 3748 2016-04-07T02:16:07.3-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34333748, 34333748, 0 committed:false 2016-04-07T02:16:07.301-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34333748, 34333748, 0 committed:false) SnapCreateDur 57.139µs SnapOpenDur 811.408µs 2016-04-07T02:16:07.308-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:07.313-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:07.318-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:07.323-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:07.328-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:07.624-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34340000 QueuedCount 49726 2016-04-07T02:16:08.039-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34350000 QueuedCount 39726 2016-04-07T02:16:08.437-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34360000 QueuedCount 29726 2016-04-07T02:16:08.838-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34370000 QueuedCount 19726 2016-04-07T02:16:09.173-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34380000 QueuedCount 9726 2016-04-07T02:16:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:16:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:16:09.374-07:00 [Info] client load stats {} 2016-04-07T02:16:09.581-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34389726, 34389726, 0 committed:false 2016-04-07T02:16:09.582-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34389726, 34389726, 0 committed:false) SnapCreateDur 55.424µs SnapOpenDur 1.174915ms 2016-04-07T02:16:09.588-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34390000 QueuedCount 8747 2016-04-07T02:16:09.66-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":5601253,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":2127342100,"bucket-1:myindex:avg_ts_items_count":46387,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":9188829297,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":26274299904,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":8320,"bucket-1:myindex:frag_percent":65,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":3606015692,"bucket-1:myindex:items_count":34288003,"bucket-1:myindex:num_commits":271,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":34390427,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":34381046,"bucket-1:myindex:num_docs_queued":5618954,"bucket-1:myindex:num_flush_queued":34398747,"bucket-1:myindex:num_items_flushed":34390427,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":542,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":2281723303,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"271 182061502377 7952847824884981973","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"34390428 182791290539 1662824177893154297","bucket-1:myindex:timings/storage_info":"1627 11873024 167413285940","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"270 96197810 38102273285142","bucket-1:myindex:timings/storage_meta_set":"271 97994683 38553414179601","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"271 11316797 522562368341","bucket-1:myindex:timings/storage_set":"68780855 1286758416824 -8278030222270894114","bucket-1:myindex:timings/storage_snapshot_close":"270 201281399 428288477941887","bucket-1:myindex:timings/storage_snapshot_create":"271 240815571 307788936547503","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":40000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"427 711824253 28250515799947123","bucket-1:ts_queue_size":634,"indexer_state":"Active","memory_quota":31457280000,"memory_used":24442703872,"memory_used_storage":18874368000,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"4948 912507192 375761412566128","uptime":"47m1.353601926s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:16:09.68-07:00 [Info] memstats {"Alloc":3087349992, "TotalAlloc":47032581552, "Sys":9855607112, "Lookups":9794, "Mallocs":1350215000,"Frees":1272682646, "HeapAlloc":3087349992, "HeapSys":9382166528, "HeapIdle":4112539648, "HeapInuse":5269626880,"HeapReleased":1959968768, "HeapObjects":77532354,"GCSys":298708992, "LastGC":1460020444345319334,"PauseTotalNs":58052255, "PauseNs":[], "NumGC":43} 2016-04-07T02:16:09.972-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:9202892536, Disk:26274299904, ExtraSnapshotData:0, Fragmentation:64.97378590628422%, mode : circular 2016-04-07T02:16:09.972-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:16:10.599-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.116µs CommitTime 503.102411ms TotalFlushTime 24m47.827650762s TotalCommitTime 3m2.564604788s 2016-04-07T02:16:10.599-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34398747, 34398747, 272 committed:true 2016-04-07T02:16:10.6-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34398747, 34398747, 272 committed:true) SnapCreateDur 511.715239ms SnapOpenDur 57.646µs 2016-04-07T02:16:10.605-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:10.61-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:10.615-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:10.62-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:10.625-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34400000 QueuedCount 43671 2016-04-07T02:16:10.626-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:11.135-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34410000 QueuedCount 42594 2016-04-07T02:16:11.492-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34420000 QueuedCount 32594 2016-04-07T02:16:11.86-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34430000 QueuedCount 22594 2016-04-07T02:16:12.236-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34440000 QueuedCount 12594 2016-04-07T02:16:12.609-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34450000 QueuedCount 2594 2016-04-07T02:16:12.699-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34452594, 34452594, 0 committed:false 2016-04-07T02:16:12.7-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34452594, 34452594, 0 committed:false) SnapCreateDur 44.267µs SnapOpenDur 527.929µs 2016-04-07T02:16:12.705-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:12.71-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:12.715-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:12.72-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:12.725-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:13.091-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34460000 QueuedCount 46610 2016-04-07T02:16:13.489-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34470000 QueuedCount 36610 2016-04-07T02:16:13.894-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34480000 QueuedCount 26610 2016-04-07T02:16:14.304-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34490000 QueuedCount 16610 2016-04-07T02:16:14.7-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34500000 QueuedCount 6610 2016-04-07T02:16:14.954-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34506610, 34506610, 0 committed:false 2016-04-07T02:16:14.955-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34506610, 34506610, 0 committed:false) SnapCreateDur 41.702µs SnapOpenDur 1.093494ms 2016-04-07T02:16:15.218-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34510000 QueuedCount 5686 2016-04-07T02:16:16.042-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 31.205µs CommitTime 572.663512ms TotalFlushTime 24m52.59641114s TotalCommitTime 3m3.1372683s 2016-04-07T02:16:16.042-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34515686, 34515686, 273 committed:true 2016-04-07T02:16:16.042-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34515686, 34515686, 273 committed:true) SnapCreateDur 581.284639ms SnapOpenDur 70.727µs 2016-04-07T02:16:16.047-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:16.053-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:16.058-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:16.062-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:16.068-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:16.382-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34520000 QueuedCount 49126 2016-04-07T02:16:16.694-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34530000 QueuedCount 39126 2016-04-07T02:16:17.058-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34540000 QueuedCount 29126 2016-04-07T02:16:17.451-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34550000 QueuedCount 19126 2016-04-07T02:16:17.853-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34560000 QueuedCount 9126 2016-04-07T02:16:18.258-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34569126, 34569126, 0 committed:false 2016-04-07T02:16:18.259-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34569126, 34569126, 0 committed:false) SnapCreateDur 34.844µs SnapOpenDur 256.763µs 2016-04-07T02:16:18.264-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:18.269-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:18.274-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:18.278-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34570000 QueuedCount 34262 2016-04-07T02:16:18.279-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:18.284-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:18.816-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34580000 QueuedCount 42873 2016-04-07T02:16:19.201-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34590000 QueuedCount 32873 2016-04-07T02:16:19.591-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34600000 QueuedCount 22873 2016-04-07T02:16:19.986-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34610000 QueuedCount 12873 2016-04-07T02:16:20.35-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34620000 QueuedCount 2873 2016-04-07T02:16:20.523-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34622873, 34622873, 0 committed:false 2016-04-07T02:16:20.523-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34622873, 34622873, 0 committed:false) SnapCreateDur 69.486µs SnapOpenDur 709.25µs 2016-04-07T02:16:20.978-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34630000 QueuedCount 2018 2016-04-07T02:16:21.677-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.531µs CommitTime 598.646094ms TotalFlushTime 24m57.534862091s TotalCommitTime 3m3.735914394s 2016-04-07T02:16:21.677-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34632018, 34632018, 274 committed:true 2016-04-07T02:16:21.677-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34632018, 34632018, 274 committed:true) SnapCreateDur 608.448112ms SnapOpenDur 80.341µs 2016-04-07T02:16:21.683-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:21.688-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:21.693-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:21.698-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:21.703-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:22.094-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34640000 QueuedCount 45296 2016-04-07T02:16:22.457-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34650000 QueuedCount 35296 2016-04-07T02:16:22.832-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34660000 QueuedCount 25296 2016-04-07T02:16:23.205-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34670000 QueuedCount 15296 2016-04-07T02:16:23.586-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34680000 QueuedCount 5296 2016-04-07T02:16:23.823-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34685296, 34685296, 0 committed:false 2016-04-07T02:16:23.823-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34685296, 34685296, 0 committed:false) SnapCreateDur 31.352µs SnapOpenDur 108.329µs 2016-04-07T02:16:23.829-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:23.834-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:23.838-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:23.843-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:23.848-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:24.13-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34690000 QueuedCount 48961 2016-04-07T02:16:24.525-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34700000 QueuedCount 38961 2016-04-07T02:16:24.921-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34710000 QueuedCount 28961 2016-04-07T02:16:25.257-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34720000 QueuedCount 18961 2016-04-07T02:16:25.659-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34730000 QueuedCount 8961 2016-04-07T02:16:26.076-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34738961, 34738961, 0 committed:false 2016-04-07T02:16:26.076-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34738961, 34738961, 0 committed:false) SnapCreateDur 52.551µs SnapOpenDur 410.465µs 2016-04-07T02:16:26.089-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34740000 QueuedCount 7968 2016-04-07T02:16:27.155-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.802µs CommitTime 634.878383ms TotalFlushTime 25m2.278195408s TotalCommitTime 3m4.370792777s 2016-04-07T02:16:27.155-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34747968, 34747968, 275 committed:true 2016-04-07T02:16:27.155-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34747968, 34747968, 275 committed:true) SnapCreateDur 643.980639ms SnapOpenDur 52.322µs 2016-04-07T02:16:27.16-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:27.165-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:27.17-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:27.174-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:27.178-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:27.188-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34750000 QueuedCount 49205 2016-04-07T02:16:27.701-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34760000 QueuedCount 39205 2016-04-07T02:16:28.077-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34770000 QueuedCount 29205 2016-04-07T02:16:28.446-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34780000 QueuedCount 19205 2016-04-07T02:16:28.814-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34790000 QueuedCount 9205 2016-04-07T02:16:29.182-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34799205, 34799205, 0 committed:false 2016-04-07T02:16:29.183-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34799205, 34799205, 0 committed:false) SnapCreateDur 31.386µs SnapOpenDur 120.426µs 2016-04-07T02:16:29.187-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:29.191-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:29.195-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:29.199-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34800000 QueuedCount 34194 2016-04-07T02:16:29.199-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:29.204-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:29.656-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34810000 QueuedCount 42632 2016-04-07T02:16:30.06-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34820000 QueuedCount 32632 2016-04-07T02:16:30.456-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34830000 QueuedCount 22632 2016-04-07T02:16:30.792-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34840000 QueuedCount 12632 2016-04-07T02:16:31.196-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34850000 QueuedCount 2632 2016-04-07T02:16:31.356-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34852632, 34852632, 0 committed:false 2016-04-07T02:16:31.357-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34852632, 34852632, 0 committed:false) SnapCreateDur 54.731µs SnapOpenDur 297.031µs 2016-04-07T02:16:31.822-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34860000 QueuedCount 1709 2016-04-07T02:16:32.508-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.904µs CommitTime 586.867292ms TotalFlushTime 25m6.960612507s TotalCommitTime 3m4.957660069s 2016-04-07T02:16:32.508-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34861709, 34861709, 276 committed:true 2016-04-07T02:16:32.508-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34861709, 34861709, 276 committed:true) SnapCreateDur 596.724022ms SnapOpenDur 56.013µs 2016-04-07T02:16:32.513-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:32.517-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:32.522-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:32.526-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:32.53-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:32.828-07:00 [Info] Indexer::ReadMemstats Time Taken 16.348986ms 2016-04-07T02:16:32.99-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34870000 QueuedCount 44523 2016-04-07T02:16:33.32-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34880000 QueuedCount 34523 2016-04-07T02:16:33.727-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34890000 QueuedCount 24523 2016-04-07T02:16:34.118-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34900000 QueuedCount 14523 2016-04-07T02:16:34.505-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34910000 QueuedCount 4523 2016-04-07T02:16:34.671-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34914523, 34914523, 0 committed:false 2016-04-07T02:16:34.673-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34914523, 34914523, 0 committed:false) SnapCreateDur 57.609µs SnapOpenDur 1.446714ms 2016-04-07T02:16:34.678-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:34.682-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:34.686-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:34.69-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:34.694-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:35.031-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34920000 QueuedCount 48786 2016-04-07T02:16:35.444-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34930000 QueuedCount 38786 2016-04-07T02:16:35.853-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34940000 QueuedCount 28786 2016-04-07T02:16:36.259-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34950000 QueuedCount 18786 2016-04-07T02:16:36.591-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34960000 QueuedCount 8786 2016-04-07T02:16:36.984-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34968786, 34968786, 0 committed:false 2016-04-07T02:16:36.984-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34968786, 34968786, 0 committed:false) SnapCreateDur 30.404µs SnapOpenDur 460.002µs 2016-04-07T02:16:36.998-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34970000 QueuedCount 7303 2016-04-07T02:16:38.146-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.224µs CommitTime 588.565444ms TotalFlushTime 25m11.928211075s TotalCommitTime 3m5.546225513s 2016-04-07T02:16:38.146-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 34977303, 34977303, 277 committed:true 2016-04-07T02:16:38.146-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 34977303, 34977303, 277 committed:true) SnapCreateDur 597.191658ms SnapOpenDur 49.216µs 2016-04-07T02:16:38.151-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:38.155-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:38.16-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:38.164-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:38.168-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:38.414-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34980000 QueuedCount 50000 2016-04-07T02:16:38.771-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 34990000 QueuedCount 40320 2016-04-07T02:16:39.133-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35000000 QueuedCount 30320 2016-04-07T02:16:39.446-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35010000 QueuedCount 20320 2016-04-07T02:16:39.822-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35020000 QueuedCount 10320 2016-04-07T02:16:39.973-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:9368612518, Disk:26274299904, ExtraSnapshotData:0, Fragmentation:64.34305556292398%, mode : circular 2016-04-07T02:16:39.973-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:16:40.225-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35030000 QueuedCount 320 2016-04-07T02:16:40.228-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35030320, 35030320, 0 committed:false 2016-04-07T02:16:40.229-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35030320, 35030320, 0 committed:false) SnapCreateDur 29.399µs SnapOpenDur 1.575586ms 2016-04-07T02:16:40.234-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:40.239-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:40.243-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:40.247-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:40.251-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:40.702-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35040000 QueuedCount 43795 2016-04-07T02:16:41.095-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35050000 QueuedCount 33795 2016-04-07T02:16:41.489-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35060000 QueuedCount 23795 2016-04-07T02:16:41.895-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35070000 QueuedCount 13795 2016-04-07T02:16:42.307-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35080000 QueuedCount 3795 2016-04-07T02:16:42.413-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35083795, 35083795, 0 committed:false 2016-04-07T02:16:42.414-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35083795, 35083795, 0 committed:false) SnapCreateDur 58.484µs SnapOpenDur 1.847941ms 2016-04-07T02:16:42.82-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35090000 QueuedCount 2748 2016-04-07T02:16:43.493-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 16.588µs CommitTime 565.616416ms TotalFlushTime 25m16.627329821s TotalCommitTime 3m6.111841929s 2016-04-07T02:16:43.493-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35092748, 35092748, 278 committed:true 2016-04-07T02:16:43.494-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35092748, 35092748, 278 committed:true) SnapCreateDur 574.705402ms SnapOpenDur 52.815µs 2016-04-07T02:16:43.498-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:43.502-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:43.506-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:43.51-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:43.514-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:43.874-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35100000 QueuedCount 46022 2016-04-07T02:16:44.258-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35110000 QueuedCount 36022 2016-04-07T02:16:44.634-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35120000 QueuedCount 26022 2016-04-07T02:16:45.009-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35130000 QueuedCount 16022 2016-04-07T02:16:45.389-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35140000 QueuedCount 6022 2016-04-07T02:16:45.624-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35146022, 35146022, 0 committed:false 2016-04-07T02:16:45.625-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35146022, 35146022, 0 committed:false) SnapCreateDur 31.155µs SnapOpenDur 110.362µs 2016-04-07T02:16:45.629-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:45.633-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:45.636-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:45.64-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:45.645-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:45.797-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35150000 QueuedCount 49105 2016-04-07T02:16:46.181-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35160000 QueuedCount 39105 2016-04-07T02:16:46.589-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35170000 QueuedCount 29105 2016-04-07T02:16:46.971-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35180000 QueuedCount 19105 2016-04-07T02:16:47.371-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35190000 QueuedCount 9105 2016-04-07T02:16:47.698-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35199105, 35199105, 0 committed:false 2016-04-07T02:16:47.7-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35199105, 35199105, 0 committed:false) SnapCreateDur 56.622µs SnapOpenDur 1.799744ms 2016-04-07T02:16:47.943-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35200000 QueuedCount 7815 2016-04-07T02:16:48.815-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 23.476µs CommitTime 531.833405ms TotalFlushTime 25m21.330829986s TotalCommitTime 3m6.643675334s 2016-04-07T02:16:48.815-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35207815, 35207815, 279 committed:true 2016-04-07T02:16:48.815-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35207815, 35207815, 279 committed:true) SnapCreateDur 541.207796ms SnapOpenDur 53.947µs 2016-04-07T02:16:48.819-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:48.823-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:48.827-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:48.831-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:48.835-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:48.967-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35210000 QueuedCount 50000 2016-04-07T02:16:49.269-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35220000 QueuedCount 40566 2016-04-07T02:16:49.629-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35230000 QueuedCount 30566 2016-04-07T02:16:49.996-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35240000 QueuedCount 20566 2016-04-07T02:16:50.367-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35250000 QueuedCount 10566 2016-04-07T02:16:50.749-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35260000 QueuedCount 566 2016-04-07T02:16:50.763-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35260566, 35260566, 0 committed:false 2016-04-07T02:16:50.765-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35260566, 35260566, 0 committed:false) SnapCreateDur 30.104µs SnapOpenDur 1.354552ms 2016-04-07T02:16:50.766-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:50.768-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:50.77-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:50.773-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:50.778-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:51.28-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35270000 QueuedCount 20700 2016-04-07T02:16:51.664-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35280000 QueuedCount 10700 2016-04-07T02:16:52.076-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35290000 QueuedCount 700 2016-04-07T02:16:52.089-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35290700, 35290700, 0 committed:false 2016-04-07T02:16:52.09-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35290700, 35290700, 0 committed:false) SnapCreateDur 29.266µs SnapOpenDur 809.858µs 2016-04-07T02:16:52.108-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:52.116-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:52.692-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35300000 QueuedCount 50000 2016-04-07T02:16:52.698-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:53.084-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:53.087-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35310000 QueuedCount 50000 2016-04-07T02:16:53.548-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35320000 QueuedCount 48521 2016-04-07T02:16:53.998-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35330000 QueuedCount 38521 2016-04-07T02:16:54.421-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35340000 QueuedCount 28521 2016-04-07T02:16:54.822-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35350000 QueuedCount 18521 2016-04-07T02:16:55.224-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35360000 QueuedCount 8521 2016-04-07T02:16:56.383-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 38.171µs CommitTime 812.212513ms TotalFlushTime 25m27.950925148s TotalCommitTime 3m7.455887847s 2016-04-07T02:16:56.383-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35368521, 35368521, 280 committed:true 2016-04-07T02:16:56.383-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35368521, 35368521, 280 committed:true) SnapCreateDur 820.941932ms SnapOpenDur 47.51µs 2016-04-07T02:16:56.388-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:56.392-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:56.397-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:56.401-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:56.405-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:56.407-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35370000 QueuedCount 48287 2016-04-07T02:16:56.902-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35380000 QueuedCount 42663 2016-04-07T02:16:57.27-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35390000 QueuedCount 32663 2016-04-07T02:16:57.64-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35400000 QueuedCount 22663 2016-04-07T02:16:58.041-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35410000 QueuedCount 12663 2016-04-07T02:16:58.448-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35420000 QueuedCount 2663 2016-04-07T02:16:58.54-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35422663, 35422663, 0 committed:false 2016-04-07T02:16:58.541-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35422663, 35422663, 0 committed:false) SnapCreateDur 32.905µs SnapOpenDur 840.737µs 2016-04-07T02:16:58.905-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35430000 QueuedCount 1434 2016-04-07T02:16:59.334-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.862µs CommitTime 398.502033ms TotalFlushTime 25m30.445019179s TotalCommitTime 3m7.85438988s 2016-04-07T02:16:59.334-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35431434, 35431434, 281 committed:true 2016-04-07T02:16:59.334-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35431434, 35431434, 281 committed:true) SnapCreateDur 408.505276ms SnapOpenDur 73.384µs 2016-04-07T02:16:59.339-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:59.344-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:59.348-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:59.353-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:59.357-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:16:59.715-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35440000 QueuedCount 45129 2016-04-07T02:17:00.106-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35450000 QueuedCount 35129 2016-04-07T02:17:00.435-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35460000 QueuedCount 25129 2016-04-07T02:17:00.839-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35470000 QueuedCount 15129 2016-04-07T02:17:01.23-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35480000 QueuedCount 5129 2016-04-07T02:17:01.467-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35485129, 35485129, 0 committed:false 2016-04-07T02:17:01.468-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35485129, 35485129, 0 committed:false) SnapCreateDur 31.024µs SnapOpenDur 454.624µs 2016-04-07T02:17:01.473-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:01.477-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:01.481-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:01.485-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:01.489-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:01.704-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35490000 QueuedCount 46948 2016-04-07T02:17:02.077-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35500000 QueuedCount 36948 2016-04-07T02:17:02.458-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35510000 QueuedCount 26948 2016-04-07T02:17:02.838-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35520000 QueuedCount 16948 2016-04-07T02:17:03.229-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35530000 QueuedCount 6948 2016-04-07T02:17:03.482-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35536948, 35536948, 0 committed:false 2016-04-07T02:17:03.483-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35536948, 35536948, 0 committed:false) SnapCreateDur 30.742µs SnapOpenDur 907.2µs 2016-04-07T02:17:03.487-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:03.491-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:03.495-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:03.499-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:03.503-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:03.751-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35540000 QueuedCount 49493 2016-04-07T02:17:04.169-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35550000 QueuedCount 39493 2016-04-07T02:17:04.577-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35560000 QueuedCount 29493 2016-04-07T02:17:04.994-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35570000 QueuedCount 19493 2016-04-07T02:17:05.41-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35580000 QueuedCount 9493 2016-04-07T02:17:05.826-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35589493, 35589493, 0 committed:false 2016-04-07T02:17:05.826-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35589493, 35589493, 0 committed:false) SnapCreateDur 46.796µs SnapOpenDur 335.848µs 2016-04-07T02:17:05.834-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35590000 QueuedCount 8277 2016-04-07T02:17:06.991-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 30.669µs CommitTime 631.419858ms TotalFlushTime 25m37.343880835s TotalCommitTime 3m8.485809738s 2016-04-07T02:17:06.991-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35598277, 35598277, 282 committed:true 2016-04-07T02:17:06.991-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35598277, 35598277, 282 committed:true) SnapCreateDur 640.192989ms SnapOpenDur 46.562µs 2016-04-07T02:17:06.996-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:07-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:07.004-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:07.008-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:07.012-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:07.018-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35600000 QueuedCount 50000 2016-04-07T02:17:07.556-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35610000 QueuedCount 40054 2016-04-07T02:17:07.935-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35620000 QueuedCount 30054 2016-04-07T02:17:08.339-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35630000 QueuedCount 20054 2016-04-07T02:17:08.726-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35640000 QueuedCount 10054 2016-04-07T02:17:09.118-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35650000 QueuedCount 54 2016-04-07T02:17:09.119-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35650054, 35650054, 0 committed:false 2016-04-07T02:17:09.119-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35650054, 35650054, 0 committed:false) SnapCreateDur 30.308µs SnapOpenDur 504.899µs 2016-04-07T02:17:09.123-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:09.127-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:09.131-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:09.135-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:09.139-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:17:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:17:09.374-07:00 [Info] client load stats {} 2016-04-07T02:17:09.635-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35660000 QueuedCount 42043 2016-04-07T02:17:09.661-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":4297957,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":1892832083,"bucket-1:myindex:avg_ts_items_count":40570,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":9525964157,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":26274299904,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":40263,"bucket-1:myindex:frag_percent":63,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":3740185050,"bucket-1:myindex:items_count":35598277,"bucket-1:myindex:num_commits":282,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":35661780,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":35641419,"bucket-1:myindex:num_docs_queued":4358581,"bucket-1:myindex:num_flush_queued":35702043,"bucket-1:myindex:num_items_flushed":35661780,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":574,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":2128087514,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"282 188485809738 -6640617733163673150","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"35661781 187695507426 1662845270542387904","bucket-1:myindex:timings/storage_info":"1713 12454673 174408150205","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"281 100939824 41135505817270","bucket-1:myindex:timings/storage_meta_set":"282 102881137 40794941030937","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"282 11775699 543174651909","bucket-1:myindex:timings/storage_set":"71323560 1333896489401 -4984676748014463125","bucket-1:myindex:timings/storage_snapshot_close":"291 215068151 452121475522313","bucket-1:myindex:timings/storage_snapshot_create":"292 255385512 324000408003448","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":40000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"439 721683934 28258700656640068","bucket-1:ts_queue_size":488,"indexer_state":"Active","memory_quota":31457280000,"memory_used":23860228096,"memory_used_storage":18874368000,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"5091 942888223 382402489207991","uptime":"48m1.354008265s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:17:09.681-07:00 [Info] memstats {"Alloc":2093061992, "TotalAlloc":47376616456, "Sys":9855607112, "Lookups":9992, "Mallocs":1361828543,"Frees":1308019995, "HeapAlloc":2093061992, "HeapSys":9382330368, "HeapIdle":4695179264, "HeapInuse":4687151104,"HeapReleased":1959968768, "HeapObjects":53808548,"GCSys":298708992, "LastGC":1460020565170640255,"PauseTotalNs":60022913, "PauseNs":[1970658], "NumGC":44} 2016-04-07T02:17:09.973-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:9538938409, Disk:26274299904, ExtraSnapshotData:0, Fragmentation:63.6947951273564%, mode : circular 2016-04-07T02:17:09.974-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:17:10.029-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35670000 QueuedCount 32043 2016-04-07T02:17:10.363-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35680000 QueuedCount 22043 2016-04-07T02:17:10.751-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35690000 QueuedCount 12043 2016-04-07T02:17:11.145-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35700000 QueuedCount 2043 2016-04-07T02:17:11.228-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35702043, 35702043, 0 committed:false 2016-04-07T02:17:11.229-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35702043, 35702043, 0 committed:false) SnapCreateDur 28.407µs SnapOpenDur 1.048254ms 2016-04-07T02:17:11.725-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35710000 QueuedCount 722 2016-04-07T02:17:12.352-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.39µs CommitTime 601.225306ms TotalFlushTime 25m42.021662352s TotalCommitTime 3m9.087035044s 2016-04-07T02:17:12.352-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35710722, 35710722, 283 committed:true 2016-04-07T02:17:12.352-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35710722, 35710722, 283 committed:true) SnapCreateDur 609.859439ms SnapOpenDur 34.489µs 2016-04-07T02:17:12.356-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:12.36-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:12.364-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:12.368-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:12.372-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:12.837-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35720000 QueuedCount 43669 2016-04-07T02:17:13.202-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35730000 QueuedCount 33669 2016-04-07T02:17:13.571-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35740000 QueuedCount 23669 2016-04-07T02:17:13.947-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35750000 QueuedCount 13669 2016-04-07T02:17:14.358-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35760000 QueuedCount 3669 2016-04-07T02:17:14.465-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35763669, 35763669, 0 committed:false 2016-04-07T02:17:14.467-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35763669, 35763669, 0 committed:false) SnapCreateDur 31.463µs SnapOpenDur 1.581098ms 2016-04-07T02:17:14.471-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:14.475-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:14.48-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:14.484-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:14.488-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:14.844-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35770000 QueuedCount 46741 2016-04-07T02:17:15.23-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35780000 QueuedCount 36741 2016-04-07T02:17:15.62-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35790000 QueuedCount 26741 2016-04-07T02:17:16.014-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35800000 QueuedCount 16741 2016-04-07T02:17:16.412-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35810000 QueuedCount 6741 2016-04-07T02:17:16.659-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35816741, 35816741, 0 committed:false 2016-04-07T02:17:16.66-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35816741, 35816741, 0 committed:false) SnapCreateDur 95.061µs SnapOpenDur 1.387024ms 2016-04-07T02:17:16.921-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35820000 QueuedCount 5619 2016-04-07T02:17:17.748-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.289µs CommitTime 575.027602ms TotalFlushTime 25m46.759335971s TotalCommitTime 3m9.662062646s 2016-04-07T02:17:17.748-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35825619, 35825619, 284 committed:true 2016-04-07T02:17:17.748-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35825619, 35825619, 284 committed:true) SnapCreateDur 584.064656ms SnapOpenDur 45.895µs 2016-04-07T02:17:17.753-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:17.756-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:17.76-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:17.764-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:17.768-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:18.075-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35830000 QueuedCount 48657 2016-04-07T02:17:18.479-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35840000 QueuedCount 38657 2016-04-07T02:17:18.82-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35850000 QueuedCount 28657 2016-04-07T02:17:19.226-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35860000 QueuedCount 18657 2016-04-07T02:17:19.616-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35870000 QueuedCount 8657 2016-04-07T02:17:19.943-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35878657, 35878657, 0 committed:false 2016-04-07T02:17:19.946-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35878657, 35878657, 0 committed:false) SnapCreateDur 30.706µs SnapOpenDur 3.149659ms 2016-04-07T02:17:19.95-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:19.954-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:19.959-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:19.963-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:19.967-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:20.136-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35880000 QueuedCount 50000 2016-04-07T02:17:20.527-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35890000 QueuedCount 42118 2016-04-07T02:17:20.915-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35900000 QueuedCount 32118 2016-04-07T02:17:21.323-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35910000 QueuedCount 22118 2016-04-07T02:17:21.721-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35920000 QueuedCount 12118 2016-04-07T02:17:22.069-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35930000 QueuedCount 2118 2016-04-07T02:17:22.168-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35932118, 35932118, 0 committed:false 2016-04-07T02:17:22.17-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35932118, 35932118, 0 committed:false) SnapCreateDur 39.18µs SnapOpenDur 1.850056ms 2016-04-07T02:17:22.732-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35940000 QueuedCount 1022 2016-04-07T02:17:23.43-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.424µs CommitTime 616.230693ms TotalFlushTime 25m51.742762445s TotalCommitTime 3m10.278293339s 2016-04-07T02:17:23.43-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35941022, 35941022, 285 committed:true 2016-04-07T02:17:23.43-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35941022, 35941022, 285 committed:true) SnapCreateDur 625.161972ms SnapOpenDur 48.735µs 2016-04-07T02:17:23.435-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:23.439-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:23.444-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:23.449-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:23.453-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:23.858-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35950000 QueuedCount 44527 2016-04-07T02:17:24.253-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35960000 QueuedCount 34527 2016-04-07T02:17:24.65-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35970000 QueuedCount 24527 2016-04-07T02:17:25.039-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35980000 QueuedCount 14527 2016-04-07T02:17:25.376-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 35990000 QueuedCount 4527 2016-04-07T02:17:25.608-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 35994527, 35994527, 0 committed:false 2016-04-07T02:17:25.61-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 35994527, 35994527, 0 committed:false) SnapCreateDur 45.138µs SnapOpenDur 1.803946ms 2016-04-07T02:17:25.614-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:25.617-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:25.621-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:25.624-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:25.628-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:25.856-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36000000 QueuedCount 48024 2016-04-07T02:17:26.245-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36010000 QueuedCount 38024 2016-04-07T02:17:26.627-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36020000 QueuedCount 28024 2016-04-07T02:17:27.01-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36030000 QueuedCount 18024 2016-04-07T02:17:27.399-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36040000 QueuedCount 8024 2016-04-07T02:17:27.72-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36048024, 36048024, 0 committed:false 2016-04-07T02:17:27.72-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36048024, 36048024, 0 committed:false) SnapCreateDur 81.881µs SnapOpenDur 424.578µs 2016-04-07T02:17:27.975-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36050000 QueuedCount 7090 2016-04-07T02:17:28.806-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 31.251µs CommitTime 552.165605ms TotalFlushTime 25m56.480204001s TotalCommitTime 3m10.830458944s 2016-04-07T02:17:28.806-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36057090, 36057090, 286 committed:true 2016-04-07T02:17:28.806-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36057090, 36057090, 286 committed:true) SnapCreateDur 561.303256ms SnapOpenDur 65.37µs 2016-04-07T02:17:28.81-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:28.814-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:28.818-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:28.82-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:28.823-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:28.967-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36060000 QueuedCount 45049 2016-04-07T02:17:29.353-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36070000 QueuedCount 35049 2016-04-07T02:17:29.727-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36080000 QueuedCount 25049 2016-04-07T02:17:30.102-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36090000 QueuedCount 15049 2016-04-07T02:17:30.423-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36100000 QueuedCount 5049 2016-04-07T02:17:30.658-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36105049, 36105049, 0 committed:false 2016-04-07T02:17:30.659-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36105049, 36105049, 0 committed:false) SnapCreateDur 41.035µs SnapOpenDur 691.321µs 2016-04-07T02:17:30.662-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:30.664-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:30.668-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:30.671-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:30.675-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:30.942-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36110000 QueuedCount 43057 2016-04-07T02:17:31.345-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36120000 QueuedCount 33057 2016-04-07T02:17:31.729-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36130000 QueuedCount 23057 2016-04-07T02:17:32.132-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36140000 QueuedCount 13057 2016-04-07T02:17:32.531-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36150000 QueuedCount 3057 2016-04-07T02:17:32.628-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36153057, 36153057, 0 committed:false 2016-04-07T02:17:32.63-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36153057, 36153057, 0 committed:false) SnapCreateDur 27.897µs SnapOpenDur 1.480267ms 2016-04-07T02:17:32.634-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:32.637-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:32.64-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:32.644-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:32.647-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:33.083-07:00 [Info] Indexer::ReadMemstats Time Taken 17.256181ms 2016-04-07T02:17:33.183-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36160000 QueuedCount 45709 2016-04-07T02:17:33.582-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36170000 QueuedCount 35709 2016-04-07T02:17:33.982-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36180000 QueuedCount 25709 2016-04-07T02:17:34.34-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36190000 QueuedCount 15709 2016-04-07T02:17:34.756-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36200000 QueuedCount 5709 2016-04-07T02:17:35.009-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36205709, 36205709, 0 committed:false 2016-04-07T02:17:35.01-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36205709, 36205709, 0 committed:false) SnapCreateDur 54.283µs SnapOpenDur 974.254µs 2016-04-07T02:17:35.487-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36210000 QueuedCount 4844 2016-04-07T02:17:36.469-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.679µs CommitTime 736.246539ms TotalFlushTime 26m3.299022979s TotalCommitTime 3m11.566705483s 2016-04-07T02:17:36.469-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36214844, 36214844, 287 committed:true 2016-04-07T02:17:36.469-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36214844, 36214844, 287 committed:true) SnapCreateDur 745.098499ms SnapOpenDur 48.179µs 2016-04-07T02:17:36.473-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:36.476-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:36.48-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:36.483-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:36.487-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:36.844-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36220000 QueuedCount 47867 2016-04-07T02:17:37.23-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36230000 QueuedCount 37867 2016-04-07T02:17:37.622-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36240000 QueuedCount 27867 2016-04-07T02:17:38.025-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36250000 QueuedCount 17867 2016-04-07T02:17:38.451-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36260000 QueuedCount 7867 2016-04-07T02:17:38.737-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36267867, 36267867, 0 committed:false 2016-04-07T02:17:38.738-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36267867, 36267867, 0 committed:false) SnapCreateDur 34.179µs SnapOpenDur 1.657218ms 2016-04-07T02:17:38.742-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:38.745-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:38.749-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:38.752-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:38.756-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:38.938-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36270000 QueuedCount 50000 2016-04-07T02:17:39.339-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36280000 QueuedCount 40995 2016-04-07T02:17:39.731-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36290000 QueuedCount 30995 2016-04-07T02:17:39.975-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:9703722098, Disk:26274299904, ExtraSnapshotData:0, Fragmentation:63.0676283156732%, mode : circular 2016-04-07T02:17:39.975-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:17:40.129-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36300000 QueuedCount 20995 2016-04-07T02:17:40.546-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36310000 QueuedCount 10995 2016-04-07T02:17:40.952-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36320000 QueuedCount 995 2016-04-07T02:17:40.97-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36320995, 36320995, 0 committed:false 2016-04-07T02:17:40.972-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36320995, 36320995, 0 committed:false) SnapCreateDur 33.227µs SnapOpenDur 1.525438ms 2016-04-07T02:17:42.094-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.108µs CommitTime 539.031312ms TotalFlushTime 26m8.299105412s TotalCommitTime 3m12.105736795s 2016-04-07T02:17:42.094-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36329636, 36329636, 288 committed:true 2016-04-07T02:17:42.094-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36329636, 36329636, 288 committed:true) SnapCreateDur 548.236776ms SnapOpenDur 52.948µs 2016-04-07T02:17:42.098-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:42.1-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36330000 QueuedCount 13779 2016-04-07T02:17:42.101-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:42.105-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:42.108-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:42.112-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:42.565-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36340000 QueuedCount 42205 2016-04-07T02:17:42.888-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36350000 QueuedCount 32205 2016-04-07T02:17:43.267-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36360000 QueuedCount 22205 2016-04-07T02:17:43.641-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36370000 QueuedCount 12205 2016-04-07T02:17:44.016-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36380000 QueuedCount 2205 2016-04-07T02:17:44.098-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36382205, 36382205, 0 committed:false 2016-04-07T02:17:44.099-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36382205, 36382205, 0 committed:false) SnapCreateDur 43.518µs SnapOpenDur 1.119616ms 2016-04-07T02:17:44.103-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:44.106-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:44.11-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:44.113-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:44.117-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:44.485-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36390000 QueuedCount 45599 2016-04-07T02:17:44.898-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36400000 QueuedCount 35599 2016-04-07T02:17:45.32-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36410000 QueuedCount 25599 2016-04-07T02:17:45.712-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36420000 QueuedCount 15599 2016-04-07T02:17:46.047-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36430000 QueuedCount 5599 2016-04-07T02:17:46.288-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36435599, 36435599, 0 committed:false 2016-04-07T02:17:46.289-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36435599, 36435599, 0 committed:false) SnapCreateDur 57.821µs SnapOpenDur 1.355382ms 2016-04-07T02:17:46.583-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36440000 QueuedCount 4513 2016-04-07T02:17:47.359-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 22.004µs CommitTime 516.217519ms TotalFlushTime 26m12.969477124s TotalCommitTime 3m12.621954314s 2016-04-07T02:17:47.359-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36444513, 36444513, 289 committed:true 2016-04-07T02:17:47.36-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36444513, 36444513, 289 committed:true) SnapCreateDur 526.119566ms SnapOpenDur 57.841µs 2016-04-07T02:17:47.363-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:47.367-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:47.37-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:47.374-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:47.378-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:47.612-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36450000 QueuedCount 48063 2016-04-07T02:17:47.98-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36460000 QueuedCount 38063 2016-04-07T02:17:48.385-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36470000 QueuedCount 28063 2016-04-07T02:17:48.76-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36480000 QueuedCount 18063 2016-04-07T02:17:49.138-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36490000 QueuedCount 8063 2016-04-07T02:17:49.457-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36498063, 36498063, 0 committed:false 2016-04-07T02:17:49.457-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36498063, 36498063, 0 committed:false) SnapCreateDur 31.31µs SnapOpenDur 272.265µs 2016-04-07T02:17:49.461-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:49.463-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:49.466-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:49.469-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:49.471-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:49.604-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36500000 QueuedCount 40280 2016-04-07T02:17:49.952-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36510000 QueuedCount 30280 2016-04-07T02:17:50.371-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36520000 QueuedCount 20280 2016-04-07T02:17:50.763-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36530000 QueuedCount 10280 2016-04-07T02:17:51.156-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36540000 QueuedCount 280 2016-04-07T02:17:51.165-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36540280, 36540280, 0 committed:false 2016-04-07T02:17:51.166-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36540280, 36540280, 0 committed:false) SnapCreateDur 32.014µs SnapOpenDur 1.262651ms 2016-04-07T02:17:51.17-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:51.173-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:51.176-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:51.18-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:51.183-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:51.676-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36550000 QueuedCount 41242 2016-04-07T02:17:52.071-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36560000 QueuedCount 31242 2016-04-07T02:17:52.488-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36570000 QueuedCount 21242 2016-04-07T02:17:52.892-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36580000 QueuedCount 11242 2016-04-07T02:17:53.308-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36590000 QueuedCount 1242 2016-04-07T02:17:53.329-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36591242, 36591242, 0 committed:false 2016-04-07T02:17:53.33-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36591242, 36591242, 0 committed:false) SnapCreateDur 37.216µs SnapOpenDur 1.39954ms 2016-04-07T02:17:53.908-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36600000 QueuedCount 613 2016-04-07T02:17:54.669-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.793µs CommitTime 675.721769ms TotalFlushTime 26m19.48545338s TotalCommitTime 3m13.297676083s 2016-04-07T02:17:54.67-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36600613, 36600613, 290 committed:true 2016-04-07T02:17:54.67-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36600613, 36600613, 290 committed:true) SnapCreateDur 685.23479ms SnapOpenDur 71.317µs 2016-04-07T02:17:54.673-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:54.677-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:54.681-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:54.685-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:54.689-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:55.093-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36610000 QueuedCount 43617 2016-04-07T02:17:55.469-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36620000 QueuedCount 33617 2016-04-07T02:17:55.831-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36630000 QueuedCount 23617 2016-04-07T02:17:56.208-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36640000 QueuedCount 13617 2016-04-07T02:17:56.603-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36650000 QueuedCount 3617 2016-04-07T02:17:56.705-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36653617, 36653617, 0 committed:false 2016-04-07T02:17:56.708-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36653617, 36653617, 0 committed:false) SnapCreateDur 44.086µs SnapOpenDur 2.987662ms 2016-04-07T02:17:56.712-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:56.716-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:56.72-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:56.724-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:56.728-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:17:57.032-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36660000 QueuedCount 48144 2016-04-07T02:17:57.441-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36670000 QueuedCount 38144 2016-04-07T02:17:57.844-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36680000 QueuedCount 28144 2016-04-07T02:17:58.271-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36690000 QueuedCount 18144 2016-04-07T02:17:58.681-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36700000 QueuedCount 8144 2016-04-07T02:17:59.014-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36708144, 36708144, 0 committed:false 2016-04-07T02:17:59.015-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36708144, 36708144, 0 committed:false) SnapCreateDur 60.947µs SnapOpenDur 956.95µs 2016-04-07T02:17:59.3-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36710000 QueuedCount 7293 2016-04-07T02:18:00.181-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 31.134µs CommitTime 612.500514ms TotalFlushTime 26m24.297105223s TotalCommitTime 3m13.910176597s 2016-04-07T02:18:00.181-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36717293, 36717293, 291 committed:true 2016-04-07T02:18:00.181-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36717293, 36717293, 291 committed:true) SnapCreateDur 621.221528ms SnapOpenDur 47.84µs 2016-04-07T02:18:00.185-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:00.189-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:00.194-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:00.198-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:00.203-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:00.413-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36720000 QueuedCount 50000 2016-04-07T02:18:00.773-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36730000 QueuedCount 41994 2016-04-07T02:18:01.137-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36740000 QueuedCount 31994 2016-04-07T02:18:01.455-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36750000 QueuedCount 21994 2016-04-07T02:18:01.844-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36760000 QueuedCount 11994 2016-04-07T02:18:02.242-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36770000 QueuedCount 1994 2016-04-07T02:18:02.329-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36771994, 36771994, 0 committed:false 2016-04-07T02:18:02.33-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36771994, 36771994, 0 committed:false) SnapCreateDur 54.799µs SnapOpenDur 1.282236ms 2016-04-07T02:18:02.335-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:02.339-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:02.344-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:02.348-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:02.352-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:02.721-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36780000 QueuedCount 44906 2016-04-07T02:18:03.136-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36790000 QueuedCount 34906 2016-04-07T02:18:03.538-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36800000 QueuedCount 24906 2016-04-07T02:18:03.928-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36810000 QueuedCount 14906 2016-04-07T02:18:04.339-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36820000 QueuedCount 4906 2016-04-07T02:18:04.514-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36824906, 36824906, 0 committed:false 2016-04-07T02:18:04.515-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36824906, 36824906, 0 committed:false) SnapCreateDur 27.721µs SnapOpenDur 1.008764ms 2016-04-07T02:18:04.962-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36830000 QueuedCount 3935 2016-04-07T02:18:06.22-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.701µs CommitTime 1.152050863s TotalFlushTime 26m29.104641061s TotalCommitTime 3m15.06222746s 2016-04-07T02:18:06.22-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36833935, 36833935, 292 committed:true 2016-04-07T02:18:06.22-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36833935, 36833935, 292 committed:true) SnapCreateDur 1.160657614s SnapOpenDur 63.211µs 2016-04-07T02:18:06.225-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:06.229-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:06.234-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:06.24-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:06.244-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:06.486-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36840000 QueuedCount 47513 2016-04-07T02:18:06.855-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36850000 QueuedCount 37513 2016-04-07T02:18:07.241-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36860000 QueuedCount 27513 2016-04-07T02:18:07.621-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36870000 QueuedCount 17513 2016-04-07T02:18:08.006-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36880000 QueuedCount 7513 2016-04-07T02:18:08.327-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36887513, 36887513, 0 committed:false 2016-04-07T02:18:08.328-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36887513, 36887513, 0 committed:false) SnapCreateDur 27.084µs SnapOpenDur 411.044µs 2016-04-07T02:18:08.333-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:08.337-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:08.342-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:08.346-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:08.351-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:08.548-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36890000 QueuedCount 50000 2016-04-07T02:18:08.969-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36900000 QueuedCount 41274 2016-04-07T02:18:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:18:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:18:09.374-07:00 [Info] client load stats {} 2016-04-07T02:18:09.378-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36910000 QueuedCount 31274 2016-04-07T02:18:09.662-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":3058726,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":1988872613,"bucket-1:myindex:avg_ts_items_count":40848,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":9859081119,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":26274299904,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":23351,"bucket-1:myindex:frag_percent":62,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":3872940435,"bucket-1:myindex:items_count":36833935,"bucket-1:myindex:num_commits":292,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":36917923,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":36878507,"bucket-1:myindex:num_docs_queued":3121493,"bucket-1:myindex:num_flush_queued":36941274,"bucket-1:myindex:num_items_flushed":36917923,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":606,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":2107411993,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"292 195062227460 -2005789771068499284","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"36917924 192587432661 1662866119277943835","bucket-1:myindex:timings/storage_info":"1797 13037137 182424113549","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"291 105797151 44723154541489","bucket-1:myindex:timings/storage_meta_set":"292 107519011 43017571765697","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"292 12157066 558802365308","bucket-1:myindex:timings/storage_set":"73835846 1380922896455 -1545841180062955189","bucket-1:myindex:timings/storage_snapshot_close":"313 240792095 509967895252653","bucket-1:myindex:timings/storage_snapshot_create":"314 279403310 355138968725014","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":40000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"451 731834468 28267341646628992","bucket-1:ts_queue_size":346,"indexer_state":"Active","memory_quota":31457280000,"memory_used":24053493760,"memory_used_storage":18874368000,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"5235 975290516 390928179475938","uptime":"49m1.354948916s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:18:09.681-07:00 [Info] memstats {"Alloc":2440608736, "TotalAlloc":47724163200, "Sys":9855607112, "Lookups":10191, "Mallocs":1373621885,"Frees":1310880959, "HeapAlloc":2440608736, "HeapSys":9382166528, "HeapIdle":4501749760, "HeapInuse":4880416768,"HeapReleased":1959968768, "HeapObjects":62740926,"GCSys":298708992, "LastGC":1460020565170640255,"PauseTotalNs":60022913, "PauseNs":[], "NumGC":44} 2016-04-07T02:18:09.769-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36920000 QueuedCount 21274 2016-04-07T02:18:09.975-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:9869522368, Disk:26274299904, ExtraSnapshotData:0, Fragmentation:62.43659239614044%, mode : circular 2016-04-07T02:18:09.975-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:18:10.11-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36930000 QueuedCount 11274 2016-04-07T02:18:10.531-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36940000 QueuedCount 1274 2016-04-07T02:18:10.612-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36941274, 36941274, 0 committed:false 2016-04-07T02:18:10.613-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36941274, 36941274, 0 committed:false) SnapCreateDur 38.752µs SnapOpenDur 695.782µs 2016-04-07T02:18:11.822-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.622µs CommitTime 615.472374ms TotalFlushTime 26m33.991431499s TotalCommitTime 3m15.677699834s 2016-04-07T02:18:11.822-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 36949949, 36949949, 293 committed:true 2016-04-07T02:18:11.822-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 36949949, 36949949, 293 committed:true) SnapCreateDur 624.259352ms SnapOpenDur 45.473µs 2016-04-07T02:18:11.823-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36950000 QueuedCount 2188 2016-04-07T02:18:11.826-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:11.83-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:11.834-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:11.838-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:11.842-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:12.305-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36960000 QueuedCount 41714 2016-04-07T02:18:12.659-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36970000 QueuedCount 31714 2016-04-07T02:18:13.021-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36980000 QueuedCount 21714 2016-04-07T02:18:13.396-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 36990000 QueuedCount 11714 2016-04-07T02:18:13.765-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37000000 QueuedCount 1714 2016-04-07T02:18:13.841-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37001714, 37001714, 0 committed:false 2016-04-07T02:18:13.841-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37001714, 37001714, 0 committed:false) SnapCreateDur 40.304µs SnapOpenDur 540.173µs 2016-04-07T02:18:13.845-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:13.849-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:13.853-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:13.857-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:13.861-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:14.231-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37010000 QueuedCount 44787 2016-04-07T02:18:14.606-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37020000 QueuedCount 34787 2016-04-07T02:18:14.98-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37030000 QueuedCount 24787 2016-04-07T02:18:15.313-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37040000 QueuedCount 14787 2016-04-07T02:18:15.705-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37050000 QueuedCount 4787 2016-04-07T02:18:15.946-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37054787, 37054787, 0 committed:false 2016-04-07T02:18:15.947-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37054787, 37054787, 0 committed:false) SnapCreateDur 42.984µs SnapOpenDur 368.322µs 2016-04-07T02:18:16.248-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37060000 QueuedCount 3731 2016-04-07T02:18:16.992-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.844µs CommitTime 571.393602ms TotalFlushTime 26m38.509149569s TotalCommitTime 3m16.249093436s 2016-04-07T02:18:16.992-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37063731, 37063731, 294 committed:true 2016-04-07T02:18:16.992-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37063731, 37063731, 294 committed:true) SnapCreateDur 580.214904ms SnapOpenDur 51.36µs 2016-04-07T02:18:16.996-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:17-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:17.004-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:17.008-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:17.012-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:17.308-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37070000 QueuedCount 47065 2016-04-07T02:18:17.643-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37080000 QueuedCount 37065 2016-04-07T02:18:18.027-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37090000 QueuedCount 27065 2016-04-07T02:18:18.437-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37100000 QueuedCount 17065 2016-04-07T02:18:18.835-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37110000 QueuedCount 7065 2016-04-07T02:18:19.136-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37117065, 37117065, 0 committed:false 2016-04-07T02:18:19.136-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37117065, 37117065, 0 committed:false) SnapCreateDur 36.431µs SnapOpenDur 172.911µs 2016-04-07T02:18:19.14-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:19.144-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:19.148-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:19.152-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:19.155-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:19.359-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37120000 QueuedCount 50000 2016-04-07T02:18:19.754-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37130000 QueuedCount 40799 2016-04-07T02:18:20.145-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37140000 QueuedCount 30799 2016-04-07T02:18:20.55-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37150000 QueuedCount 20799 2016-04-07T02:18:20.887-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37160000 QueuedCount 10799 2016-04-07T02:18:21.289-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37170000 QueuedCount 799 2016-04-07T02:18:21.36-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37170799, 37170799, 0 committed:false 2016-04-07T02:18:21.36-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37170799, 37170799, 0 committed:false) SnapCreateDur 28.346µs SnapOpenDur 456.949µs 2016-04-07T02:18:22.537-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.182µs CommitTime 642.800326ms TotalFlushTime 26m43.338273942s TotalCommitTime 3m16.891893762s 2016-04-07T02:18:22.537-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37179646, 37179646, 295 committed:true 2016-04-07T02:18:22.537-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37179646, 37179646, 295 committed:true) SnapCreateDur 651.870283ms SnapOpenDur 33.607µs 2016-04-07T02:18:22.542-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:22.543-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37180000 QueuedCount 11492 2016-04-07T02:18:22.546-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:22.551-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:22.555-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:22.559-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:23.097-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37190000 QueuedCount 43615 2016-04-07T02:18:23.434-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37200000 QueuedCount 33615 2016-04-07T02:18:23.809-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37210000 QueuedCount 23615 2016-04-07T02:18:24.215-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37220000 QueuedCount 13615 2016-04-07T02:18:24.624-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37230000 QueuedCount 3615 2016-04-07T02:18:24.789-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37233615, 37233615, 0 committed:false 2016-04-07T02:18:24.79-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37233615, 37233615, 0 committed:false) SnapCreateDur 59.902µs SnapOpenDur 738.405µs 2016-04-07T02:18:24.796-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:24.801-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:24.806-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:24.811-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:24.817-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:25.181-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37240000 QueuedCount 48048 2016-04-07T02:18:25.574-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37250000 QueuedCount 38048 2016-04-07T02:18:25.966-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37260000 QueuedCount 28048 2016-04-07T02:18:26.363-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37270000 QueuedCount 18048 2016-04-07T02:18:26.697-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37280000 QueuedCount 8048 2016-04-07T02:18:27.019-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37288048, 37288048, 0 committed:false 2016-04-07T02:18:27.021-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37288048, 37288048, 0 committed:false) SnapCreateDur 30.519µs SnapOpenDur 1.786737ms 2016-04-07T02:18:27.314-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37290000 QueuedCount 7096 2016-04-07T02:18:28.256-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.28µs CommitTime 611.198639ms TotalFlushTime 26m48.354733541s TotalCommitTime 3m17.503092401s 2016-04-07T02:18:28.256-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37297096, 37297096, 296 committed:true 2016-04-07T02:18:28.256-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37297096, 37297096, 296 committed:true) SnapCreateDur 619.945317ms SnapOpenDur 33.38µs 2016-04-07T02:18:28.262-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:28.266-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:28.271-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:28.276-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:28.28-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:28.46-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37300000 QueuedCount 50000 2016-04-07T02:18:28.878-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37310000 QueuedCount 40071 2016-04-07T02:18:29.28-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37320000 QueuedCount 30071 2016-04-07T02:18:29.67-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37330000 QueuedCount 20071 2016-04-07T02:18:30.003-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37340000 QueuedCount 10071 2016-04-07T02:18:30.425-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37350000 QueuedCount 71 2016-04-07T02:18:30.431-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37350071, 37350071, 0 committed:false 2016-04-07T02:18:30.433-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37350071, 37350071, 0 committed:false) SnapCreateDur 29.752µs SnapOpenDur 1.61279ms 2016-04-07T02:18:30.438-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:30.442-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:30.447-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:30.452-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:30.457-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:30.938-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37360000 QueuedCount 43277 2016-04-07T02:18:31.344-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37370000 QueuedCount 33277 2016-04-07T02:18:31.74-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37380000 QueuedCount 23277 2016-04-07T02:18:32.142-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37390000 QueuedCount 13277 2016-04-07T02:18:32.555-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37400000 QueuedCount 3277 2016-04-07T02:18:32.656-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37403277, 37403277, 0 committed:false 2016-04-07T02:18:32.657-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37403277, 37403277, 0 committed:false) SnapCreateDur 48.014µs SnapOpenDur 1.488789ms 2016-04-07T02:18:33.202-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37410000 QueuedCount 2315 2016-04-07T02:18:33.337-07:00 [Info] Indexer::ReadMemstats Time Taken 14.92506ms 2016-04-07T02:18:33.903-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 31.035µs CommitTime 602.151531ms TotalFlushTime 26m53.306870893s TotalCommitTime 3m18.105243932s 2016-04-07T02:18:33.903-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37412315, 37412315, 297 committed:true 2016-04-07T02:18:33.903-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37412315, 37412315, 297 committed:true) SnapCreateDur 610.80894ms SnapOpenDur 48.706µs 2016-04-07T02:18:33.908-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:33.913-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:33.918-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:33.923-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:33.927-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:34.301-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37420000 QueuedCount 43662 2016-04-07T02:18:34.664-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37430000 QueuedCount 33662 2016-04-07T02:18:35.029-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37440000 QueuedCount 23662 2016-04-07T02:18:35.402-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37450000 QueuedCount 13662 2016-04-07T02:18:35.795-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37460000 QueuedCount 3662 2016-04-07T02:18:35.963-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37463662, 37463662, 0 committed:false 2016-04-07T02:18:35.963-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37463662, 37463662, 0 committed:false) SnapCreateDur 57.94µs SnapOpenDur 210.039µs 2016-04-07T02:18:35.968-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:35.973-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:35.977-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:35.982-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:35.987-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:36.276-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37470000 QueuedCount 46685 2016-04-07T02:18:36.655-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37480000 QueuedCount 36685 2016-04-07T02:18:36.988-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37490000 QueuedCount 26685 2016-04-07T02:18:37.39-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37500000 QueuedCount 16685 2016-04-07T02:18:37.783-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37510000 QueuedCount 6685 2016-04-07T02:18:38.033-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37516685, 37516685, 0 committed:false 2016-04-07T02:18:38.035-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37516685, 37516685, 0 committed:false) SnapCreateDur 28.22µs SnapOpenDur 1.760621ms 2016-04-07T02:18:38.319-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37520000 QueuedCount 5825 2016-04-07T02:18:39.132-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.966µs CommitTime 543.474724ms TotalFlushTime 26m57.892808681s TotalCommitTime 3m18.648718656s 2016-04-07T02:18:39.132-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37525825, 37525825, 298 committed:true 2016-04-07T02:18:39.132-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37525825, 37525825, 298 committed:true) SnapCreateDur 552.252842ms SnapOpenDur 42.85µs 2016-04-07T02:18:39.137-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:39.141-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:39.145-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:39.15-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:39.154-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:39.368-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37530000 QueuedCount 48720 2016-04-07T02:18:39.677-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37540000 QueuedCount 38720 2016-04-07T02:18:39.977-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:10033334297, Disk:26274299904, ExtraSnapshotData:0, Fragmentation:61.81312410355594%, mode : circular 2016-04-07T02:18:39.977-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:18:40.047-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37550000 QueuedCount 28720 2016-04-07T02:18:40.423-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37560000 QueuedCount 18720 2016-04-07T02:18:40.804-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37570000 QueuedCount 8720 2016-04-07T02:18:41.122-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37578720, 37578720, 0 committed:false 2016-04-07T02:18:41.124-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37578720, 37578720, 0 committed:false) SnapCreateDur 30.899µs SnapOpenDur 1.498601ms 2016-04-07T02:18:41.128-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:41.132-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:41.137-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:41.141-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:41.145-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:41.261-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37580000 QueuedCount 50000 2016-04-07T02:18:41.643-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37590000 QueuedCount 41223 2016-04-07T02:18:42.023-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37600000 QueuedCount 31223 2016-04-07T02:18:42.421-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37610000 QueuedCount 21223 2016-04-07T02:18:42.752-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37620000 QueuedCount 11223 2016-04-07T02:18:43.143-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37630000 QueuedCount 1223 2016-04-07T02:18:43.223-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37631223, 37631223, 0 committed:false 2016-04-07T02:18:43.224-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37631223, 37631223, 0 committed:false) SnapCreateDur 31.798µs SnapOpenDur 754.15µs 2016-04-07T02:18:43.79-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37640000 QueuedCount 8 2016-04-07T02:18:44.427-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.955µs CommitTime 619.726346ms TotalFlushTime 27m2.48098945s TotalCommitTime 3m19.268445002s 2016-04-07T02:18:44.427-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37640008, 37640008, 299 committed:true 2016-04-07T02:18:44.428-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37640008, 37640008, 299 committed:true) SnapCreateDur 628.43678ms SnapOpenDur 46.132µs 2016-04-07T02:18:44.433-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:44.437-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:44.441-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:44.446-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:44.451-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:44.829-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37650000 QueuedCount 42902 2016-04-07T02:18:45.193-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37660000 QueuedCount 32902 2016-04-07T02:18:45.563-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37670000 QueuedCount 22902 2016-04-07T02:18:45.94-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37680000 QueuedCount 12902 2016-04-07T02:18:46.337-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37690000 QueuedCount 2902 2016-04-07T02:18:46.43-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37692902, 37692902, 0 committed:false 2016-04-07T02:18:46.431-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37692902, 37692902, 0 committed:false) SnapCreateDur 27.719µs SnapOpenDur 1.488506ms 2016-04-07T02:18:46.436-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:46.441-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:46.445-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:46.449-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:46.453-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:46.813-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37700000 QueuedCount 45805 2016-04-07T02:18:47.225-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37710000 QueuedCount 35805 2016-04-07T02:18:47.614-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37720000 QueuedCount 25805 2016-04-07T02:18:47.946-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37730000 QueuedCount 15805 2016-04-07T02:18:48.349-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37740000 QueuedCount 5805 2016-04-07T02:18:48.606-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37745805, 37745805, 0 committed:false 2016-04-07T02:18:48.607-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37745805, 37745805, 0 committed:false) SnapCreateDur 34.544µs SnapOpenDur 1.217632ms 2016-04-07T02:18:49.001-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37750000 QueuedCount 4922 2016-04-07T02:18:49.863-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.587µs CommitTime 581.671673ms TotalFlushTime 27m7.241884423s TotalCommitTime 3m19.850116675s 2016-04-07T02:18:49.864-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37754922, 37754922, 300 committed:true 2016-04-07T02:18:49.864-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37754922, 37754922, 300 committed:true) SnapCreateDur 591.577258ms SnapOpenDur 79.793µs 2016-04-07T02:18:49.868-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:49.873-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:49.877-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:49.882-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:49.887-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:50.127-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37760000 QueuedCount 47641 2016-04-07T02:18:50.519-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37770000 QueuedCount 37641 2016-04-07T02:18:50.896-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37780000 QueuedCount 27641 2016-04-07T02:18:51.282-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37790000 QueuedCount 17641 2016-04-07T02:18:51.667-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37800000 QueuedCount 7641 2016-04-07T02:18:51.925-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37807641, 37807641, 0 committed:false 2016-04-07T02:18:51.926-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37807641, 37807641, 0 committed:false) SnapCreateDur 30.959µs SnapOpenDur 1.378575ms 2016-04-07T02:18:51.931-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:51.936-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:51.941-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:51.946-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:51.951-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:52.141-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37810000 QueuedCount 50000 2016-04-07T02:18:52.56-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37820000 QueuedCount 40518 2016-04-07T02:18:52.969-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37830000 QueuedCount 30518 2016-04-07T02:18:53.374-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37840000 QueuedCount 20518 2016-04-07T02:18:53.788-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37850000 QueuedCount 10518 2016-04-07T02:18:54.199-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37860000 QueuedCount 518 2016-04-07T02:18:54.206-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37860518, 37860518, 0 committed:false 2016-04-07T02:18:54.207-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37860518, 37860518, 0 committed:false) SnapCreateDur 30.412µs SnapOpenDur 858.216µs 2016-04-07T02:18:55.388-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.675µs CommitTime 626.304515ms TotalFlushTime 27m12.056768106s TotalCommitTime 3m20.47642119s 2016-04-07T02:18:55.388-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37869531, 37869531, 301 committed:true 2016-04-07T02:18:55.388-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37869531, 37869531, 301 committed:true) SnapCreateDur 635.572736ms SnapOpenDur 55.355µs 2016-04-07T02:18:55.394-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:55.398-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:55.398-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37870000 QueuedCount 16469 2016-04-07T02:18:55.402-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:55.407-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:55.412-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:55.987-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37880000 QueuedCount 38748 2016-04-07T02:18:56.329-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37890000 QueuedCount 28748 2016-04-07T02:18:56.754-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37900000 QueuedCount 18748 2016-04-07T02:18:57.165-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37910000 QueuedCount 8748 2016-04-07T02:18:57.568-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37918748, 37918748, 0 committed:false 2016-04-07T02:18:57.568-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37918748, 37918748, 0 committed:false) SnapCreateDur 33.765µs SnapOpenDur 154.669µs 2016-04-07T02:18:57.573-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:57.578-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:57.584-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:57.59-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:57.595-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:18:57.597-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37920000 QueuedCount 46250 2016-04-07T02:18:58.067-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37930000 QueuedCount 41637 2016-04-07T02:18:58.477-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37940000 QueuedCount 31637 2016-04-07T02:18:58.9-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37950000 QueuedCount 21637 2016-04-07T02:18:59.301-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37960000 QueuedCount 11637 2016-04-07T02:18:59.705-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37970000 QueuedCount 1637 2016-04-07T02:18:59.728-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37971637, 37971637, 0 committed:false 2016-04-07T02:18:59.73-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37971637, 37971637, 0 committed:false) SnapCreateDur 30.523µs SnapOpenDur 1.561684ms 2016-04-07T02:19:00.222-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37980000 QueuedCount 552 2016-04-07T02:19:00.842-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 30.555µs CommitTime 536.940008ms TotalFlushTime 27m16.872456798s TotalCommitTime 3m21.013361198s 2016-04-07T02:19:00.842-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 37980552, 37980552, 302 committed:true 2016-04-07T02:19:00.842-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 37980552, 37980552, 302 committed:true) SnapCreateDur 545.974847ms SnapOpenDur 58.398µs 2016-04-07T02:19:00.847-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:00.852-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:00.857-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:00.862-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:00.866-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:01.259-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 37990000 QueuedCount 43290 2016-04-07T02:19:01.643-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38000000 QueuedCount 33290 2016-04-07T02:19:02.032-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38010000 QueuedCount 23290 2016-04-07T02:19:02.437-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38020000 QueuedCount 13290 2016-04-07T02:19:02.851-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38030000 QueuedCount 3290 2016-04-07T02:19:02.95-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38033290, 38033290, 0 committed:false 2016-04-07T02:19:02.951-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38033290, 38033290, 0 committed:false) SnapCreateDur 29.439µs SnapOpenDur 1.294078ms 2016-04-07T02:19:02.957-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:02.962-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:02.967-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:02.971-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:02.976-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:03.346-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38040000 QueuedCount 46997 2016-04-07T02:19:03.695-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38050000 QueuedCount 36997 2016-04-07T02:19:04.109-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38060000 QueuedCount 26997 2016-04-07T02:19:04.527-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38070000 QueuedCount 16997 2016-04-07T02:19:04.938-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38080000 QueuedCount 6997 2016-04-07T02:19:05.201-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38086997, 38086997, 0 committed:false 2016-04-07T02:19:05.204-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38086997, 38086997, 0 committed:false) SnapCreateDur 63.592µs SnapOpenDur 2.18107ms 2016-04-07T02:19:05.534-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38090000 QueuedCount 5957 2016-04-07T02:19:06.334-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.43µs CommitTime 547.325617ms TotalFlushTime 27m21.720257231s TotalCommitTime 3m21.560686815s 2016-04-07T02:19:06.335-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38095957, 38095957, 303 committed:true 2016-04-07T02:19:06.335-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38095957, 38095957, 303 committed:true) SnapCreateDur 555.913333ms SnapOpenDur 57.78µs 2016-04-07T02:19:06.34-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:06.345-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:06.349-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:06.354-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:06.359-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:06.589-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38100000 QueuedCount 48998 2016-04-07T02:19:06.966-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38110000 QueuedCount 38998 2016-04-07T02:19:07.338-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38120000 QueuedCount 28998 2016-04-07T02:19:07.716-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38130000 QueuedCount 18998 2016-04-07T02:19:08.099-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38140000 QueuedCount 8998 2016-04-07T02:19:08.42-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38148998, 38148998, 0 committed:false 2016-04-07T02:19:08.421-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38148998, 38148998, 0 committed:false) SnapCreateDur 31.01µs SnapOpenDur 1.627272ms 2016-04-07T02:19:08.609-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38150000 QueuedCount 8168 2016-04-07T02:19:08.779-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:08.783-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:08.787-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:08.791-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:08.795-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:09.07-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38160000 QueuedCount 42108 2016-04-07T02:19:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:19:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:19:09.374-07:00 [Info] client load stats {} 2016-04-07T02:19:09.479-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38170000 QueuedCount 32108 2016-04-07T02:19:09.663-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":1797892,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":1837727915,"bucket-1:myindex:avg_ts_items_count":40476,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":10191498991,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":26274299904,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":26633,"bucket-1:myindex:frag_percent":61,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":4005667847,"bucket-1:myindex:items_count":38095957,"bucket-1:myindex:num_commits":303,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":38175474,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":38158202,"bucket-1:myindex:num_docs_queued":1841798,"bucket-1:myindex:num_flush_queued":38202108,"bucket-1:myindex:num_items_flushed":38175475,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":639,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":2086754460,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"303 201560686815 1846744325650896353","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"38175478 197577480116 1662895332175001032","bucket-1:myindex:timings/storage_info":"1885 13626177 190317354203","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"302 109319255 45898032829281","bucket-1:myindex:timings/storage_meta_set":"303 112376607 45189883053723","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"303 12500483 570399626191","bucket-1:myindex:timings/storage_set":"76350954 1427907242107 1825313865622117311","bucket-1:myindex:timings/storage_snapshot_close":"335 259406633 554765473272233","bucket-1:myindex:timings/storage_snapshot_create":"336 301930219 386035301723983","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":40000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"463 1076904112 140987595860124820","bucket-1:ts_queue_size":203,"indexer_state":"Active","memory_quota":31457280000,"memory_used":22928224256,"memory_used_storage":18874368000,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"5378 1343015026 113915020785676216","uptime":"50m1.356314436s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:19:09.681-07:00 [Info] memstats {"Alloc":1325293088, "TotalAlloc":48056881128, "Sys":9855607112, "Lookups":10390, "Mallocs":1384975659,"Frees":1348864022, "HeapAlloc":1325293088, "HeapSys":9382395904, "HeapIdle":5627248640, "HeapInuse":3755147264,"HeapReleased":1959968768, "HeapObjects":36111637,"GCSys":298708992, "LastGC":1460020685652737702,"PauseTotalNs":61608840, "PauseNs":[1585927], "NumGC":45} 2016-04-07T02:19:09.886-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38180000 QueuedCount 22108 2016-04-07T02:19:09.977-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:10200434452, Disk:26274299904, ExtraSnapshotData:0, Fragmentation:61.177140820992584%, mode : circular 2016-04-07T02:19:09.977-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:19:10.219-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38190000 QueuedCount 12108 2016-04-07T02:19:10.624-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38200000 QueuedCount 2108 2016-04-07T02:19:10.711-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38202108, 38202108, 0 committed:false 2016-04-07T02:19:10.712-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38202108, 38202108, 0 committed:false) SnapCreateDur 38.19µs SnapOpenDur 1.437087ms 2016-04-07T02:19:11.178-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38210000 QueuedCount 909 2016-04-07T02:19:11.798-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.355µs CommitTime 531.494099ms TotalFlushTime 27m26.567750353s TotalCommitTime 3m22.092180914s 2016-04-07T02:19:11.798-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38210909, 38210909, 304 committed:true 2016-04-07T02:19:11.798-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38210909, 38210909, 304 committed:true) SnapCreateDur 540.185286ms SnapOpenDur 55.409µs 2016-04-07T02:19:11.803-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:11.808-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:11.814-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:11.819-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:11.824-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:12.227-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38220000 QueuedCount 43683 2016-04-07T02:19:12.617-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38230000 QueuedCount 33683 2016-04-07T02:19:13.01-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38240000 QueuedCount 23683 2016-04-07T02:19:13.404-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38250000 QueuedCount 13683 2016-04-07T02:19:13.744-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38260000 QueuedCount 3683 2016-04-07T02:19:13.919-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38263683, 38263683, 0 committed:false 2016-04-07T02:19:13.921-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38263683, 38263683, 0 committed:false) SnapCreateDur 55.036µs SnapOpenDur 1.782717ms 2016-04-07T02:19:13.927-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:13.931-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:13.937-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:13.942-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:13.947-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:14.295-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38270000 QueuedCount 45316 2016-04-07T02:19:14.698-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38280000 QueuedCount 35316 2016-04-07T02:19:15.096-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38290000 QueuedCount 25316 2016-04-07T02:19:15.493-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38300000 QueuedCount 15316 2016-04-07T02:19:15.897-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38310000 QueuedCount 5316 2016-04-07T02:19:16.09-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38315316, 38315316, 0 committed:false 2016-04-07T02:19:16.092-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38315316, 38315316, 0 committed:false) SnapCreateDur 59.373µs SnapOpenDur 2.453732ms 2016-04-07T02:19:16.586-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38320000 QueuedCount 4372 2016-04-07T02:19:17.371-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 37.687µs CommitTime 603.970571ms TotalFlushTime 27m31.449704557s TotalCommitTime 3m22.696151485s 2016-04-07T02:19:17.371-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38324372, 38324372, 305 committed:true 2016-04-07T02:19:17.371-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38324372, 38324372, 305 committed:true) SnapCreateDur 612.67713ms SnapOpenDur 57.302µs 2016-04-07T02:19:17.376-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:17.381-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:17.386-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:17.391-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:17.396-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:17.737-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38330000 QueuedCount 44948 2016-04-07T02:19:18.124-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38340000 QueuedCount 34948 2016-04-07T02:19:18.528-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38350000 QueuedCount 24948 2016-04-07T02:19:18.927-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38360000 QueuedCount 14948 2016-04-07T02:19:19.325-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38370000 QueuedCount 4948 2016-04-07T02:19:19.502-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38374948, 38374948, 0 committed:false 2016-04-07T02:19:19.504-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38374948, 38374948, 0 committed:false) SnapCreateDur 51.967µs SnapOpenDur 1.545624ms 2016-04-07T02:19:19.509-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:19.515-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:19.52-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:19.525-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:19.53-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:19.824-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38380000 QueuedCount 47628 2016-04-07T02:19:20.227-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38390000 QueuedCount 37628 2016-04-07T02:19:20.586-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38400000 QueuedCount 27628 2016-04-07T02:19:20.992-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38410000 QueuedCount 17628 2016-04-07T02:19:21.39-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38420000 QueuedCount 7628 2016-04-07T02:19:21.707-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38427628, 38427628, 0 committed:false 2016-04-07T02:19:21.708-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38427628, 38427628, 0 committed:false) SnapCreateDur 32.168µs SnapOpenDur 744.505µs 2016-04-07T02:19:22.067-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38430000 QueuedCount 6848 2016-04-07T02:19:23.001-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 23.634µs CommitTime 667.937614ms TotalFlushTime 27m36.322916066s TotalCommitTime 3m23.364089099s 2016-04-07T02:19:23.001-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38436848, 38436848, 306 committed:true 2016-04-07T02:19:23.001-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38436848, 38436848, 306 committed:true) SnapCreateDur 676.780706ms SnapOpenDur 57.113µs 2016-04-07T02:19:23.007-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:23.012-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:23.018-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:23.023-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:23.028-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:23.17-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38440000 QueuedCount 50000 2016-04-07T02:19:23.57-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38450000 QueuedCount 41511 2016-04-07T02:19:23.973-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38460000 QueuedCount 31511 2016-04-07T02:19:24.372-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38470000 QueuedCount 21511 2016-04-07T02:19:24.791-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38480000 QueuedCount 11511 2016-04-07T02:19:25.129-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38490000 QueuedCount 1511 2016-04-07T02:19:25.211-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38491511, 38491511, 0 committed:false 2016-04-07T02:19:25.212-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38491511, 38491511, 0 committed:false) SnapCreateDur 27.812µs SnapOpenDur 1.235543ms 2016-04-07T02:19:25.217-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:25.222-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:25.227-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:25.232-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:25.237-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:25.623-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38500000 QueuedCount 45448 2016-04-07T02:19:26.019-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38510000 QueuedCount 35448 2016-04-07T02:19:26.428-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38520000 QueuedCount 25448 2016-04-07T02:19:26.826-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38530000 QueuedCount 15448 2016-04-07T02:19:27.229-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38540000 QueuedCount 5448 2016-04-07T02:19:27.477-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38545448, 38545448, 0 committed:false 2016-04-07T02:19:27.477-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38545448, 38545448, 0 committed:false) SnapCreateDur 30.121µs SnapOpenDur 139.067µs 2016-04-07T02:19:27.908-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38550000 QueuedCount 4273 2016-04-07T02:19:28.717-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 33.226µs CommitTime 626.405454ms TotalFlushTime 27m41.318975887s TotalCommitTime 3m23.990494553s 2016-04-07T02:19:28.717-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38554273, 38554273, 307 committed:true 2016-04-07T02:19:28.717-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38554273, 38554273, 307 committed:true) SnapCreateDur 635.311922ms SnapOpenDur 64.635µs 2016-04-07T02:19:28.722-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:28.726-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:28.73-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:28.734-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:28.739-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:28.974-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38560000 QueuedCount 48132 2016-04-07T02:19:29.365-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38570000 QueuedCount 38132 2016-04-07T02:19:29.773-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38580000 QueuedCount 28132 2016-04-07T02:19:30.166-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38590000 QueuedCount 18132 2016-04-07T02:19:30.58-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38600000 QueuedCount 8132 2016-04-07T02:19:30.916-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38608132, 38608132, 0 committed:false 2016-04-07T02:19:30.917-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38608132, 38608132, 0 committed:false) SnapCreateDur 27.846µs SnapOpenDur 694.715µs 2016-04-07T02:19:30.922-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:30.927-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:30.931-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:30.936-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:30.94-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:31.133-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38610000 QueuedCount 50000 2016-04-07T02:19:31.533-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38620000 QueuedCount 41526 2016-04-07T02:19:31.871-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38630000 QueuedCount 31526 2016-04-07T02:19:32.276-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38640000 QueuedCount 21526 2016-04-07T02:19:32.7-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38650000 QueuedCount 11526 2016-04-07T02:19:33.113-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38660000 QueuedCount 1526 2016-04-07T02:19:33.2-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38661526, 38661526, 0 committed:false 2016-04-07T02:19:33.201-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38661526, 38661526, 0 committed:false) SnapCreateDur 27.019µs SnapOpenDur 767.166µs 2016-04-07T02:19:33.785-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38670000 QueuedCount 137 2016-04-07T02:19:33.882-07:00 [Info] Indexer::ReadMemstats Time Taken 15.373431ms 2016-04-07T02:19:34.434-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.209µs CommitTime 638.94023ms TotalFlushTime 27m46.299233694s TotalCommitTime 3m24.629434783s 2016-04-07T02:19:34.435-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38670137, 38670137, 308 committed:true 2016-04-07T02:19:34.435-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38670137, 38670137, 308 committed:true) SnapCreateDur 649.009304ms SnapOpenDur 62.126µs 2016-04-07T02:19:34.44-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:34.445-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:34.449-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:34.965-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38680000 QueuedCount 25457 2016-04-07T02:19:35.361-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38690000 QueuedCount 15457 2016-04-07T02:19:35.767-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38700000 QueuedCount 5457 2016-04-07T02:19:36.014-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38705457, 38705457, 0 committed:false 2016-04-07T02:19:36.015-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38705457, 38705457, 0 committed:false) SnapCreateDur 41.619µs SnapOpenDur 471.908µs 2016-04-07T02:19:36.019-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:36.023-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:36.028-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:36.233-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38710000 QueuedCount 30902 2016-04-07T02:19:36.618-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38720000 QueuedCount 20902 2016-04-07T02:19:37.006-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38730000 QueuedCount 10902 2016-04-07T02:19:37.396-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38740000 QueuedCount 902 2016-04-07T02:19:37.412-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38740902, 38740902, 0 committed:false 2016-04-07T02:19:37.413-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38740902, 38740902, 0 committed:false) SnapCreateDur 29.972µs SnapOpenDur 961.074µs 2016-04-07T02:19:37.417-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:37.422-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:37.426-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:37.877-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38750000 QueuedCount 26059 2016-04-07T02:19:38.273-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38760000 QueuedCount 16059 2016-04-07T02:19:38.673-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38770000 QueuedCount 6059 2016-04-07T02:19:38.92-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38776059, 38776059, 0 committed:false 2016-04-07T02:19:38.922-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38776059, 38776059, 0 committed:false) SnapCreateDur 26.832µs SnapOpenDur 1.350065ms 2016-04-07T02:19:39.219-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38780000 QueuedCount 4620 2016-04-07T02:19:39.977-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:10358965304, Disk:26274299904, ExtraSnapshotData:0, Fragmentation:60.57377231039769%, mode : circular 2016-04-07T02:19:39.977-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:19:39.982-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 28.727µs CommitTime 576.703749ms TotalFlushTime 27m51.177138859s TotalCommitTime 3m25.206138532s 2016-04-07T02:19:39.982-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38784620, 38784620, 309 committed:true 2016-04-07T02:19:39.982-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38784620, 38784620, 309 committed:true) SnapCreateDur 585.734958ms SnapOpenDur 43.761µs 2016-04-07T02:19:39.987-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:39.991-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:39.995-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:40.23-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38790000 QueuedCount 27037 2016-04-07T02:19:40.628-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38800000 QueuedCount 17037 2016-04-07T02:19:41.007-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38810000 QueuedCount 7037 2016-04-07T02:19:41.25-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38817037, 38817037, 0 committed:false 2016-04-07T02:19:41.252-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38817037, 38817037, 0 committed:false) SnapCreateDur 28.398µs SnapOpenDur 1.586987ms 2016-04-07T02:19:41.257-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:41.262-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:41.266-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:41.491-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38820000 QueuedCount 32099 2016-04-07T02:19:41.888-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38830000 QueuedCount 22099 2016-04-07T02:19:42.309-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38840000 QueuedCount 12099 2016-04-07T02:19:42.661-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38850000 QueuedCount 2099 2016-04-07T02:19:42.75-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38852099, 38852099, 0 committed:false 2016-04-07T02:19:42.752-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38852099, 38852099, 0 committed:false) SnapCreateDur 31.383µs SnapOpenDur 1.717468ms 2016-04-07T02:19:42.758-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:42.762-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:42.767-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:43.185-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38860000 QueuedCount 27594 2016-04-07T02:19:43.592-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38870000 QueuedCount 17594 2016-04-07T02:19:44.003-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38880000 QueuedCount 7594 2016-04-07T02:19:44.342-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38887594, 38887594, 0 committed:false 2016-04-07T02:19:44.344-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38887594, 38887594, 0 committed:false) SnapCreateDur 32.765µs SnapOpenDur 1.738976ms 2016-04-07T02:19:44.691-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38890000 QueuedCount 7244 2016-04-07T02:19:45.599-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.655µs CommitTime 631.162896ms TotalFlushTime 27m56.067702329s TotalCommitTime 3m25.837301428s 2016-04-07T02:19:45.599-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38897244, 38897244, 310 committed:true 2016-04-07T02:19:45.599-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38897244, 38897244, 310 committed:true) SnapCreateDur 639.912706ms SnapOpenDur 58.927µs 2016-04-07T02:19:45.603-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:45.607-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:45.618-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:45.761-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38900000 QueuedCount 40594 2016-04-07T02:19:46.133-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38910000 QueuedCount 30594 2016-04-07T02:19:46.528-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38920000 QueuedCount 20594 2016-04-07T02:19:46.868-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38930000 QueuedCount 10594 2016-04-07T02:19:47.259-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38940000 QueuedCount 594 2016-04-07T02:19:47.334-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38940594, 38940594, 0 committed:false 2016-04-07T02:19:47.334-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38940594, 38940594, 0 committed:false) SnapCreateDur 43.075µs SnapOpenDur 347.01µs 2016-04-07T02:19:47.339-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:47.344-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:47.349-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:47.801-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38950000 QueuedCount 26096 2016-04-07T02:19:48.199-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38960000 QueuedCount 16096 2016-04-07T02:19:48.616-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38970000 QueuedCount 6096 2016-04-07T02:19:48.894-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 38976096, 38976096, 0 committed:false 2016-04-07T02:19:48.895-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 38976096, 38976096, 0 committed:false) SnapCreateDur 32.409µs SnapOpenDur 875.625µs 2016-04-07T02:19:48.9-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:48.905-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:48.91-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:49.231-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38980000 QueuedCount 31310 2016-04-07T02:19:49.634-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 38990000 QueuedCount 21310 2016-04-07T02:19:50.057-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39000000 QueuedCount 11310 2016-04-07T02:19:50.466-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39010000 QueuedCount 1310 2016-04-07T02:19:50.485-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39011310, 39011310, 0 committed:false 2016-04-07T02:19:50.487-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39011310, 39011310, 0 committed:false) SnapCreateDur 31.187µs SnapOpenDur 1.199185ms 2016-04-07T02:19:51.095-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39020000 QueuedCount 606 2016-04-07T02:19:51.796-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 19.635µs CommitTime 615.346868ms TotalFlushTime 28m1.532605825s TotalCommitTime 3m26.452648296s 2016-04-07T02:19:51.796-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39020606, 39020606, 311 committed:true 2016-04-07T02:19:51.797-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39020606, 39020606, 311 committed:true) SnapCreateDur 624.574742ms SnapOpenDur 55.356µs 2016-04-07T02:19:51.801-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:51.806-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:51.811-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:52.286-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39030000 QueuedCount 25792 2016-04-07T02:19:52.676-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39040000 QueuedCount 15792 2016-04-07T02:19:53.074-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39050000 QueuedCount 5792 2016-04-07T02:19:53.316-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39055792, 39055792, 0 committed:false 2016-04-07T02:19:53.316-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39055792, 39055792, 0 committed:false) SnapCreateDur 41.451µs SnapOpenDur 372.482µs 2016-04-07T02:19:53.321-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:53.326-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:53.33-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:53.524-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39060000 QueuedCount 31470 2016-04-07T02:19:53.9-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39070000 QueuedCount 21470 2016-04-07T02:19:54.248-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39080000 QueuedCount 11470 2016-04-07T02:19:54.644-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39090000 QueuedCount 1470 2016-04-07T02:19:54.725-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39091470, 39091470, 0 committed:false 2016-04-07T02:19:54.726-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39091470, 39091470, 0 committed:false) SnapCreateDur 30.648µs SnapOpenDur 1.099429ms 2016-04-07T02:19:54.732-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:54.736-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:54.741-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:55.293-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39100000 QueuedCount 27375 2016-04-07T02:19:55.699-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39110000 QueuedCount 17375 2016-04-07T02:19:56.111-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39120000 QueuedCount 7375 2016-04-07T02:19:56.456-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39127375, 39127375, 0 committed:false 2016-04-07T02:19:56.456-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39127375, 39127375, 0 committed:false) SnapCreateDur 32.013µs SnapOpenDur 316.949µs 2016-04-07T02:19:56.763-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39130000 QueuedCount 6157 2016-04-07T02:19:57.647-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 21.156µs CommitTime 605.410299ms TotalFlushTime 28m6.684340516s TotalCommitTime 3m27.058058595s 2016-04-07T02:19:57.647-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39136157, 39136157, 312 committed:true 2016-04-07T02:19:57.647-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39136157, 39136157, 312 committed:true) SnapCreateDur 615.309632ms SnapOpenDur 62.383µs 2016-04-07T02:19:57.651-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:57.656-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:57.661-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:57.903-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39140000 QueuedCount 31794 2016-04-07T02:19:58.305-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39150000 QueuedCount 21794 2016-04-07T02:19:58.705-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39160000 QueuedCount 11794 2016-04-07T02:19:59.104-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39170000 QueuedCount 1794 2016-04-07T02:19:59.185-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39171794, 39171794, 0 committed:false 2016-04-07T02:19:59.186-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39171794, 39171794, 0 committed:false) SnapCreateDur 28.452µs SnapOpenDur 680.493µs 2016-04-07T02:19:59.19-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:59.194-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:59.199-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:19:59.606-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39180000 QueuedCount 26908 2016-04-07T02:20:00.003-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39190000 QueuedCount 16908 2016-04-07T02:20:00.412-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39200000 QueuedCount 6908 2016-04-07T02:20:00.664-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39206908, 39206908, 0 committed:false 2016-04-07T02:20:00.665-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39206908, 39206908, 0 committed:false) SnapCreateDur 39.204µs SnapOpenDur 883.947µs 2016-04-07T02:20:00.669-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:00.672-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:00.677-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:01.018-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39210000 QueuedCount 31394 2016-04-07T02:20:01.359-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39220000 QueuedCount 21394 2016-04-07T02:20:01.783-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39230000 QueuedCount 11394 2016-04-07T02:20:02.211-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39240000 QueuedCount 1394 2016-04-07T02:20:02.302-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39241394, 39241394, 0 committed:false 2016-04-07T02:20:02.303-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39241394, 39241394, 0 committed:false) SnapCreateDur 31.099µs SnapOpenDur 873.635µs 2016-04-07T02:20:02.913-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39250000 QueuedCount 342 2016-04-07T02:20:03.592-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 34.555µs CommitTime 665.683507ms TotalFlushTime 28m11.873487263s TotalCommitTime 3m27.723742102s 2016-04-07T02:20:03.592-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39250342, 39250342, 313 committed:true 2016-04-07T02:20:03.592-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39250342, 39250342, 313 committed:true) SnapCreateDur 674.340792ms SnapOpenDur 67.706µs 2016-04-07T02:20:03.595-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:03.599-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:03.604-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:04.06-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39260000 QueuedCount 24362 2016-04-07T02:20:04.42-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39270000 QueuedCount 14362 2016-04-07T02:20:04.789-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39280000 QueuedCount 4362 2016-04-07T02:20:04.949-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39284362, 39284362, 0 committed:false 2016-04-07T02:20:04.95-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39284362, 39284362, 0 committed:false) SnapCreateDur 31.872µs SnapOpenDur 1.140066ms 2016-04-07T02:20:04.955-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:04.959-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:04.963-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:05.23-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39290000 QueuedCount 30282 2016-04-07T02:20:05.621-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39300000 QueuedCount 20282 2016-04-07T02:20:06.577-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39310000 QueuedCount 10282 2016-04-07T02:20:06.962-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39320000 QueuedCount 282 2016-04-07T02:20:06.968-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39320282, 39320282, 0 committed:false 2016-04-07T02:20:06.968-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39320282, 39320282, 0 committed:false) SnapCreateDur 33.014µs SnapOpenDur 329.452µs 2016-04-07T02:20:06.973-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:06.978-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:06.983-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:07.487-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39330000 QueuedCount 25640 2016-04-07T02:20:07.876-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39340000 QueuedCount 15640 2016-04-07T02:20:08.291-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39350000 QueuedCount 5640 2016-04-07T02:20:08.538-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39355640, 39355640, 0 committed:false 2016-04-07T02:20:08.539-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39355640, 39355640, 0 committed:false) SnapCreateDur 32.833µs SnapOpenDur 754.276µs 2016-04-07T02:20:08.923-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39360000 QueuedCount 4801 2016-04-07T02:20:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:20:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:20:09.374-07:00 [Info] client load stats {} 2016-04-07T02:20:09.664-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":635199,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":1637424458,"bucket-1:myindex:avg_ts_items_count":33542,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":10510352636,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":26459156480,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":0,"bucket-1:myindex:frag_percent":60,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":4133115945,"bucket-1:myindex:items_count":39250342,"bucket-1:myindex:num_commits":313,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":39364801,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":39355640,"bucket-1:myindex:num_docs_queued":644360,"bucket-1:myindex:num_flush_queued":39364801,"bucket-1:myindex:num_items_flushed":39364801,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":676,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":1570724568,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"313 207723742102 5660041565048455898","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"39364801 203049962272 1893992051230123970","bucket-1:myindex:timings/storage_info":"1982 14214455 196726702305","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"313 113016846 47163578009906","bucket-1:myindex:timings/storage_meta_set":"314 117337492 47467018733138","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"313 12900989 587193704971","bucket-1:myindex:timings/storage_set":"78729602 1474191136844 5644131182933251530","bucket-1:myindex:timings/storage_snapshot_close":"362 289605541 776772125806503","bucket-1:myindex:timings/storage_snapshot_create":"363 326637227 416144894167901","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":40000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"475 1086897664 140995978242848346","bucket-1:ts_queue_size":72,"indexer_state":"Active","memory_quota":31457280000,"memory_used":23123062784,"memory_used_storage":18874368000,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"5522 1373134441 113921596707172479","uptime":"51m1.356993977s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:20:09.681-07:00 [Info] memstats {"Alloc":1667992040, "TotalAlloc":48399580080, "Sys":9855607112, "Lookups":10589, "Mallocs":1396529657,"Frees":1351671460, "HeapAlloc":1667992040, "HeapSys":9382100992, "HeapIdle":5432115200, "HeapInuse":3949985792,"HeapReleased":3344220160, "HeapObjects":44858197,"GCSys":298708992, "LastGC":1460020685652737702,"PauseTotalNs":61608840, "PauseNs":[], "NumGC":45} 2016-04-07T02:20:09.811-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 18.179µs CommitTime 686.805546ms TotalFlushTime 28m17.30456593s TotalCommitTime 3m28.410547648s 2016-04-07T02:20:09.811-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39364801, 39364801, 314 committed:true 2016-04-07T02:20:09.811-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39364801, 39364801, 314 committed:true) SnapCreateDur 696.383597ms SnapOpenDur 101.648µs 2016-04-07T02:20:09.817-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:09.822-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:09.827-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:09.977-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:10511839330, Disk:26757767168, ExtraSnapshotData:0, Fragmentation:60.714811277036375%, mode : circular 2016-04-07T02:20:09.977-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:20:10.168-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39370000 QueuedCount 30290 2016-04-07T02:20:10.576-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39380000 QueuedCount 20290 2016-04-07T02:20:10.965-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39390000 QueuedCount 10290 2016-04-07T02:20:11.356-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39400000 QueuedCount 290 2016-04-07T02:20:11.363-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39400290, 39400290, 0 committed:false 2016-04-07T02:20:11.364-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39400290, 39400290, 0 committed:false) SnapCreateDur 29.06µs SnapOpenDur 673.517µs 2016-04-07T02:20:11.369-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:11.375-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:11.38-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:11.856-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39410000 QueuedCount 26084 2016-04-07T02:20:12.182-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39420000 QueuedCount 16084 2016-04-07T02:20:12.576-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39430000 QueuedCount 6084 2016-04-07T02:20:12.815-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39436084, 39436084, 0 committed:false 2016-04-07T02:20:12.817-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39436084, 39436084, 0 committed:false) SnapCreateDur 28.66µs SnapOpenDur 1.801301ms 2016-04-07T02:20:12.823-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:12.828-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:12.834-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:13.179-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39440000 QueuedCount 32492 2016-04-07T02:20:13.577-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39450000 QueuedCount 22492 2016-04-07T02:20:13.98-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39460000 QueuedCount 12492 2016-04-07T02:20:14.408-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39470000 QueuedCount 2492 2016-04-07T02:20:14.499-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39472492, 39472492, 0 committed:false 2016-04-07T02:20:14.5-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39472492, 39472492, 0 committed:false) SnapCreateDur 30.436µs SnapOpenDur 1.019511ms 2016-04-07T02:20:15.028-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39480000 QueuedCount 947 2016-04-07T02:20:15.659-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.188µs CommitTime 604.446533ms TotalFlushTime 28m22.448096969s TotalCommitTime 3m29.014994181s 2016-04-07T02:20:15.659-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39480947, 39480947, 315 committed:true 2016-04-07T02:20:15.659-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39480947, 39480947, 315 committed:true) SnapCreateDur 613.021597ms SnapOpenDur 50.164µs 2016-04-07T02:20:15.664-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:15.67-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:15.674-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:16.149-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39490000 QueuedCount 27258 2016-04-07T02:20:16.54-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39500000 QueuedCount 17258 2016-04-07T02:20:16.956-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39510000 QueuedCount 7258 2016-04-07T02:20:17.22-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39517258, 39517258, 0 committed:false 2016-04-07T02:20:17.221-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39517258, 39517258, 0 committed:false) SnapCreateDur 41.085µs SnapOpenDur 1.369358ms 2016-04-07T02:20:17.226-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:17.231-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:17.236-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:17.468-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39520000 QueuedCount 33377 2016-04-07T02:20:17.808-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39530000 QueuedCount 23377 2016-04-07T02:20:18.196-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39540000 QueuedCount 13377 2016-04-07T02:20:18.593-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39550000 QueuedCount 3377 2016-04-07T02:20:18.752-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39553377, 39553377, 0 committed:false 2016-04-07T02:20:18.752-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39553377, 39553377, 0 committed:false) SnapCreateDur 31.517µs SnapOpenDur 580.742µs 2016-04-07T02:20:18.758-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:18.764-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:19.146-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39560000 QueuedCount 20847 2016-04-07T02:20:19.532-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39570000 QueuedCount 10847 2016-04-07T02:20:19.92-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39580000 QueuedCount 847 2016-04-07T02:20:19.929-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39580847, 39580847, 0 committed:false 2016-04-07T02:20:19.93-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39580847, 39580847, 0 committed:false) SnapCreateDur 38.384µs SnapOpenDur 1.317883ms 2016-04-07T02:20:21.091-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 20.89µs CommitTime 575.62937ms TotalFlushTime 28m27.206803126s TotalCommitTime 3m29.590623551s 2016-04-07T02:20:21.091-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39589936, 39589936, 316 committed:true 2016-04-07T02:20:21.092-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39589936, 39589936, 316 committed:true) SnapCreateDur 585.721347ms SnapOpenDur 75.804µs 2016-04-07T02:20:21.094-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39590000 QueuedCount 3091 2016-04-07T02:20:21.097-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:21.101-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:21.468-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39600000 QueuedCount 16871 2016-04-07T02:20:21.836-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39610000 QueuedCount 6871 2016-04-07T02:20:22.136-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39616871, 39616871, 0 committed:false 2016-04-07T02:20:22.138-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39616871, 39616871, 0 committed:false) SnapCreateDur 36.406µs SnapOpenDur 1.565574ms 2016-04-07T02:20:22.142-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:22.147-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:22.267-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39620000 QueuedCount 22904 2016-04-07T02:20:22.632-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39630000 QueuedCount 12904 2016-04-07T02:20:23.005-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39640000 QueuedCount 2904 2016-04-07T02:20:23.091-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39642904, 39642904, 0 committed:false 2016-04-07T02:20:23.093-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39642904, 39642904, 0 committed:false) SnapCreateDur 38.3µs SnapOpenDur 1.589837ms 2016-04-07T02:20:23.097-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:23.102-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:23.486-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39650000 QueuedCount 19083 2016-04-07T02:20:23.879-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39660000 QueuedCount 9083 2016-04-07T02:20:24.216-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39669083, 39669083, 0 committed:false 2016-04-07T02:20:24.218-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39669083, 39669083, 0 committed:false) SnapCreateDur 44.514µs SnapOpenDur 2.556753ms 2016-04-07T02:20:24.223-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:24.227-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:24.479-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39670000 QueuedCount 24866 2016-04-07T02:20:24.838-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39680000 QueuedCount 14866 2016-04-07T02:20:25.248-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39690000 QueuedCount 4866 2016-04-07T02:20:25.481-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39694866, 39694866, 0 committed:false 2016-04-07T02:20:25.481-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39694866, 39694866, 0 committed:false) SnapCreateDur 38.136µs SnapOpenDur 370.944µs 2016-04-07T02:20:25.781-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39700000 QueuedCount 4141 2016-04-07T02:20:26.593-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 29.659µs CommitTime 638.58948ms TotalFlushTime 28m31.983687848s TotalCommitTime 3m30.229213031s 2016-04-07T02:20:26.593-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39704141, 39704141, 317 committed:true 2016-04-07T02:20:26.593-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39704141, 39704141, 317 committed:true) SnapCreateDur 647.600031ms SnapOpenDur 68.506µs 2016-04-07T02:20:26.597-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:26.601-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:26.889-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39710000 QueuedCount 18837 2016-04-07T02:20:27.261-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39720000 QueuedCount 8837 2016-04-07T02:20:27.645-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39728837, 39728837, 0 committed:false 2016-04-07T02:20:27.646-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39728837, 39728837, 0 committed:false) SnapCreateDur 34.348µs SnapOpenDur 1.469683ms 2016-04-07T02:20:27.65-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:27.654-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:27.664-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39730000 QueuedCount 25682 2016-04-07T02:20:28.072-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39740000 QueuedCount 15682 2016-04-07T02:20:28.488-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39750000 QueuedCount 5682 2016-04-07T02:20:28.718-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39755682, 39755682, 0 committed:false 2016-04-07T02:20:28.718-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39755682, 39755682, 0 committed:false) SnapCreateDur 29.599µs SnapOpenDur 305.058µs 2016-04-07T02:20:28.722-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:28.725-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:28.971-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39760000 QueuedCount 22528 2016-04-07T02:20:29.363-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39770000 QueuedCount 12528 2016-04-07T02:20:29.748-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39780000 QueuedCount 2528 2016-04-07T02:20:29.839-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39782528, 39782528, 0 committed:false 2016-04-07T02:20:29.84-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39782528, 39782528, 0 committed:false) SnapCreateDur 43.034µs SnapOpenDur 533.894µs 2016-04-07T02:20:29.843-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:29.847-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:30.305-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39790000 QueuedCount 18961 2016-04-07T02:20:30.751-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39800000 QueuedCount 8961 2016-04-07T02:20:31.151-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39808961, 39808961, 0 committed:false 2016-04-07T02:20:31.151-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39808961, 39808961, 0 committed:false) SnapCreateDur 31.657µs SnapOpenDur 379.309µs 2016-04-07T02:20:31.166-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39810000 QueuedCount 8019 2016-04-07T02:20:32.263-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 24.03µs CommitTime 608.764728ms TotalFlushTime 28m36.951371661s TotalCommitTime 3m30.837977759s 2016-04-07T02:20:32.263-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39818019, 39818019, 318 committed:true 2016-04-07T02:20:32.263-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39818019, 39818019, 318 committed:true) SnapCreateDur 617.878426ms SnapOpenDur 76.336µs 2016-04-07T02:20:32.267-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:32.27-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:32.285-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39820000 QueuedCount 24930 2016-04-07T02:20:32.769-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39830000 QueuedCount 14930 2016-04-07T02:20:33.128-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39840000 QueuedCount 4930 2016-04-07T02:20:33.344-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39844930, 39844930, 0 committed:false 2016-04-07T02:20:33.344-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39844930, 39844930, 0 committed:false) SnapCreateDur 42.782µs SnapOpenDur 326.442µs 2016-04-07T02:20:33.348-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:33.352-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:33.546-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39850000 QueuedCount 20725 2016-04-07T02:20:33.945-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39860000 QueuedCount 10725 2016-04-07T02:20:34.181-07:00 [Info] Indexer::ReadMemstats Time Taken 12.068073ms 2016-04-07T02:20:34.355-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39870000 QueuedCount 725 2016-04-07T02:20:34.365-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39870725, 39870725, 0 committed:false 2016-04-07T02:20:34.367-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39870725, 39870725, 0 committed:false) SnapCreateDur 31.057µs SnapOpenDur 1.426025ms 2016-04-07T02:20:34.371-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:34.374-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:34.903-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39880000 QueuedCount 16449 2016-04-07T02:20:35.29-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39890000 QueuedCount 6449 2016-04-07T02:20:35.537-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39896449, 39896449, 0 committed:false 2016-04-07T02:20:35.538-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39896449, 39896449, 0 committed:false) SnapCreateDur 45.82µs SnapOpenDur 530.191µs 2016-04-07T02:20:35.542-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:35.791-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39900000 QueuedCount 14571 2016-04-07T02:20:36.132-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39910000 QueuedCount 4571 2016-04-07T02:20:36.376-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39914571, 39914571, 0 committed:false 2016-04-07T02:20:36.377-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39914571, 39914571, 0 committed:false) SnapCreateDur 32.765µs SnapOpenDur 316.553µs 2016-04-07T02:20:36.735-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39920000 QueuedCount 3120 2016-04-07T02:20:37.476-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 17.567µs CommitTime 576.321662ms TotalFlushTime 28m41.503490898s TotalCommitTime 3m31.414299421s 2016-04-07T02:20:37.476-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39923120, 39923120, 319 committed:true 2016-04-07T02:20:37.476-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39923120, 39923120, 319 committed:true) SnapCreateDur 584.984146ms SnapOpenDur 60.227µs 2016-04-07T02:20:37.48-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:37.766-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39930000 QueuedCount 11386 2016-04-07T02:20:38.134-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39940000 QueuedCount 1386 2016-04-07T02:20:38.154-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39941386, 39941386, 0 committed:false 2016-04-07T02:20:38.156-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39941386, 39941386, 0 committed:false) SnapCreateDur 33.81µs SnapOpenDur 1.818832ms 2016-04-07T02:20:38.16-07:00 [Info] StorageMgr::handleCreateSnapshot Skip Snapshot For MAINT_STREAM bucket-1 SnapType NO_SNAP 2016-04-07T02:20:38.568-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39950000 QueuedCount 9423 2016-04-07T02:20:38.905-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39959423, 39959423, 0 committed:false 2016-04-07T02:20:38.906-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39959423, 39959423, 0 committed:false) SnapCreateDur 33.517µs SnapOpenDur 1.406747ms 2016-04-07T02:20:39.024-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39960000 QueuedCount 8438 2016-04-07T02:20:39.36-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39968438, 39968438, 0 committed:false 2016-04-07T02:20:39.361-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39968438, 39968438, 0 committed:false) SnapCreateDur 31.871µs SnapOpenDur 273.276µs 2016-04-07T02:20:39.378-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39970000 QueuedCount 7569 2016-04-07T02:20:39.815-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39977569, 39977569, 0 committed:false 2016-04-07T02:20:39.816-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39977569, 39977569, 0 committed:false) SnapCreateDur 31.045µs SnapOpenDur 983.23µs 2016-04-07T02:20:39.978-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:10672401022, Disk:28232024064, ExtraSnapshotData:0, Fragmentation:62.1975349772782%, mode : circular 2016-04-07T02:20:39.978-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:20:40.03-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39980000 QueuedCount 6540 2016-04-07T02:20:40.281-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39986540, 39986540, 0 committed:false 2016-04-07T02:20:40.283-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39986540, 39986540, 0 committed:false) SnapCreateDur 26.553µs SnapOpenDur 1.944373ms 2016-04-07T02:20:40.558-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 39990000 QueuedCount 5463 2016-04-07T02:20:40.797-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 39995463, 39995463, 0 committed:false 2016-04-07T02:20:40.798-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 39995463, 39995463, 0 committed:false) SnapCreateDur 26.52µs SnapOpenDur 620.352µs 2016-04-07T02:20:41.086-07:00 [Info] logWriterStat:: 8343618505678776570 FlushedCount 40000000 QueuedCount 0 2016-04-07T02:20:41.09-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 40000000, 40000000, 0 committed:false 2016-04-07T02:20:41.091-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 40000000, 40000000, 0 committed:false) SnapCreateDur 30.695µs SnapOpenDur 843.283µs 2016-04-07T02:20:41.516-07:00 [Info] Timekeeper:: MAINT_STREAM bucket-1 Forcing Overdue Commit 2016-04-07T02:20:42.105-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 8343618505678776570 FlushTime 11.179µs CommitTime 579.591739ms TotalFlushTime 28m45.041146429s TotalCommitTime 3m31.99389116s 2016-04-07T02:20:42.105-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 8343618505678776570 Creating New Snapshot SnapshotInfo: seqnos: 40000000, 40000000, 320 committed:true 2016-04-07T02:20:42.105-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 8343618505678776570 PartitionId: 0 SliceId: 0 Crc64: 18387263461464417394 (SnapshotInfo: seqnos: 40000000, 40000000, 320 committed:true) SnapCreateDur 588.194112ms SnapOpenDur 61.781µs 2016-04-07T02:21:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:21:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:21:09.374-07:00 [Info] client load stats {} 2016-04-07T02:21:09.664-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":0,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":707730645,"bucket-1:myindex:avg_ts_items_count":3507,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":10677745181,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":28468518912,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":0,"bucket-1:myindex:frag_percent":62,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":4200000000,"bucket-1:myindex:items_count":40000000,"bucket-1:myindex:num_commits":320,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":40000000,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":40000000,"bucket-1:myindex:num_docs_queued":0,"bucket-1:myindex:num_flush_queued":40000000,"bucket-1:myindex:num_items_flushed":40000000,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":708,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":1013899591,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"320 211993891160 8274912465958201752","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"40000000 205746230879 1894004437321408995","bucket-1:myindex:timings/storage_info":"2057 14665703 201893480937","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"319 116317663 49873359266743","bucket-1:myindex:timings/storage_meta_set":"320 119837853 48549966311003","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"320 13253768 606645402750","bucket-1:myindex:timings/storage_set":"80000000 1498938754058 7651207434122473474","bucket-1:myindex:timings/storage_snapshot_close":"388 305036478 797634233787140","bucket-1:myindex:timings/storage_snapshot_create":"388 347346078 440937519486148","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":40000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"487 1096722730 141004049458617774","bucket-1:ts_queue_size":0,"indexer_state":"Active","memory_quota":31457280000,"memory_used":20848242688,"memory_used_storage":18874368000,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"5637 1396326653 113926489955897957","uptime":"52m1.357443347s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:21:09.681-07:00 [Info] memstats {"Alloc":582905096, "TotalAlloc":48727793936, "Sys":9855607112, "Lookups":10789, "Mallocs":1407549945,"Frees":1388659926, "HeapAlloc":582905096, "HeapSys":9382330368, "HeapIdle":7707164672, "HeapInuse":1675165696,"HeapReleased":3343507456, "HeapObjects":18890019,"GCSys":298708992, "LastGC":1460020805823357804,"PauseTotalNs":63395831, "PauseNs":[1786991], "NumGC":46} 2016-04-07T02:21:09.978-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:10677745181, Disk:28468518912, ExtraSnapshotData:0, Fragmentation:62.49279699444028%, mode : circular 2016-04-07T02:21:09.978-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:21:34.417-07:00 [Info] Indexer::ReadMemstats Time Taken 12.599331ms 2016-04-07T02:21:39.978-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:10677745181, Disk:28468518912, ExtraSnapshotData:0, Fragmentation:62.49279699444028%, mode : circular 2016-04-07T02:21:39.978-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:22:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:22:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:22:09.374-07:00 [Info] client load stats {} 2016-04-07T02:22:09.664-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":0,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":707730645,"bucket-1:myindex:avg_ts_items_count":3507,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":10677745181,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":28468518912,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":0,"bucket-1:myindex:frag_percent":62,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":4200000000,"bucket-1:myindex:items_count":40000000,"bucket-1:myindex:num_commits":320,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":40000000,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":40000000,"bucket-1:myindex:num_docs_queued":0,"bucket-1:myindex:num_flush_queued":40000000,"bucket-1:myindex:num_items_flushed":40000000,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":708,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":1013899591,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"320 211993891160 8274912465958201752","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"40000000 205746230879 1894004437321408995","bucket-1:myindex:timings/storage_info":"2057 14665703 201893480937","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"319 116317663 49873359266743","bucket-1:myindex:timings/storage_meta_set":"320 119837853 48549966311003","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"320 13253768 606645402750","bucket-1:myindex:timings/storage_set":"80000000 1498938754058 7651207434122473474","bucket-1:myindex:timings/storage_snapshot_close":"388 305036478 797634233787140","bucket-1:myindex:timings/storage_snapshot_create":"388 347346078 440937519486148","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":40000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"499 1106315488 141011725500063138","bucket-1:ts_queue_size":0,"indexer_state":"Active","memory_quota":31457280000,"memory_used":20883279872,"memory_used_storage":18874368000,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"5709 1409331707 113928921394149605","uptime":"53m1.357747489s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:22:09.681-07:00 [Info] memstats {"Alloc":645916496, "TotalAlloc":48790805336, "Sys":9855607112, "Lookups":10893, "Mallocs":1409471690,"Frees":1389119540, "HeapAlloc":645916496, "HeapSys":9382297600, "HeapIdle":7672094720, "HeapInuse":1710202880,"HeapReleased":3343368192, "HeapObjects":20352150,"GCSys":298708992, "LastGC":1460020805823357804,"PauseTotalNs":63395831, "PauseNs":[], "NumGC":46} 2016-04-07T02:22:09.978-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:10677745181, Disk:28468518912, ExtraSnapshotData:0, Fragmentation:62.49279699444028%, mode : circular 2016-04-07T02:22:09.978-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:22:34.646-07:00 [Info] Indexer::ReadMemstats Time Taken 11.142228ms 2016-04-07T02:22:39.979-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:10677745181, Disk:28468518912, ExtraSnapshotData:0, Fragmentation:62.49279699444028%, mode : circular 2016-04-07T02:22:39.979-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:23:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:23:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:23:09.374-07:00 [Info] client load stats {} 2016-04-07T02:23:09.664-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":0,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":707730645,"bucket-1:myindex:avg_ts_items_count":3507,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":10677745181,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":28468518912,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":0,"bucket-1:myindex:frag_percent":62,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":4200000000,"bucket-1:myindex:items_count":40000000,"bucket-1:myindex:num_commits":320,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":40000000,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":40000000,"bucket-1:myindex:num_docs_queued":0,"bucket-1:myindex:num_flush_queued":40000000,"bucket-1:myindex:num_items_flushed":40000000,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":708,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":1013899591,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"320 211993891160 8274912465958201752","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"40000000 205746230879 1894004437321408995","bucket-1:myindex:timings/storage_info":"2057 14665703 201893480937","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"319 116317663 49873359266743","bucket-1:myindex:timings/storage_meta_set":"320 119837853 48549966311003","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"320 13253768 606645402750","bucket-1:myindex:timings/storage_set":"80000000 1498938754058 7651207434122473474","bucket-1:myindex:timings/storage_snapshot_close":"388 305036478 797634233787140","bucket-1:myindex:timings/storage_snapshot_create":"388 347346078 440937519486148","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":40000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"510 1115254420 141018998261722466","bucket-1:ts_queue_size":0,"indexer_state":"Active","memory_quota":31457280000,"memory_used":19885232128,"memory_used_storage":18874368000,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"5780 1421799543 113931193222052963","uptime":"54m1.358065209s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:23:09.682-07:00 [Info] memstats {"Alloc":237111800, "TotalAlloc":48818598736, "Sys":9855607112, "Lookups":10971, "Mallocs":1410228697,"Frees":1399718130, "HeapAlloc":237111800, "HeapSys":9387081728, "HeapIdle":8674926592, "HeapInuse":712155136,"HeapReleased":3459530752, "HeapObjects":10510567,"GCSys":298708992, "LastGC":1460020925932025509,"PauseTotalNs":64858338, "PauseNs":[1462507], "NumGC":47} 2016-04-07T02:23:09.979-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:10677745181, Disk:28468518912, ExtraSnapshotData:0, Fragmentation:62.49279699444028%, mode : circular 2016-04-07T02:23:09.979-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:23:34.881-07:00 [Info] Indexer::ReadMemstats Time Taken 10.715966ms 2016-04-07T02:23:39.979-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:10677745181, Disk:28468518912, ExtraSnapshotData:0, Fragmentation:62.49279699444028%, mode : circular 2016-04-07T02:23:39.979-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday] 2016-04-07T02:24:09.374-07:00 [Info] connected with 1 indexers 2016-04-07T02:24:09.374-07:00 [Info] index 8343618505678776570 has 1 replicas 2016-04-07T02:24:09.374-07:00 [Info] client load stats {} 2016-04-07T02:24:09.665-07:00 [Info] PeriodicStats = {"bucket-1:mutation_queue_size":0,"bucket-1:myindex:avg_scan_latency":0,"bucket-1:myindex:avg_scan_request_alloc_latency":0,"bucket-1:myindex:avg_scan_request_init_latency":0,"bucket-1:myindex:avg_scan_request_latency":0,"bucket-1:myindex:avg_scan_wait_latency":0,"bucket-1:myindex:avg_ts_interval":707730645,"bucket-1:myindex:avg_ts_items_count":3507,"bucket-1:myindex:build_progress":100,"bucket-1:myindex:data_size":10677745181,"bucket-1:myindex:delete_bytes":0,"bucket-1:myindex:disk_load_duration":0,"bucket-1:myindex:disk_size":28468518912,"bucket-1:myindex:disk_store_duration":0,"bucket-1:myindex:flush_queue_size":0,"bucket-1:myindex:frag_percent":62,"bucket-1:myindex:get_bytes":0,"bucket-1:myindex:insert_bytes":4200000000,"bucket-1:myindex:items_count":40000000,"bucket-1:myindex:num_commits":320,"bucket-1:myindex:num_compactions":0,"bucket-1:myindex:num_completed_requests":0,"bucket-1:myindex:num_docs_indexed":40000000,"bucket-1:myindex:num_docs_pending":0,"bucket-1:myindex:num_docs_processed":40000000,"bucket-1:myindex:num_docs_queued":0,"bucket-1:myindex:num_flush_queued":40000000,"bucket-1:myindex:num_items_flushed":40000000,"bucket-1:myindex:num_items_restored":0,"bucket-1:myindex:num_last_snapshot_reply":0,"bucket-1:myindex:num_requests":0,"bucket-1:myindex:num_rows_returned":0,"bucket-1:myindex:num_snapshot_waiters":0,"bucket-1:myindex:num_snapshots":708,"bucket-1:myindex:scan_bytes_read":0,"bucket-1:myindex:scan_wait_duration":0,"bucket-1:myindex:since_last_snapshot":1013899591,"bucket-1:myindex:timings/dcp_getseqs":"0 0 0","bucket-1:myindex:timings/scan_pipeline_iterate":"0 0 0","bucket-1:myindex:timings/storage_clone_handle":"0 0 0","bucket-1:myindex:timings/storage_commit":"320 211993891160 8274912465958201752","bucket-1:myindex:timings/storage_del":"0 0 0","bucket-1:myindex:timings/storage_get":"40000000 205746230879 1894004437321408995","bucket-1:myindex:timings/storage_info":"2057 14665703 201893480937","bucket-1:myindex:timings/storage_iterator_next":"0 0 0","bucket-1:myindex:timings/storage_meta_get":"319 116317663 49873359266743","bucket-1:myindex:timings/storage_meta_set":"320 119837853 48549966311003","bucket-1:myindex:timings/storage_new_iterator":"0 0 0","bucket-1:myindex:timings/storage_persist_snapshot_create":"320 13253768 606645402750","bucket-1:myindex:timings/storage_set":"80000000 1498938754058 7651207434122473474","bucket-1:myindex:timings/storage_snapshot_close":"388 305036478 797634233787140","bucket-1:myindex:timings/storage_snapshot_create":"388 347346078 440937519486148","bucket-1:myindex:total_scan_duration":0,"bucket-1:myindex:total_scan_request_duration":0,"bucket-1:num_mutations_queued":40000000,"bucket-1:num_nonalign_ts":0,"bucket-1:timings/dcp_getseqs":"522 1125254136 141027396302703968","bucket-1:ts_queue_size":0,"indexer_state":"Active","memory_quota":31457280000,"memory_used":19906121728,"memory_used_storage":18874368000,"needs_restart":false,"num_connections":0,"storage_mode":"forestdb","timings/stats_response":"5853 1434975867 113933627125141149","uptime":"55m1.35841281s"} ==== StorageStats ==== ==== Index Instance 8343618505678776570 ==== ======== 2016-04-07T02:24:09.682-07:00 [Info] memstats {"Alloc":265080624, "TotalAlloc":48846567560, "Sys":9855607112, "Lookups":11050, "Mallocs":1410990886,"Frees":1399889185, "HeapAlloc":265080624, "HeapSys":9387081728, "HeapIdle":8654036992, "HeapInuse":733044736,"HeapReleased":3455107072, "HeapObjects":11101701,"GCSys":298708992, "LastGC":1460020925932025509,"PauseTotalNs":64858338, "PauseNs":[], "NumGC":47} 2016-04-07T02:24:09.979-07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 8343618505678776570 Data:10677745181, Disk:28468518912, ExtraSnapshotData:0, Fragmentation:62.49279699444028%, mode : circular 2016-04-07T02:24:09.979-07:00 [Info] CompactionDaemon: Compaction attempt skipped since compaction day is configured for [sunday]