Rebalance hang post swaprebalance on latest 181

Failing test case:-
swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=1,num-swap=1,keys-count=2000000,do-access=False

This cluster is available at 10.1.3.152:8091

Node 152 is operating above 90% bucket quota. Mem_used is still operating at 1.3G, active resident ratio has gone down 0% and replica resident ratio is 3% due to replica items sitting in checkpoint.

This seems to be a regression, we have not seen this issue on build 865.

[2012-06-05 11:51:52,198] - [swaprebalance:198] INFO - removing node ns_1@10.1.3.153 and rebalance afterwards
[2012-06-05 11:51:52,198] - [rest_client:636] INFO - adding remote node : 10.1.3.156 to this cluster @ : 10.1.3.152

Tap stats
10.1.3.156
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_bg_max_pending: 500
ep_tap_count: 5
ep_tap_deletes: 0
ep_tap_fg_fetched: 628028
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 156264
ep_tap_queue_drain: 628028
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 0
ep_tap_total_backlog_size: 0
ep_tap_total_fetched: 630803
ep_tap_total_queue: 0
eq_tapq:anon_1760:connected: true
eq_tapq:anon_1760:created: 717
eq_tapq:anon_1760:num_checkpoint_end: 0
eq_tapq:anon_1760:num_checkpoint_end_failed: 0
eq_tapq:anon_1760:num_checkpoint_start: 1
eq_tapq:anon_1760:num_checkpoint_start_failed: 0
eq_tapq:anon_1760:num_delete: 0
eq_tapq:anon_1760:num_delete_failed: 0
eq_tapq:anon_1760:num_flush: 0
eq_tapq:anon_1760:num_flush_failed: 0
eq_tapq:anon_1760:num_mutation: 0
eq_tapq:anon_1760:num_mutation_failed: 0
eq_tapq:anon_1760:num_opaque: 2
eq_tapq:anon_1760:num_opaque_failed: 0
eq_tapq:anon_1760:num_unknown: 0
eq_tapq:anon_1760:num_vbucket_set: 0
eq_tapq:anon_1760:num_vbucket_set_failed: 0
eq_tapq:anon_1760:pending_disconnect: false
eq_tapq:anon_1760:reserved: 0
eq_tapq:anon_1760:supports_ack: true
eq_tapq:anon_1760:type: consumer
eq_tapq:anon_2020:connected: true
eq_tapq:anon_2020:created: 816
eq_tapq:anon_2020:num_checkpoint_end: 0
eq_tapq:anon_2020:num_checkpoint_end_failed: 0
eq_tapq:anon_2020:num_checkpoint_start: 0
eq_tapq:anon_2020:num_checkpoint_start_failed: 0
eq_tapq:anon_2020:num_delete: 0
eq_tapq:anon_2020:num_delete_failed: 0
eq_tapq:anon_2020:num_flush: 0
eq_tapq:anon_2020:num_flush_failed: 0
eq_tapq:anon_2020:num_mutation: 471
eq_tapq:anon_2020:num_mutation_failed: 0
eq_tapq:anon_2020:num_opaque: 3
eq_tapq:anon_2020:num_opaque_failed: 0
eq_tapq:anon_2020:num_unknown: 0
eq_tapq:anon_2020:num_vbucket_set: 0
eq_tapq:anon_2020:num_vbucket_set_failed: 0
eq_tapq:anon_2020:pending_disconnect: false
eq_tapq:anon_2020:reserved: 0
eq_tapq:anon_2020:supports_ack: true
eq_tapq:anon_2020:type: consumer
eq_tapq:anon_2023:connected: true
eq_tapq:anon_2023:created: 817
eq_tapq:anon_2023:num_checkpoint_end: 282
eq_tapq:anon_2023:num_checkpoint_end_failed: 0
eq_tapq:anon_2023:num_checkpoint_start: 283
eq_tapq:anon_2023:num_checkpoint_start_failed: 0
eq_tapq:anon_2023:num_delete: 0
eq_tapq:anon_2023:num_delete_failed: 0
eq_tapq:anon_2023:num_flush: 0
eq_tapq:anon_2023:num_flush_failed: 0
eq_tapq:anon_2023:num_mutation: 39154
eq_tapq:anon_2023:num_mutation_failed: 0
eq_tapq:anon_2023:num_opaque: 2
eq_tapq:anon_2023:num_opaque_failed: 0
eq_tapq:anon_2023:num_unknown: 0
eq_tapq:anon_2023:num_vbucket_set: 0
eq_tapq:anon_2023:num_vbucket_set_failed: 0
eq_tapq:anon_2023:pending_disconnect: false
eq_tapq:anon_2023:reserved: 0
eq_tapq:anon_2023:supports_ack: true
eq_tapq:anon_2023:type: consumer
eq_tapq:replication_ns_1@10.1.3.152:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.152:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.152:ack_seqno: 549607
eq_tapq:replication_ns_1@10.1.3.152:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.152:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.152:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.152:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.152:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.152:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.152:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.152:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.152:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.152:complete: false
eq_tapq:replication_ns_1@10.1.3.152:connected: true
eq_tapq:replication_ns_1@10.1.3.152:created: 18
eq_tapq:replication_ns_1@10.1.3.152:disconnects: 340
eq_tapq:replication_ns_1@10.1.3.152:empty: true
eq_tapq:replication_ns_1@10.1.3.152:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.152:has_item: false
eq_tapq:replication_ns_1@10.1.3.152:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.152:idle: true
eq_tapq:replication_ns_1@10.1.3.152:num_tap_nack: 144050
eq_tapq:replication_ns_1@10.1.3.152:num_tap_tmpfail_survivors: 144050
eq_tapq:replication_ns_1@10.1.3.152:paused: 1
eq_tapq:replication_ns_1@10.1.3.152:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.152:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.152:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.152:qlen: 0
eq_tapq:replication_ns_1@10.1.3.152:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.152:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.152:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.152:queue_backoff: 144050
eq_tapq:replication_ns_1@10.1.3.152:queue_drain: 546834
eq_tapq:replication_ns_1@10.1.3.152:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.152:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.152:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.152:rec_fetched: 548489
eq_tapq:replication_ns_1@10.1.3.152:reconnects: 340
eq_tapq:replication_ns_1@10.1.3.152:recv_ack_seqno: 549606
eq_tapq:replication_ns_1@10.1.3.152:reserved: 1
eq_tapq:replication_ns_1@10.1.3.152:seqno_ack_requested: 549606
eq_tapq:replication_ns_1@10.1.3.152:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.152:suspended: false
eq_tapq:replication_ns_1@10.1.3.152:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.152:total_noops: 412
eq_tapq:replication_ns_1@10.1.3.152:type: producer
eq_tapq:replication_ns_1@10.1.3.152:vb_filter:

{ [342,512], [854,1023] }

eq_tapq:replication_ns_1@10.1.3.152:vb_filters: 341
eq_tapq:replication_ns_1@10.1.3.155:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.155:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.155:ack_seqno: 82773
eq_tapq:replication_ns_1@10.1.3.155:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.155:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.155:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.155:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.155:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.155:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.155:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.155:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.155:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.155:complete: false
eq_tapq:replication_ns_1@10.1.3.155:connected: true
eq_tapq:replication_ns_1@10.1.3.155:created: 15
eq_tapq:replication_ns_1@10.1.3.155:disconnects: 226
eq_tapq:replication_ns_1@10.1.3.155:empty: true
eq_tapq:replication_ns_1@10.1.3.155:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.155:has_item: false
eq_tapq:replication_ns_1@10.1.3.155:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.155:idle: true
eq_tapq:replication_ns_1@10.1.3.155:num_tap_nack: 12214
eq_tapq:replication_ns_1@10.1.3.155:num_tap_tmpfail_survivors: 12214
eq_tapq:replication_ns_1@10.1.3.155:paused: 1
eq_tapq:replication_ns_1@10.1.3.155:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.155:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.155:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.155:qlen: 0
eq_tapq:replication_ns_1@10.1.3.155:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.155:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.155:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.155:queue_backoff: 12214
eq_tapq:replication_ns_1@10.1.3.155:queue_drain: 81194
eq_tapq:replication_ns_1@10.1.3.155:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.155:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.155:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.155:rec_fetched: 82314
eq_tapq:replication_ns_1@10.1.3.155:reconnects: 226
eq_tapq:replication_ns_1@10.1.3.155:recv_ack_seqno: 82772
eq_tapq:replication_ns_1@10.1.3.155:reserved: 1
eq_tapq:replication_ns_1@10.1.3.155:seqno_ack_requested: 82772
eq_tapq:replication_ns_1@10.1.3.155:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.155:suspended: false
eq_tapq:replication_ns_1@10.1.3.155:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.155:total_noops: 418
eq_tapq:replication_ns_1@10.1.3.155:type: producer
eq_tapq:replication_ns_1@10.1.3.155:vb_filter:

{ [0,56], [513,682] }

eq_tapq:replication_ns_1@10.1.3.155:vb_filters: 227

10.1.3.152
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 2
ep_tap_bg_fetched: 2201504
ep_tap_bg_load_avg: 71
ep_tap_bg_max_load: 384692
ep_tap_bg_max_pending: 500
ep_tap_bg_max_wait: 21109900
ep_tap_bg_min_load: 13
ep_tap_bg_min_wait: 20
ep_tap_bg_num_samples: 168763
ep_tap_bg_wait_avg: 581535
ep_tap_count: 7
ep_tap_deletes: 0
ep_tap_fg_fetched: 4385161
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 133419
ep_tap_queue_drain: 5375845
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 316427
ep_tap_total_backlog_size: 12245
ep_tap_total_fetched: 6596091
ep_tap_total_queue: 0
eq_tapq:anon_2655:connected: true
eq_tapq:anon_2655:created: 1886
eq_tapq:anon_2655:num_checkpoint_end: 0
eq_tapq:anon_2655:num_checkpoint_end_failed: 0
eq_tapq:anon_2655:num_checkpoint_start: 1
eq_tapq:anon_2655:num_checkpoint_start_failed: 0
eq_tapq:anon_2655:num_delete: 0
eq_tapq:anon_2655:num_delete_failed: 0
eq_tapq:anon_2655:num_flush: 0
eq_tapq:anon_2655:num_flush_failed: 0
eq_tapq:anon_2655:num_mutation: 0
eq_tapq:anon_2655:num_mutation_failed: 0
eq_tapq:anon_2655:num_opaque: 2
eq_tapq:anon_2655:num_opaque_failed: 0
eq_tapq:anon_2655:num_unknown: 0
eq_tapq:anon_2655:num_vbucket_set: 0
eq_tapq:anon_2655:num_vbucket_set_failed: 0
eq_tapq:anon_2655:pending_disconnect: false
eq_tapq:anon_2655:reserved: 0
eq_tapq:anon_2655:supports_ack: true
eq_tapq:anon_2655:type: consumer
eq_tapq:anon_2964:connected: true
eq_tapq:anon_2964:created: 1998
eq_tapq:anon_2964:num_checkpoint_end: 452
eq_tapq:anon_2964:num_checkpoint_end_failed: 0
eq_tapq:anon_2964:num_checkpoint_start: 462
eq_tapq:anon_2964:num_checkpoint_start_failed: 0
eq_tapq:anon_2964:num_delete: 0
eq_tapq:anon_2964:num_delete_failed: 0
eq_tapq:anon_2964:num_flush: 0
eq_tapq:anon_2964:num_flush_failed: 0
eq_tapq:anon_2964:num_mutation: 129088
eq_tapq:anon_2964:num_mutation_failed: 19767
eq_tapq:anon_2964:num_opaque: 10
eq_tapq:anon_2964:num_opaque_failed: 0
eq_tapq:anon_2964:num_unknown: 0
eq_tapq:anon_2964:num_vbucket_set: 0
eq_tapq:anon_2964:num_vbucket_set_failed: 0
eq_tapq:anon_2964:pending_disconnect: false
eq_tapq:anon_2964:reserved: 0
eq_tapq:anon_2964:supports_ack: true
eq_tapq:anon_2964:type: consumer
eq_tapq:replication_building_57_'ns_1@10.1.3.155':ack_log_size: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':ack_playback_size: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':ack_seqno: 478
eq_tapq:replication_building_57_'ns_1@10.1.3.155':ack_window_full: false
eq_tapq:replication_building_57_'ns_1@10.1.3.155':backfill_completed: false
eq_tapq:replication_building_57_'ns_1@10.1.3.155':bg_backlog_size: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':bg_jobs_completed: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':bg_jobs_issued: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':bg_queued: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':bg_result_size: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':bg_results: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':bg_wait_for_results: false
eq_tapq:replication_building_57_'ns_1@10.1.3.155':complete: false
eq_tapq:replication_building_57_'ns_1@10.1.3.155':connected: true
eq_tapq:replication_building_57_'ns_1@10.1.3.155':created: 1985
eq_tapq:replication_building_57_'ns_1@10.1.3.155':empty: false
eq_tapq:replication_building_57_'ns_1@10.1.3.155':flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_building_57_'ns_1@10.1.3.155':has_item: false
eq_tapq:replication_building_57_'ns_1@10.1.3.155':has_queued_item: true
eq_tapq:replication_building_57_'ns_1@10.1.3.155':idle: false
eq_tapq:replication_building_57_'ns_1@10.1.3.155':num_tap_nack: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':num_tap_tmpfail_survivors: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':paused: 1
eq_tapq:replication_building_57_'ns_1@10.1.3.155':pending_backfill: true
eq_tapq:replication_building_57_'ns_1@10.1.3.155':pending_disconnect: false
eq_tapq:replication_building_57_'ns_1@10.1.3.155':pending_disk_backfill: true
eq_tapq:replication_building_57_'ns_1@10.1.3.155':qlen: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':qlen_high_pri: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':qlen_low_pri: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':queue_backfillremaining: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':queue_backoff: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':queue_drain: 471
eq_tapq:replication_building_57_'ns_1@10.1.3.155':queue_fill: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':queue_itemondisk: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':queue_memory: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':rec_fetched: 474
eq_tapq:replication_building_57_'ns_1@10.1.3.155':recv_ack_seqno: 477
eq_tapq:replication_building_57_'ns_1@10.1.3.155':reserved: 1
eq_tapq:replication_building_57_'ns_1@10.1.3.155':seqno_ack_requested: 477
eq_tapq:replication_building_57_'ns_1@10.1.3.155':supports_ack: true
eq_tapq:replication_building_57_'ns_1@10.1.3.155':suspended: false
eq_tapq:replication_building_57_'ns_1@10.1.3.155':total_backlog_size: 1
eq_tapq:replication_building_57_'ns_1@10.1.3.155':total_noops: 382
eq_tapq:replication_building_57_'ns_1@10.1.3.155':type: producer
eq_tapq:replication_building_57_'ns_1@10.1.3.155':vb_filter:

{ 57 }
eq_tapq:replication_building_57_'ns_1@10.1.3.155':vb_filters: 1
eq_tapq:replication_building_57_'ns_1@10.1.3.156':ack_log_size: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':ack_playback_size: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':ack_seqno: 478
eq_tapq:replication_building_57_'ns_1@10.1.3.156':ack_window_full: false
eq_tapq:replication_building_57_'ns_1@10.1.3.156':backfill_completed: false
eq_tapq:replication_building_57_'ns_1@10.1.3.156':bg_backlog_size: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':bg_jobs_completed: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':bg_jobs_issued: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':bg_queued: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':bg_result_size: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':bg_results: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':bg_wait_for_results: false
eq_tapq:replication_building_57_'ns_1@10.1.3.156':complete: false
eq_tapq:replication_building_57_'ns_1@10.1.3.156':connected: true
eq_tapq:replication_building_57_'ns_1@10.1.3.156':created: 1985
eq_tapq:replication_building_57_'ns_1@10.1.3.156':empty: false
eq_tapq:replication_building_57_'ns_1@10.1.3.156':flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_building_57_'ns_1@10.1.3.156':has_item: false
eq_tapq:replication_building_57_'ns_1@10.1.3.156':has_queued_item: true
eq_tapq:replication_building_57_'ns_1@10.1.3.156':idle: false
eq_tapq:replication_building_57_'ns_1@10.1.3.156':num_tap_nack: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':num_tap_tmpfail_survivors: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':paused: 1
eq_tapq:replication_building_57_'ns_1@10.1.3.156':pending_backfill: true
eq_tapq:replication_building_57_'ns_1@10.1.3.156':pending_disconnect: false
eq_tapq:replication_building_57_'ns_1@10.1.3.156':pending_disk_backfill: true
eq_tapq:replication_building_57_'ns_1@10.1.3.156':qlen: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':qlen_high_pri: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':qlen_low_pri: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':queue_backfillremaining: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':queue_backoff: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':queue_drain: 471
eq_tapq:replication_building_57_'ns_1@10.1.3.156':queue_fill: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':queue_itemondisk: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':queue_memory: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':rec_fetched: 474
eq_tapq:replication_building_57_'ns_1@10.1.3.156':recv_ack_seqno: 477
eq_tapq:replication_building_57_'ns_1@10.1.3.156':reserved: 1
eq_tapq:replication_building_57_'ns_1@10.1.3.156':seqno_ack_requested: 477
eq_tapq:replication_building_57_'ns_1@10.1.3.156':supports_ack: true
eq_tapq:replication_building_57_'ns_1@10.1.3.156':suspended: false
eq_tapq:replication_building_57_'ns_1@10.1.3.156':total_backlog_size: 1
eq_tapq:replication_building_57_'ns_1@10.1.3.156':total_noops: 382
eq_tapq:replication_building_57_'ns_1@10.1.3.156':type: producer
eq_tapq:replication_building_57_'ns_1@10.1.3.156':vb_filter: { 57 }

eq_tapq:replication_building_57_'ns_1@10.1.3.156':vb_filters: 1
eq_tapq:replication_ns_1@10.1.3.153:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.153:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.153:ack_seqno: 4134621
eq_tapq:replication_ns_1@10.1.3.153:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.153:backfill_completed: false
eq_tapq:replication_ns_1@10.1.3.153:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.153:bg_jobs_completed: 3391241
eq_tapq:replication_ns_1@10.1.3.153:bg_jobs_issued: 3391241
eq_tapq:replication_ns_1@10.1.3.153:bg_queued: 3391241
eq_tapq:replication_ns_1@10.1.3.153:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.153:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.153:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.153:complete: false
eq_tapq:replication_ns_1@10.1.3.153:connected: true
eq_tapq:replication_ns_1@10.1.3.153:created: 875
eq_tapq:replication_ns_1@10.1.3.153:disconnects: 228
eq_tapq:replication_ns_1@10.1.3.153:empty: false
eq_tapq:replication_ns_1@10.1.3.153:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.153:has_item: false
eq_tapq:replication_ns_1@10.1.3.153:has_queued_item: true
eq_tapq:replication_ns_1@10.1.3.153:idle: false
eq_tapq:replication_ns_1@10.1.3.153:num_tap_nack: 56234
eq_tapq:replication_ns_1@10.1.3.153:num_tap_tmpfail_survivors: 56234
eq_tapq:replication_ns_1@10.1.3.153:paused: 1
eq_tapq:replication_ns_1@10.1.3.153:pending_backfill: true
eq_tapq:replication_ns_1@10.1.3.153:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.153:pending_disk_backfill: true
eq_tapq:replication_ns_1@10.1.3.153:qlen: 0
eq_tapq:replication_ns_1@10.1.3.153:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.153:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.153:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.153:queue_backoff: 56234
eq_tapq:replication_ns_1@10.1.3.153:queue_drain: 4131591
eq_tapq:replication_ns_1@10.1.3.153:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.153:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.153:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.153:rec_fetched: 2505461
eq_tapq:replication_ns_1@10.1.3.153:reconnects: 228
eq_tapq:replication_ns_1@10.1.3.153:recv_ack_seqno: 4134620
eq_tapq:replication_ns_1@10.1.3.153:reserved: 1
eq_tapq:replication_ns_1@10.1.3.153:seqno_ack_requested: 4134620
eq_tapq:replication_ns_1@10.1.3.153:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.153:suspended: false
eq_tapq:replication_ns_1@10.1.3.153:total_backlog_size: 114
eq_tapq:replication_ns_1@10.1.3.153:total_noops: 427
eq_tapq:replication_ns_1@10.1.3.153:type: producer
eq_tapq:replication_ns_1@10.1.3.153:vb_filter:

{ [57,170] }

eq_tapq:replication_ns_1@10.1.3.153:vb_filters: 114
eq_tapq:replication_ns_1@10.1.3.155:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.155:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.155:ack_seqno: 646303
eq_tapq:replication_ns_1@10.1.3.155:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.155:backfill_completed: false
eq_tapq:replication_ns_1@10.1.3.155:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.155:bg_jobs_completed: 552990
eq_tapq:replication_ns_1@10.1.3.155:bg_jobs_issued: 552990
eq_tapq:replication_ns_1@10.1.3.155:bg_queued: 552990
eq_tapq:replication_ns_1@10.1.3.155:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.155:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.155:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.155:complete: false
eq_tapq:replication_ns_1@10.1.3.155:connected: true
eq_tapq:replication_ns_1@10.1.3.155:created: 1695
eq_tapq:replication_ns_1@10.1.3.155:empty: false
eq_tapq:replication_ns_1@10.1.3.155:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.155:has_item: false
eq_tapq:replication_ns_1@10.1.3.155:has_queued_item: true
eq_tapq:replication_ns_1@10.1.3.155:idle: false
eq_tapq:replication_ns_1@10.1.3.155:num_tap_nack: 77185
eq_tapq:replication_ns_1@10.1.3.155:num_tap_tmpfail_survivors: 77185
eq_tapq:replication_ns_1@10.1.3.155:paused: 1
eq_tapq:replication_ns_1@10.1.3.155:pending_backfill: true
eq_tapq:replication_ns_1@10.1.3.155:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.155:pending_disk_backfill: true
eq_tapq:replication_ns_1@10.1.3.155:qlen: 0
eq_tapq:replication_ns_1@10.1.3.155:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.155:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.155:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.155:queue_backoff: 77185
eq_tapq:replication_ns_1@10.1.3.155:queue_drain: 646120
eq_tapq:replication_ns_1@10.1.3.155:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.155:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.155:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.155:rec_fetched: 366680
eq_tapq:replication_ns_1@10.1.3.155:recv_ack_seqno: 646302
eq_tapq:replication_ns_1@10.1.3.155:reserved: 1
eq_tapq:replication_ns_1@10.1.3.155:seqno_ack_requested: 646302
eq_tapq:replication_ns_1@10.1.3.155:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.155:suspended: false
eq_tapq:replication_ns_1@10.1.3.155:total_backlog_size: 12129
eq_tapq:replication_ns_1@10.1.3.155:total_noops: 396
eq_tapq:replication_ns_1@10.1.3.155:type: producer
eq_tapq:replication_ns_1@10.1.3.155:vb_filter:

{ [171,512] }

eq_tapq:replication_ns_1@10.1.3.155:vb_filters: 342
eq_tapq:replication_ns_1@10.1.3.156:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.156:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.156:ack_seqno: 599443
eq_tapq:replication_ns_1@10.1.3.156:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.156:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.156:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.156:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.156:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.156:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.156:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.156:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.156:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.156:complete: false
eq_tapq:replication_ns_1@10.1.3.156:connected: true
eq_tapq:replication_ns_1@10.1.3.156:created: 1177
eq_tapq:replication_ns_1@10.1.3.156:disconnects: 227
eq_tapq:replication_ns_1@10.1.3.156:empty: true
eq_tapq:replication_ns_1@10.1.3.156:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.156:has_item: false
eq_tapq:replication_ns_1@10.1.3.156:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.156:idle: true
eq_tapq:replication_ns_1@10.1.3.156:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.156:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.156:paused: 1
eq_tapq:replication_ns_1@10.1.3.156:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.156:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.156:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.156:qlen: 0
eq_tapq:replication_ns_1@10.1.3.156:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.156:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.156:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.156:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.156:queue_drain: 597192
eq_tapq:replication_ns_1@10.1.3.156:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.156:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.156:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.156:rec_fetched: 598754
eq_tapq:replication_ns_1@10.1.3.156:reconnects: 227
eq_tapq:replication_ns_1@10.1.3.156:recv_ack_seqno: 599442
eq_tapq:replication_ns_1@10.1.3.156:reserved: 1
eq_tapq:replication_ns_1@10.1.3.156:seqno_ack_requested: 599442
eq_tapq:replication_ns_1@10.1.3.156:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.156:suspended: false
eq_tapq:replication_ns_1@10.1.3.156:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.156:total_noops: 415
eq_tapq:replication_ns_1@10.1.3.156:type: producer
eq_tapq:replication_ns_1@10.1.3.156:vb_filter:

{ [0,56], [683,853] }

eq_tapq:replication_ns_1@10.1.3.156:vb_filters: 228

10.1.3.153
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 735193
ep_tap_bg_load_avg: 222
ep_tap_bg_max_load: 56284
ep_tap_bg_max_pending: 500
ep_tap_bg_max_wait: 66955
ep_tap_bg_min_load: 13
ep_tap_bg_min_wait: 28
ep_tap_bg_num_samples: 1061
ep_tap_bg_wait_avg: 18209
ep_tap_count: 3
ep_tap_deletes: 0
ep_tap_fg_fetched: 28835182
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 0
ep_tap_queue_drain: 97884
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 128230
ep_tap_total_backlog_size: 0
ep_tap_total_fetched: 29604660
ep_tap_total_queue: 0
eq_tapq:anon_1673:connected: true
eq_tapq:anon_1673:created: 1064
eq_tapq:anon_1673:num_checkpoint_end: 0
eq_tapq:anon_1673:num_checkpoint_end_failed: 0
eq_tapq:anon_1673:num_checkpoint_start: 0
eq_tapq:anon_1673:num_checkpoint_start_failed: 0
eq_tapq:anon_1673:num_delete: 0
eq_tapq:anon_1673:num_delete_failed: 0
eq_tapq:anon_1673:num_flush: 0
eq_tapq:anon_1673:num_flush_failed: 0
eq_tapq:anon_1673:num_mutation: 212971
eq_tapq:anon_1673:num_mutation_failed: 3068
eq_tapq:anon_1673:num_opaque: 2
eq_tapq:anon_1673:num_opaque_failed: 0
eq_tapq:anon_1673:num_unknown: 0
eq_tapq:anon_1673:num_vbucket_set: 0
eq_tapq:anon_1673:num_vbucket_set_failed: 0
eq_tapq:anon_1673:pending_disconnect: false
eq_tapq:anon_1673:reserved: 0
eq_tapq:anon_1673:supports_ack: true
eq_tapq:anon_1673:type: consumer
eq_tapq:anon_1967:connected: true
eq_tapq:anon_1967:created: 1164
eq_tapq:anon_1967:num_checkpoint_end: 0
eq_tapq:anon_1967:num_checkpoint_end_failed: 0
eq_tapq:anon_1967:num_checkpoint_start: 0
eq_tapq:anon_1967:num_checkpoint_start_failed: 0
eq_tapq:anon_1967:num_delete: 0
eq_tapq:anon_1967:num_delete_failed: 0
eq_tapq:anon_1967:num_flush: 0
eq_tapq:anon_1967:num_flush_failed: 0
eq_tapq:anon_1967:num_mutation: 453851
eq_tapq:anon_1967:num_mutation_failed: 0
eq_tapq:anon_1967:num_opaque: 2
eq_tapq:anon_1967:num_opaque_failed: 0
eq_tapq:anon_1967:num_unknown: 0
eq_tapq:anon_1967:num_vbucket_set: 0
eq_tapq:anon_1967:num_vbucket_set_failed: 0
eq_tapq:anon_1967:pending_disconnect: false
eq_tapq:anon_1967:reserved: 0
eq_tapq:anon_1967:supports_ack: true
eq_tapq:anon_1967:type: consumer
eq_tapq:replication_ns_1@10.1.3.155:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.155:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.155:ack_seqno: 98014
eq_tapq:replication_ns_1@10.1.3.155:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.155:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.155:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.155:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.155:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.155:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.155:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.155:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.155:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.155:complete: false
eq_tapq:replication_ns_1@10.1.3.155:connected: true
eq_tapq:replication_ns_1@10.1.3.155:created: 9157
eq_tapq:replication_ns_1@10.1.3.155:empty: true
eq_tapq:replication_ns_1@10.1.3.155:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.155:has_item: false
eq_tapq:replication_ns_1@10.1.3.155:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.155:idle: true
eq_tapq:replication_ns_1@10.1.3.155:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.155:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.155:paused: 1
eq_tapq:replication_ns_1@10.1.3.155:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.155:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.155:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.155:qlen: 0
eq_tapq:replication_ns_1@10.1.3.155:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.155:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.155:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.155:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.155:queue_drain: 97884
eq_tapq:replication_ns_1@10.1.3.155:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.155:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.155:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.155:rec_fetched: 97961
eq_tapq:replication_ns_1@10.1.3.155:recv_ack_seqno: 98013
eq_tapq:replication_ns_1@10.1.3.155:reserved: 1
eq_tapq:replication_ns_1@10.1.3.155:seqno_ack_requested: 98013
eq_tapq:replication_ns_1@10.1.3.155:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.155:suspended: false
eq_tapq:replication_ns_1@10.1.3.155:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.155:total_noops: 1
eq_tapq:replication_ns_1@10.1.3.155:type: producer
eq_tapq:replication_ns_1@10.1.3.155:vb_filter:

{ [146,170] }

eq_tapq:replication_ns_1@10.1.3.155:vb_filters: 25

10.1.3.155
ep_tap_ack_grace_period: 300
ep_tap_ack_interval: 1000
ep_tap_ack_window_size: 10
ep_tap_backoff_period: 5
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 305206
ep_tap_bg_max_pending: 500
ep_tap_count: 7
ep_tap_deletes: 0
ep_tap_fg_fetched: 3139520
ep_tap_keepalive: 300
ep_tap_noop_interval: 20
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 183006
ep_tap_queue_drain: 2512606
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 1000000
ep_tap_throttle_threshold: 90
ep_tap_throttled: 307726
ep_tap_total_backlog_size: 0
ep_tap_total_fetched: 3464616
ep_tap_total_queue: 0
eq_tapq:anon_1588:connected: true
eq_tapq:anon_1588:created: 875
eq_tapq:anon_1588:num_checkpoint_end: 0
eq_tapq:anon_1588:num_checkpoint_end_failed: 0
eq_tapq:anon_1588:num_checkpoint_start: 0
eq_tapq:anon_1588:num_checkpoint_start_failed: 0
eq_tapq:anon_1588:num_delete: 0
eq_tapq:anon_1588:num_delete_failed: 0
eq_tapq:anon_1588:num_flush: 0
eq_tapq:anon_1588:num_flush_failed: 0
eq_tapq:anon_1588:num_mutation: 568935
eq_tapq:anon_1588:num_mutation_failed: 77185
eq_tapq:anon_1588:num_opaque: 91
eq_tapq:anon_1588:num_opaque_failed: 0
eq_tapq:anon_1588:num_unknown: 0
eq_tapq:anon_1588:num_vbucket_set: 0
eq_tapq:anon_1588:num_vbucket_set_failed: 0
eq_tapq:anon_1588:pending_disconnect: false
eq_tapq:anon_1588:reserved: 0
eq_tapq:anon_1588:supports_ack: true
eq_tapq:anon_1588:type: consumer
eq_tapq:anon_2868:connected: true
eq_tapq:anon_2868:created: 1165
eq_tapq:anon_2868:num_checkpoint_end: 0
eq_tapq:anon_2868:num_checkpoint_end_failed: 0
eq_tapq:anon_2868:num_checkpoint_start: 0
eq_tapq:anon_2868:num_checkpoint_start_failed: 0
eq_tapq:anon_2868:num_delete: 0
eq_tapq:anon_2868:num_delete_failed: 0
eq_tapq:anon_2868:num_flush: 0
eq_tapq:anon_2868:num_flush_failed: 0
eq_tapq:anon_2868:num_mutation: 471
eq_tapq:anon_2868:num_mutation_failed: 0
eq_tapq:anon_2868:num_opaque: 3
eq_tapq:anon_2868:num_opaque_failed: 0
eq_tapq:anon_2868:num_unknown: 0
eq_tapq:anon_2868:num_vbucket_set: 0
eq_tapq:anon_2868:num_vbucket_set_failed: 0
eq_tapq:anon_2868:pending_disconnect: false
eq_tapq:anon_2868:reserved: 0
eq_tapq:anon_2868:supports_ack: true
eq_tapq:anon_2868:type: consumer
eq_tapq:anon_2907:connected: true
eq_tapq:anon_2907:created: 1176
eq_tapq:anon_2907:num_checkpoint_end: 170
eq_tapq:anon_2907:num_checkpoint_end_failed: 0
eq_tapq:anon_2907:num_checkpoint_start: 171
eq_tapq:anon_2907:num_checkpoint_start_failed: 0
eq_tapq:anon_2907:num_delete: 0
eq_tapq:anon_2907:num_delete_failed: 0
eq_tapq:anon_2907:num_flush: 0
eq_tapq:anon_2907:num_flush_failed: 0
eq_tapq:anon_2907:num_mutation: 256
eq_tapq:anon_2907:num_mutation_failed: 0
eq_tapq:anon_2907:num_opaque: 2
eq_tapq:anon_2907:num_opaque_failed: 0
eq_tapq:anon_2907:num_unknown: 0
eq_tapq:anon_2907:num_vbucket_set: 0
eq_tapq:anon_2907:num_vbucket_set_failed: 0
eq_tapq:anon_2907:pending_disconnect: false
eq_tapq:anon_2907:reserved: 0
eq_tapq:anon_2907:supports_ack: true
eq_tapq:anon_2907:type: consumer
eq_tapq:anon_3173:connected: true
eq_tapq:anon_3173:created: 9189
eq_tapq:anon_3173:num_checkpoint_end: 0
eq_tapq:anon_3173:num_checkpoint_end_failed: 0
eq_tapq:anon_3173:num_checkpoint_start: 0
eq_tapq:anon_3173:num_checkpoint_start_failed: 0
eq_tapq:anon_3173:num_delete: 0
eq_tapq:anon_3173:num_delete_failed: 0
eq_tapq:anon_3173:num_flush: 0
eq_tapq:anon_3173:num_flush_failed: 0
eq_tapq:anon_3173:num_mutation: 48942
eq_tapq:anon_3173:num_mutation_failed: 0
eq_tapq:anon_3173:num_opaque: 27
eq_tapq:anon_3173:num_opaque_failed: 0
eq_tapq:anon_3173:num_unknown: 0
eq_tapq:anon_3173:num_vbucket_set: 0
eq_tapq:anon_3173:num_vbucket_set_failed: 0
eq_tapq:anon_3173:pending_disconnect: false
eq_tapq:anon_3173:reserved: 0
eq_tapq:anon_3173:supports_ack: true
eq_tapq:anon_3173:type: consumer
eq_tapq:replication_ns_1@10.1.3.152:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.152:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.152:ack_seqno: 982349
eq_tapq:replication_ns_1@10.1.3.152:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.152:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.152:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.152:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.152:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.152:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.152:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.152:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.152:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.152:complete: false
eq_tapq:replication_ns_1@10.1.3.152:connected: true
eq_tapq:replication_ns_1@10.1.3.152:created: 55
eq_tapq:replication_ns_1@10.1.3.152:disconnects: 682
eq_tapq:replication_ns_1@10.1.3.152:empty: true
eq_tapq:replication_ns_1@10.1.3.152:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.152:has_item: false
eq_tapq:replication_ns_1@10.1.3.152:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.152:idle: true
eq_tapq:replication_ns_1@10.1.3.152:num_tap_nack: 111589
eq_tapq:replication_ns_1@10.1.3.152:num_tap_tmpfail_survivors: 111589
eq_tapq:replication_ns_1@10.1.3.152:paused: 1
eq_tapq:replication_ns_1@10.1.3.152:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.152:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.152:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.152:qlen: 0
eq_tapq:replication_ns_1@10.1.3.152:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.152:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.152:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.152:queue_backoff: 111589
eq_tapq:replication_ns_1@10.1.3.152:queue_drain: 960739
eq_tapq:replication_ns_1@10.1.3.152:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.152:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.152:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.152:rec_fetched: 972975
eq_tapq:replication_ns_1@10.1.3.152:reconnects: 682
eq_tapq:replication_ns_1@10.1.3.152:recv_ack_seqno: 982348
eq_tapq:replication_ns_1@10.1.3.152:reserved: 1
eq_tapq:replication_ns_1@10.1.3.152:seqno_ack_requested: 982348
eq_tapq:replication_ns_1@10.1.3.152:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.152:suspended: false
eq_tapq:replication_ns_1@10.1.3.152:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.152:total_noops: 407
eq_tapq:replication_ns_1@10.1.3.152:type: producer
eq_tapq:replication_ns_1@10.1.3.152:vb_filter:

{ [513,853] }

eq_tapq:replication_ns_1@10.1.3.152:vb_filters: 341
eq_tapq:replication_ns_1@10.1.3.153:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.153:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.153:ack_seqno: 1552854
eq_tapq:replication_ns_1@10.1.3.153:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.153:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.153:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.153:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.153:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.153:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.153:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.153:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.153:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.153:complete: false
eq_tapq:replication_ns_1@10.1.3.153:connected: true
eq_tapq:replication_ns_1@10.1.3.153:created: 55
eq_tapq:replication_ns_1@10.1.3.153:disconnects: 170
eq_tapq:replication_ns_1@10.1.3.153:empty: true
eq_tapq:replication_ns_1@10.1.3.153:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.153:has_item: false
eq_tapq:replication_ns_1@10.1.3.153:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.153:idle: true
eq_tapq:replication_ns_1@10.1.3.153:num_tap_nack: 71417
eq_tapq:replication_ns_1@10.1.3.153:num_tap_tmpfail_survivors: 71417
eq_tapq:replication_ns_1@10.1.3.153:paused: 1
eq_tapq:replication_ns_1@10.1.3.153:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.153:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.153:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.153:qlen: 0
eq_tapq:replication_ns_1@10.1.3.153:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.153:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.153:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.153:queue_backoff: 71417
eq_tapq:replication_ns_1@10.1.3.153:queue_drain: 1547533
eq_tapq:replication_ns_1@10.1.3.153:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.153:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.153:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.153:rec_fetched: 1551293
eq_tapq:replication_ns_1@10.1.3.153:reconnects: 170
eq_tapq:replication_ns_1@10.1.3.153:recv_ack_seqno: 1552853
eq_tapq:replication_ns_1@10.1.3.153:reserved: 1
eq_tapq:replication_ns_1@10.1.3.153:seqno_ack_requested: 1552853
eq_tapq:replication_ns_1@10.1.3.153:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.153:suspended: false
eq_tapq:replication_ns_1@10.1.3.153:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.153:total_noops: 424
eq_tapq:replication_ns_1@10.1.3.153:type: producer
eq_tapq:replication_ns_1@10.1.3.153:vb_filter:

{ [171,341] }

eq_tapq:replication_ns_1@10.1.3.153:vb_filters: 171
eq_tapq:replication_ns_1@10.1.3.156:ack_log_size: 0
eq_tapq:replication_ns_1@10.1.3.156:ack_playback_size: 0
eq_tapq:replication_ns_1@10.1.3.156:ack_seqno: 5382
eq_tapq:replication_ns_1@10.1.3.156:ack_window_full: false
eq_tapq:replication_ns_1@10.1.3.156:backfill_completed: true
eq_tapq:replication_ns_1@10.1.3.156:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.156:bg_jobs_completed: 0
eq_tapq:replication_ns_1@10.1.3.156:bg_jobs_issued: 0
eq_tapq:replication_ns_1@10.1.3.156:bg_queued: 0
eq_tapq:replication_ns_1@10.1.3.156:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.156:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.156:bg_wait_for_results: false
eq_tapq:replication_ns_1@10.1.3.156:complete: false
eq_tapq:replication_ns_1@10.1.3.156:connected: true
eq_tapq:replication_ns_1@10.1.3.156:created: 755
eq_tapq:replication_ns_1@10.1.3.156:disconnects: 169
eq_tapq:replication_ns_1@10.1.3.156:empty: true
eq_tapq:replication_ns_1@10.1.3.156:flags: 85 (ack,backfill,vblist,checkpoints)
eq_tapq:replication_ns_1@10.1.3.156:has_item: false
eq_tapq:replication_ns_1@10.1.3.156:has_queued_item: false
eq_tapq:replication_ns_1@10.1.3.156:idle: true
eq_tapq:replication_ns_1@10.1.3.156:num_tap_nack: 0
eq_tapq:replication_ns_1@10.1.3.156:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@10.1.3.156:paused: 1
eq_tapq:replication_ns_1@10.1.3.156:pending_backfill: false
eq_tapq:replication_ns_1@10.1.3.156:pending_disconnect: false
eq_tapq:replication_ns_1@10.1.3.156:pending_disk_backfill: false
eq_tapq:replication_ns_1@10.1.3.156:qlen: 0
eq_tapq:replication_ns_1@10.1.3.156:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.156:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.156:queue_backfillremaining: 0
eq_tapq:replication_ns_1@10.1.3.156:queue_backoff: 0
eq_tapq:replication_ns_1@10.1.3.156:queue_drain: 4334
eq_tapq:replication_ns_1@10.1.3.156:queue_fill: 0
eq_tapq:replication_ns_1@10.1.3.156:queue_itemondisk: 0
eq_tapq:replication_ns_1@10.1.3.156:queue_memory: 0
eq_tapq:replication_ns_1@10.1.3.156:rec_fetched: 5037
eq_tapq:replication_ns_1@10.1.3.156:reconnects: 169
eq_tapq:replication_ns_1@10.1.3.156:recv_ack_seqno: 5381
eq_tapq:replication_ns_1@10.1.3.156:reserved: 1
eq_tapq:replication_ns_1@10.1.3.156:seqno_ack_requested: 5381
eq_tapq:replication_ns_1@10.1.3.156:supports_ack: true
eq_tapq:replication_ns_1@10.1.3.156:suspended: false
eq_tapq:replication_ns_1@10.1.3.156:total_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.156:total_noops: 393
eq_tapq:replication_ns_1@10.1.3.156:type: producer
eq_tapq:replication_ns_1@10.1.3.156:vb_filter:

{ [854,1023] }

eq_tapq:replication_ns_1@10.1.3.156:vb_filters: 170

ep_tap_total_backlog_size: 12245
eq_tapq:replication_building_57_'ns_1@10.1.3.155':bg_backlog_size: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':bg_result_size: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':bg_results: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':qlen: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':qlen_high_pri: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':qlen_low_pri: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.155':total_backlog_size: 1
eq_tapq:replication_building_57_'ns_1@10.1.3.156':bg_backlog_size: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':bg_result_size: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':bg_results: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':qlen: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':qlen_high_pri: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':qlen_low_pri: 0
eq_tapq:replication_building_57_'ns_1@10.1.3.156':total_backlog_size: 1
eq_tapq:replication_ns_1@10.1.3.153:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.153:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.153:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.153:qlen: 0
eq_tapq:replication_ns_1@10.1.3.153:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.153:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.153:total_backlog_size: 114
eq_tapq:replication_ns_1@10.1.3.155:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.155:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.155:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.155:qlen: 0
eq_tapq:replication_ns_1@10.1.3.155:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.155:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.155:total_backlog_size: 12129
eq_tapq:replication_ns_1@10.1.3.156:bg_backlog_size: 0
eq_tapq:replication_ns_1@10.1.3.156:bg_result_size: 0
eq_tapq:replication_ns_1@10.1.3.156:bg_results: 0
eq_tapq:replication_ns_1@10.1.3.156:qlen: 0
eq_tapq:replication_ns_1@10.1.3.156:qlen_high_pri: 0
eq_tapq:replication_ns_1@10.1.3.156:qlen_low_pri: 0
eq_tapq:replication_ns_1@10.1.3.156:total_backlog_size: 0

10.1.3.156
ep_kv_size: 653455224
ep_max_data_size: 1435500544
ep_mem_high_wat: 1076625408
ep_mem_low_wat: 861300326
ep_oom_errors: 0
ep_overhead: 19487152
ep_tmp_oom_errors: 0
ep_value_size: 522819352
mem_used: 758283888
tcmalloc_current_thread_cache_bytes: 2164376
tcmalloc_max_thread_cache_bytes: 4194304
tcmalloc_unmapped_bytes: 0
total_allocated_bytes: 762787240
total_fragmentation_bytes: 35236440
total_free_bytes: 104800256
total_heap_bytes: 902823936

10.1.3.153
ep_kv_size: 246443038
ep_max_data_size: 1435500544
ep_mem_high_wat: 1076625408
ep_mem_low_wat: 861300326
ep_oom_errors: 0
ep_overhead: 32395368
ep_tmp_oom_errors: 0
ep_value_size: 198040173
mem_used: 377805888
tcmalloc_current_thread_cache_bytes: 3309480
tcmalloc_max_thread_cache_bytes: 4194304
tcmalloc_unmapped_bytes: 3006464
total_allocated_bytes: 382131688
total_fragmentation_bytes: 454566424
total_free_bytes: 369164288
total_heap_bytes: 1205862400

10.1.3.152
ep_kv_size: 913724022
ep_max_data_size: 1435500544
ep_mem_high_wat: 1076625408
ep_mem_low_wat: 861300326
ep_oom_errors: 0
ep_overhead: 194621520
ep_tmp_oom_errors: 0
ep_value_size: 731748257
mem_used: 1292531856
tcmalloc_current_thread_cache_bytes: 2850936
tcmalloc_max_thread_cache_bytes: 4194304
tcmalloc_unmapped_bytes: 0
total_allocated_bytes: 1297049080
total_fragmentation_bytes: 164592136
total_free_bytes: 35725312
total_heap_bytes: 1497366528

10.1.3.155
ep_kv_size: 621349543
ep_max_data_size: 1435500544
ep_mem_high_wat: 1076625408
ep_mem_low_wat: 861300326
ep_oom_errors: 0
ep_overhead: 69707200
ep_tmp_oom_errors: 0
ep_value_size: 446423673
mem_used: 834486528
tcmalloc_current_thread_cache_bytes: 2350920
tcmalloc_max_thread_cache_bytes: 4194304
tcmalloc_unmapped_bytes: 540672
total_allocated_bytes: 838814872
total_fragmentation_bytes: 352744296
total_free_bytes: 54149120
total_heap_bytes: 1245708288