Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7214

[windows] 7k/9M items are lost after failing over one node and rebooting another one in a 4 node cluster after replication was completed ( items re-appear upon 2nd reboot )

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Cannot Reproduce
    • Affects Version/s: 2.0-beta
    • Fix Version/s: 2.0.1
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
      None
    • Environment:

      Description

      4 nodes cluster/1 bucket /9 M items/ 1 ddoc/2 views

      autofailover is enabled

      2 steps to reproduce:

      1) rebooted .135 and .147 nodes -> one of them(.135) is failed over as expected
      after waiting for warmups 7K are lost
      I added back the node and rebalanced cluster
      tap stats now are:
      [root@localhost bin]# ./cbstats 10.1.3.135:11210 tap
      ep_tap_ack_grace_period: 300
      ep_tap_ack_interval: 1000
      ep_tap_ack_window_size: 10
      ep_tap_backoff_period: 5
      ep_tap_bg_fetch_requeued: 0
      ep_tap_bg_fetched: 0
      ep_tap_bg_max_pending: 500
      ep_tap_count: 6
      ep_tap_deletes: 0
      ep_tap_fg_fetched: 0
      ep_tap_noop_interval: 20
      ep_tap_queue_backfillremaining: 0
      ep_tap_queue_backoff: 0
      ep_tap_queue_drain: 0
      ep_tap_queue_fill: 0
      ep_tap_queue_itemondisk: 0
      ep_tap_throttle_queue_cap: 1000000
      ep_tap_throttle_threshold: 90
      ep_tap_throttled: 0
      ep_tap_total_backlog_size: 0
      ep_tap_total_fetched: 262
      ep_tap_total_queue: 0
      eq_tapq:anon_1:connected: true
      eq_tapq:anon_1:created: 125
      eq_tapq:anon_1:num_checkpoint_end: 0
      eq_tapq:anon_1:num_checkpoint_end_failed: 0
      eq_tapq:anon_1:num_checkpoint_start: 85
      eq_tapq:anon_1:num_checkpoint_start_failed: 0
      eq_tapq:anon_1:num_delete: 0
      eq_tapq:anon_1:num_delete_failed: 0
      eq_tapq:anon_1:num_flush: 0
      eq_tapq:anon_1:num_flush_failed: 0
      eq_tapq:anon_1:num_mutation: 0
      eq_tapq:anon_1:num_mutation_failed: 0
      eq_tapq:anon_1:num_opaque: 2
      eq_tapq:anon_1:num_opaque_failed: 0
      eq_tapq:anon_1:num_unknown: 0
      eq_tapq:anon_1:num_vbucket_set: 0
      eq_tapq:anon_1:num_vbucket_set_failed: 0
      eq_tapq:anon_1:pending_disconnect: false
      eq_tapq:anon_1:reserved: 0
      eq_tapq:anon_1:supports_ack: true
      eq_tapq:anon_1:type: consumer
      eq_tapq:anon_2:connected: true
      eq_tapq:anon_2:created: 125
      eq_tapq:anon_2:num_checkpoint_end: 0
      eq_tapq:anon_2:num_checkpoint_end_failed: 0
      eq_tapq:anon_2:num_checkpoint_start: 86
      eq_tapq:anon_2:num_checkpoint_start_failed: 0
      eq_tapq:anon_2:num_delete: 0
      eq_tapq:anon_2:num_delete_failed: 0
      eq_tapq:anon_2:num_flush: 0
      eq_tapq:anon_2:num_flush_failed: 0
      eq_tapq:anon_2:num_mutation: 0
      eq_tapq:anon_2:num_mutation_failed: 0
      eq_tapq:anon_2:num_opaque: 2
      eq_tapq:anon_2:num_opaque_failed: 0
      eq_tapq:anon_2:num_unknown: 0
      eq_tapq:anon_2:num_vbucket_set: 0
      eq_tapq:anon_2:num_vbucket_set_failed: 0
      eq_tapq:anon_2:pending_disconnect: false
      eq_tapq:anon_2:reserved: 0
      eq_tapq:anon_2:supports_ack: true
      eq_tapq:anon_2:type: consumer
      eq_tapq:anon_3:connected: true
      eq_tapq:anon_3:created: 125
      eq_tapq:anon_3:num_checkpoint_end: 0
      eq_tapq:anon_3:num_checkpoint_end_failed: 0
      eq_tapq:anon_3:num_checkpoint_start: 85
      eq_tapq:anon_3:num_checkpoint_start_failed: 0
      eq_tapq:anon_3:num_delete: 0
      eq_tapq:anon_3:num_delete_failed: 0
      eq_tapq:anon_3:num_flush: 0
      eq_tapq:anon_3:num_flush_failed: 0
      eq_tapq:anon_3:num_mutation: 0
      eq_tapq:anon_3:num_mutation_failed: 0
      eq_tapq:anon_3:num_opaque: 2
      eq_tapq:anon_3:num_opaque_failed: 0
      eq_tapq:anon_3:num_unknown: 0
      eq_tapq:anon_3:num_vbucket_set: 0
      eq_tapq:anon_3:num_vbucket_set_failed: 0
      eq_tapq:anon_3:pending_disconnect: false
      eq_tapq:anon_3:reserved: 0
      eq_tapq:anon_3:supports_ack: true
      eq_tapq:anon_3:type: consumer
      eq_tapq:replication_ns_1@10.1.3.146:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.146:ack_seqno: 91
      eq_tapq:replication_ns_1@10.1.3.146:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.146:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.146:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.146:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.146:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.146:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.146:connected: true
      eq_tapq:replication_ns_1@10.1.3.146:created: 126
      eq_tapq:replication_ns_1@10.1.3.146:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.146:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.146:idle: true
      eq_tapq:replication_ns_1@10.1.3.146:paused: 1
      eq_tapq:replication_ns_1@10.1.3.146:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.146:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.146:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.146:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.146:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.146:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.146:rec_fetched: 88
      eq_tapq:replication_ns_1@10.1.3.146:recv_ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.146:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.146:seqno_ack_requested: 90
      eq_tapq:replication_ns_1@10.1.3.146:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.146:suspended: false
      eq_tapq:replication_ns_1@10.1.3.146:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.146:total_noops: 1
      eq_tapq:replication_ns_1@10.1.3.146:type: producer
      eq_tapq:replication_ns_1@10.1.3.146:vb_filter:

      { [0,85] }

      eq_tapq:replication_ns_1@10.1.3.146:vb_filters: 86
      eq_tapq:replication_ns_1@10.1.3.147:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.147:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.147:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.147:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:connected: true
      eq_tapq:replication_ns_1@10.1.3.147:created: 126
      eq_tapq:replication_ns_1@10.1.3.147:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.147:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.147:idle: true
      eq_tapq:replication_ns_1@10.1.3.147:paused: 1
      eq_tapq:replication_ns_1@10.1.3.147:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.147:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.147:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.147:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.147:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.147:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.147:rec_fetched: 87
      eq_tapq:replication_ns_1@10.1.3.147:recv_ack_seqno: 89
      eq_tapq:replication_ns_1@10.1.3.147:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.147:seqno_ack_requested: 89
      eq_tapq:replication_ns_1@10.1.3.147:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.147:suspended: false
      eq_tapq:replication_ns_1@10.1.3.147:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:total_noops: 1
      eq_tapq:replication_ns_1@10.1.3.147:type: producer
      eq_tapq:replication_ns_1@10.1.3.147:vb_filter:

      { [86,170] }

      eq_tapq:replication_ns_1@10.1.3.147:vb_filters: 85
      eq_tapq:replication_ns_1@10.1.3.151:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.151:ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.151:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.151:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.151:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.151:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.151:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.151:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.151:connected: true
      eq_tapq:replication_ns_1@10.1.3.151:created: 125
      eq_tapq:replication_ns_1@10.1.3.151:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.151:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.151:idle: true
      eq_tapq:replication_ns_1@10.1.3.151:paused: 1
      eq_tapq:replication_ns_1@10.1.3.151:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.151:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.151:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.151:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.151:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.151:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.151:rec_fetched: 87
      eq_tapq:replication_ns_1@10.1.3.151:recv_ack_seqno: 89
      eq_tapq:replication_ns_1@10.1.3.151:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.151:seqno_ack_requested: 89
      eq_tapq:replication_ns_1@10.1.3.151:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.151:suspended: false
      eq_tapq:replication_ns_1@10.1.3.151:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.151:total_noops: 1
      eq_tapq:replication_ns_1@10.1.3.151:type: producer
      eq_tapq:replication_ns_1@10.1.3.151:vb_filter:

      { [171,255] }

      eq_tapq:replication_ns_1@10.1.3.151:vb_filters: 85

      [root@localhost bin]# ./cbstats 10.1.3.146:11210 tap
      ep_tap_ack_grace_period: 300
      ep_tap_ack_interval: 1000
      ep_tap_ack_window_size: 10
      ep_tap_backoff_period: 5
      ep_tap_bg_fetch_requeued: 0
      ep_tap_bg_fetched: 3076107
      ep_tap_bg_max_pending: 500
      ep_tap_count: 6
      ep_tap_deletes: 0
      ep_tap_fg_fetched: 3257334
      ep_tap_noop_interval: 20
      ep_tap_queue_backfillremaining: 0
      ep_tap_queue_backoff: 0
      ep_tap_queue_drain: 0
      ep_tap_queue_fill: 0
      ep_tap_queue_itemondisk: 0
      ep_tap_throttle_queue_cap: 1000000
      ep_tap_throttle_threshold: 90
      ep_tap_throttled: 0
      ep_tap_total_backlog_size: 0
      ep_tap_total_fetched: 6343257
      ep_tap_total_queue: 0
      eq_tapq:anon_182:connected: true
      eq_tapq:anon_182:created: 20593
      eq_tapq:anon_182:num_checkpoint_end: 0
      eq_tapq:anon_182:num_checkpoint_end_failed: 0
      eq_tapq:anon_182:num_checkpoint_start: 85
      eq_tapq:anon_182:num_checkpoint_start_failed: 0
      eq_tapq:anon_182:num_delete: 0
      eq_tapq:anon_182:num_delete_failed: 0
      eq_tapq:anon_182:num_flush: 0
      eq_tapq:anon_182:num_flush_failed: 0
      eq_tapq:anon_182:num_mutation: 0
      eq_tapq:anon_182:num_mutation_failed: 0
      eq_tapq:anon_182:num_opaque: 2
      eq_tapq:anon_182:num_opaque_failed: 0
      eq_tapq:anon_182:num_unknown: 0
      eq_tapq:anon_182:num_vbucket_set: 0
      eq_tapq:anon_182:num_vbucket_set_failed: 0
      eq_tapq:anon_182:pending_disconnect: false
      eq_tapq:anon_182:reserved: 0
      eq_tapq:anon_182:supports_ack: true
      eq_tapq:anon_182:type: consumer
      eq_tapq:anon_185:connected: true
      eq_tapq:anon_185:created: 22495
      eq_tapq:anon_185:num_checkpoint_end: 0
      eq_tapq:anon_185:num_checkpoint_end_failed: 0
      eq_tapq:anon_185:num_checkpoint_start: 86
      eq_tapq:anon_185:num_checkpoint_start_failed: 0
      eq_tapq:anon_185:num_delete: 0
      eq_tapq:anon_185:num_delete_failed: 0
      eq_tapq:anon_185:num_flush: 0
      eq_tapq:anon_185:num_flush_failed: 0
      eq_tapq:anon_185:num_mutation: 0
      eq_tapq:anon_185:num_mutation_failed: 0
      eq_tapq:anon_185:num_opaque: 2
      eq_tapq:anon_185:num_opaque_failed: 0
      eq_tapq:anon_185:num_unknown: 0
      eq_tapq:anon_185:num_vbucket_set: 0
      eq_tapq:anon_185:num_vbucket_set_failed: 0
      eq_tapq:anon_185:pending_disconnect: false
      eq_tapq:anon_185:reserved: 0
      eq_tapq:anon_185:supports_ack: true
      eq_tapq:anon_185:type: consumer
      eq_tapq:anon_187:connected: true
      eq_tapq:anon_187:created: 22495
      eq_tapq:anon_187:num_checkpoint_end: 0
      eq_tapq:anon_187:num_checkpoint_end_failed: 0
      eq_tapq:anon_187:num_checkpoint_start: 85
      eq_tapq:anon_187:num_checkpoint_start_failed: 0
      eq_tapq:anon_187:num_delete: 0
      eq_tapq:anon_187:num_delete_failed: 0
      eq_tapq:anon_187:num_flush: 0
      eq_tapq:anon_187:num_flush_failed: 0
      eq_tapq:anon_187:num_mutation: 0
      eq_tapq:anon_187:num_mutation_failed: 0
      eq_tapq:anon_187:num_opaque: 2
      eq_tapq:anon_187:num_opaque_failed: 0
      eq_tapq:anon_187:num_unknown: 0
      eq_tapq:anon_187:num_vbucket_set: 0
      eq_tapq:anon_187:num_vbucket_set_failed: 0
      eq_tapq:anon_187:pending_disconnect: false
      eq_tapq:anon_187:reserved: 0
      eq_tapq:anon_187:supports_ack: true
      eq_tapq:anon_187:type: consumer
      eq_tapq:replication_ns_1@10.1.3.135:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.135:ack_seqno: 91
      eq_tapq:replication_ns_1@10.1.3.135:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.135:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.135:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.135:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.135:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.135:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.135:connected: true
      eq_tapq:replication_ns_1@10.1.3.135:created: 22494
      eq_tapq:replication_ns_1@10.1.3.135:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.135:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.135:idle: true
      eq_tapq:replication_ns_1@10.1.3.135:paused: 1
      eq_tapq:replication_ns_1@10.1.3.135:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.135:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.135:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.135:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.135:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.135:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.135:rec_fetched: 88
      eq_tapq:replication_ns_1@10.1.3.135:recv_ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.135:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.135:seqno_ack_requested: 90
      eq_tapq:replication_ns_1@10.1.3.135:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.135:suspended: false
      eq_tapq:replication_ns_1@10.1.3.135:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.135:total_noops: 5
      eq_tapq:replication_ns_1@10.1.3.135:type: producer
      eq_tapq:replication_ns_1@10.1.3.135:vb_filter:

      { [256,341] }
      eq_tapq:replication_ns_1@10.1.3.135:vb_filters: 86
      eq_tapq:replication_ns_1@10.1.3.147:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.147:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.147:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.147:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:connected: true
      eq_tapq:replication_ns_1@10.1.3.147:created: 22495
      eq_tapq:replication_ns_1@10.1.3.147:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.147:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.147:idle: true
      eq_tapq:replication_ns_1@10.1.3.147:paused: 1
      eq_tapq:replication_ns_1@10.1.3.147:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.147:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.147:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.147:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.147:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.147:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.147:rec_fetched: 87
      eq_tapq:replication_ns_1@10.1.3.147:recv_ack_seqno: 89
      eq_tapq:replication_ns_1@10.1.3.147:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.147:seqno_ack_requested: 89
      eq_tapq:replication_ns_1@10.1.3.147:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.147:suspended: false
      eq_tapq:replication_ns_1@10.1.3.147:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:total_noops: 5
      eq_tapq:replication_ns_1@10.1.3.147:type: producer
      eq_tapq:replication_ns_1@10.1.3.147:vb_filter: { [342,426] }
      eq_tapq:replication_ns_1@10.1.3.147:vb_filters: 85
      eq_tapq:replication_ns_1@10.1.3.151:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.151:ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.151:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.151:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.151:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.151:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.151:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.151:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.151:connected: true
      eq_tapq:replication_ns_1@10.1.3.151:created: 20593
      eq_tapq:replication_ns_1@10.1.3.151:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.151:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.151:idle: true
      eq_tapq:replication_ns_1@10.1.3.151:paused: 1
      eq_tapq:replication_ns_1@10.1.3.151:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.151:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.151:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.151:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.151:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.151:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.151:rec_fetched: 87
      eq_tapq:replication_ns_1@10.1.3.151:recv_ack_seqno: 89
      eq_tapq:replication_ns_1@10.1.3.151:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.151:seqno_ack_requested: 89
      eq_tapq:replication_ns_1@10.1.3.151:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.151:suspended: false
      eq_tapq:replication_ns_1@10.1.3.151:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.151:total_noops: 92
      eq_tapq:replication_ns_1@10.1.3.151:type: producer
      eq_tapq:replication_ns_1@10.1.3.151:vb_filter: { [427,511] }
      eq_tapq:replication_ns_1@10.1.3.151:vb_filters: 85


      [root@localhost bin]# ./cbstats 10.1.3.147:11210 tap
      ep_tap_ack_grace_period: 300
      ep_tap_ack_interval: 1000
      ep_tap_ack_window_size: 10
      ep_tap_backoff_period: 5
      ep_tap_bg_fetch_requeued: 0
      ep_tap_bg_fetched: 0
      ep_tap_bg_max_pending: 500
      ep_tap_count: 6
      ep_tap_deletes: 0
      ep_tap_fg_fetched: 0
      ep_tap_noop_interval: 20
      ep_tap_queue_backfillremaining: 0
      ep_tap_queue_backoff: 0
      ep_tap_queue_drain: 0
      ep_tap_queue_fill: 0
      ep_tap_queue_itemondisk: 0
      ep_tap_throttle_queue_cap: 1000000
      ep_tap_throttle_threshold: 90
      ep_tap_throttled: 0
      ep_tap_total_backlog_size: 0
      ep_tap_total_fetched: 262
      ep_tap_total_queue: 0
      eq_tapq:anon_1:connected: true
      eq_tapq:anon_1:created: 104
      eq_tapq:anon_1:num_checkpoint_end: 0
      eq_tapq:anon_1:num_checkpoint_end_failed: 0
      eq_tapq:anon_1:num_checkpoint_start: 86
      eq_tapq:anon_1:num_checkpoint_start_failed: 0
      eq_tapq:anon_1:num_delete: 0
      eq_tapq:anon_1:num_delete_failed: 0
      eq_tapq:anon_1:num_flush: 0
      eq_tapq:anon_1:num_flush_failed: 0
      eq_tapq:anon_1:num_mutation: 0
      eq_tapq:anon_1:num_mutation_failed: 0
      eq_tapq:anon_1:num_opaque: 2
      eq_tapq:anon_1:num_opaque_failed: 0
      eq_tapq:anon_1:num_unknown: 0
      eq_tapq:anon_1:num_vbucket_set: 0
      eq_tapq:anon_1:num_vbucket_set_failed: 0
      eq_tapq:anon_1:pending_disconnect: false
      eq_tapq:anon_1:reserved: 0
      eq_tapq:anon_1:supports_ack: true
      eq_tapq:anon_1:type: consumer
      eq_tapq:anon_2:connected: true
      eq_tapq:anon_2:created: 104
      eq_tapq:anon_2:num_checkpoint_end: 0
      eq_tapq:anon_2:num_checkpoint_end_failed: 0
      eq_tapq:anon_2:num_checkpoint_start: 85
      eq_tapq:anon_2:num_checkpoint_start_failed: 0
      eq_tapq:anon_2:num_delete: 0
      eq_tapq:anon_2:num_delete_failed: 0
      eq_tapq:anon_2:num_flush: 0
      eq_tapq:anon_2:num_flush_failed: 0
      eq_tapq:anon_2:num_mutation: 0
      eq_tapq:anon_2:num_mutation_failed: 0
      eq_tapq:anon_2:num_opaque: 2
      eq_tapq:anon_2:num_opaque_failed: 0
      eq_tapq:anon_2:num_unknown: 0
      eq_tapq:anon_2:num_vbucket_set: 0
      eq_tapq:anon_2:num_vbucket_set_failed: 0
      eq_tapq:anon_2:pending_disconnect: false
      eq_tapq:anon_2:reserved: 0
      eq_tapq:anon_2:supports_ack: true
      eq_tapq:anon_2:type: consumer
      eq_tapq:anon_3:connected: true
      eq_tapq:anon_3:created: 104
      eq_tapq:anon_3:num_checkpoint_end: 0
      eq_tapq:anon_3:num_checkpoint_end_failed: 0
      eq_tapq:anon_3:num_checkpoint_start: 85
      eq_tapq:anon_3:num_checkpoint_start_failed: 0
      eq_tapq:anon_3:num_delete: 0
      eq_tapq:anon_3:num_delete_failed: 0
      eq_tapq:anon_3:num_flush: 0
      eq_tapq:anon_3:num_flush_failed: 0
      eq_tapq:anon_3:num_mutation: 0
      eq_tapq:anon_3:num_mutation_failed: 0
      eq_tapq:anon_3:num_opaque: 2
      eq_tapq:anon_3:num_opaque_failed: 0
      eq_tapq:anon_3:num_unknown: 0
      eq_tapq:anon_3:num_vbucket_set: 0
      eq_tapq:anon_3:num_vbucket_set_failed: 0
      eq_tapq:anon_3:pending_disconnect: false
      eq_tapq:anon_3:reserved: 0
      eq_tapq:anon_3:supports_ack: true
      eq_tapq:anon_3:type: consumer
      eq_tapq:replication_ns_1@10.1.3.135:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.135:ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.135:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.135:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.135:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.135:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.135:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.135:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.135:connected: true
      eq_tapq:replication_ns_1@10.1.3.135:created: 103
      eq_tapq:replication_ns_1@10.1.3.135:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.135:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.135:idle: true
      eq_tapq:replication_ns_1@10.1.3.135:paused: 1
      eq_tapq:replication_ns_1@10.1.3.135:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.135:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.135:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.135:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.135:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.135:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.135:rec_fetched: 87
      eq_tapq:replication_ns_1@10.1.3.135:recv_ack_seqno: 89
      eq_tapq:replication_ns_1@10.1.3.135:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.135:seqno_ack_requested: 89
      eq_tapq:replication_ns_1@10.1.3.135:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.135:suspended: false
      eq_tapq:replication_ns_1@10.1.3.135:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.135:total_noops: 3
      eq_tapq:replication_ns_1@10.1.3.135:type: producer
      eq_tapq:replication_ns_1@10.1.3.135:vb_filter: { [512,596] }
      eq_tapq:replication_ns_1@10.1.3.135:vb_filters: 85
      eq_tapq:replication_ns_1@10.1.3.146:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.146:ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.146:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.146:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.146:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.146:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.146:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.146:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.146:connected: true
      eq_tapq:replication_ns_1@10.1.3.146:created: 104
      eq_tapq:replication_ns_1@10.1.3.146:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.146:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.146:idle: true
      eq_tapq:replication_ns_1@10.1.3.146:paused: 1
      eq_tapq:replication_ns_1@10.1.3.146:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.146:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.146:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.146:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.146:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.146:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.146:rec_fetched: 87
      eq_tapq:replication_ns_1@10.1.3.146:recv_ack_seqno: 89
      eq_tapq:replication_ns_1@10.1.3.146:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.146:seqno_ack_requested: 89
      eq_tapq:replication_ns_1@10.1.3.146:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.146:suspended: false
      eq_tapq:replication_ns_1@10.1.3.146:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.146:total_noops: 3
      eq_tapq:replication_ns_1@10.1.3.146:type: producer
      eq_tapq:replication_ns_1@10.1.3.146:vb_filter: { [597,681] }
      eq_tapq:replication_ns_1@10.1.3.146:vb_filters: 85
      eq_tapq:replication_ns_1@10.1.3.151:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.151:ack_seqno: 91
      eq_tapq:replication_ns_1@10.1.3.151:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.151:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.151:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.151:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.151:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.151:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.151:connected: true
      eq_tapq:replication_ns_1@10.1.3.151:created: 103
      eq_tapq:replication_ns_1@10.1.3.151:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.151:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.151:idle: true
      eq_tapq:replication_ns_1@10.1.3.151:paused: 1
      eq_tapq:replication_ns_1@10.1.3.151:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.151:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.151:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.151:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.151:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.151:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.151:rec_fetched: 88
      eq_tapq:replication_ns_1@10.1.3.151:recv_ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.151:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.151:seqno_ack_requested: 90
      eq_tapq:replication_ns_1@10.1.3.151:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.151:suspended: false
      eq_tapq:replication_ns_1@10.1.3.151:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.151:total_noops: 3
      eq_tapq:replication_ns_1@10.1.3.151:type: producer
      eq_tapq:replication_ns_1@10.1.3.151:vb_filter: { [682,767] }
      eq_tapq:replication_ns_1@10.1.3.151:vb_filters: 86

      [root@localhost bin]# ./cbstats 10.1.3.151:11210 tap
      ep_tap_ack_grace_period: 300
      ep_tap_ack_interval: 1000
      ep_tap_ack_window_size: 10
      ep_tap_backoff_period: 5
      ep_tap_bg_fetch_requeued: 0
      ep_tap_bg_fetched: 0
      ep_tap_bg_max_pending: 500
      ep_tap_count: 6
      ep_tap_deletes: 0
      ep_tap_fg_fetched: 0
      ep_tap_noop_interval: 20
      ep_tap_queue_backfillremaining: 0
      ep_tap_queue_backoff: 0
      ep_tap_queue_drain: 0
      ep_tap_queue_fill: 0
      ep_tap_queue_itemondisk: 0
      ep_tap_throttle_queue_cap: 1000000
      ep_tap_throttle_threshold: 90
      ep_tap_throttled: 0
      ep_tap_total_backlog_size: 0
      ep_tap_total_fetched: 437
      ep_tap_total_queue: 0
      eq_tapq:anon_1:connected: true
      eq_tapq:anon_1:created: 113
      eq_tapq:anon_1:num_checkpoint_end: 0
      eq_tapq:anon_1:num_checkpoint_end_failed: 0
      eq_tapq:anon_1:num_checkpoint_start: 85
      eq_tapq:anon_1:num_checkpoint_start_failed: 0
      eq_tapq:anon_1:num_delete: 0
      eq_tapq:anon_1:num_delete_failed: 0
      eq_tapq:anon_1:num_flush: 0
      eq_tapq:anon_1:num_flush_failed: 0
      eq_tapq:anon_1:num_mutation: 0
      eq_tapq:anon_1:num_mutation_failed: 0
      eq_tapq:anon_1:num_opaque: 2
      eq_tapq:anon_1:num_opaque_failed: 0
      eq_tapq:anon_1:num_unknown: 0
      eq_tapq:anon_1:num_vbucket_set: 0
      eq_tapq:anon_1:num_vbucket_set_failed: 0
      eq_tapq:anon_1:pending_disconnect: false
      eq_tapq:anon_1:reserved: 0
      eq_tapq:anon_1:supports_ack: true
      eq_tapq:anon_1:type: consumer
      eq_tapq:anon_5:connected: true
      eq_tapq:anon_5:created: 2015
      eq_tapq:anon_5:num_checkpoint_end: 0
      eq_tapq:anon_5:num_checkpoint_end_failed: 0
      eq_tapq:anon_5:num_checkpoint_start: 85
      eq_tapq:anon_5:num_checkpoint_start_failed: 0
      eq_tapq:anon_5:num_delete: 0
      eq_tapq:anon_5:num_delete_failed: 0
      eq_tapq:anon_5:num_flush: 0
      eq_tapq:anon_5:num_flush_failed: 0
      eq_tapq:anon_5:num_mutation: 0
      eq_tapq:anon_5:num_mutation_failed: 0
      eq_tapq:anon_5:num_opaque: 2
      eq_tapq:anon_5:num_opaque_failed: 0
      eq_tapq:anon_5:num_unknown: 0
      eq_tapq:anon_5:num_vbucket_set: 0
      eq_tapq:anon_5:num_vbucket_set_failed: 0
      eq_tapq:anon_5:pending_disconnect: false
      eq_tapq:anon_5:reserved: 0
      eq_tapq:anon_5:supports_ack: true
      eq_tapq:anon_5:type: consumer
      eq_tapq:anon_6:connected: true
      eq_tapq:anon_6:created: 2015
      eq_tapq:anon_6:num_checkpoint_end: 0
      eq_tapq:anon_6:num_checkpoint_end_failed: 0
      eq_tapq:anon_6:num_checkpoint_start: 86
      eq_tapq:anon_6:num_checkpoint_start_failed: 0
      eq_tapq:anon_6:num_delete: 0
      eq_tapq:anon_6:num_delete_failed: 0
      eq_tapq:anon_6:num_flush: 0
      eq_tapq:anon_6:num_flush_failed: 0
      eq_tapq:anon_6:num_mutation: 0
      eq_tapq:anon_6:num_mutation_failed: 0
      eq_tapq:anon_6:num_opaque: 2
      eq_tapq:anon_6:num_opaque_failed: 0
      eq_tapq:anon_6:num_unknown: 0
      eq_tapq:anon_6:num_vbucket_set: 0
      eq_tapq:anon_6:num_vbucket_set_failed: 0
      eq_tapq:anon_6:pending_disconnect: false
      eq_tapq:anon_6:reserved: 0
      eq_tapq:anon_6:supports_ack: true
      eq_tapq:anon_6:type: consumer
      eq_tapq:replication_ns_1@10.1.3.135:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.135:ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.135:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.135:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.135:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.135:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.135:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.135:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.135:connected: true
      eq_tapq:replication_ns_1@10.1.3.135:created: 2015
      eq_tapq:replication_ns_1@10.1.3.135:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.135:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.135:idle: true
      eq_tapq:replication_ns_1@10.1.3.135:paused: 1
      eq_tapq:replication_ns_1@10.1.3.135:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.135:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.135:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.135:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.135:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.135:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.135:rec_fetched: 87
      eq_tapq:replication_ns_1@10.1.3.135:recv_ack_seqno: 89
      eq_tapq:replication_ns_1@10.1.3.135:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.135:seqno_ack_requested: 89
      eq_tapq:replication_ns_1@10.1.3.135:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.135:suspended: false
      eq_tapq:replication_ns_1@10.1.3.135:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.135:total_noops: 3
      eq_tapq:replication_ns_1@10.1.3.135:type: producer
      eq_tapq:replication_ns_1@10.1.3.135:vb_filter: { [768,852] }
      eq_tapq:replication_ns_1@10.1.3.135:vb_filters: 85
      eq_tapq:replication_ns_1@10.1.3.146:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.146:ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.146:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.146:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.146:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.146:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.146:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.146:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.146:connected: true
      eq_tapq:replication_ns_1@10.1.3.146:created: 113
      eq_tapq:replication_ns_1@10.1.3.146:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.146:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.146:idle: true
      eq_tapq:replication_ns_1@10.1.3.146:paused: 1
      eq_tapq:replication_ns_1@10.1.3.146:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.146:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.146:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.146:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.146:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.146:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.146:rec_fetched: 87
      eq_tapq:replication_ns_1@10.1.3.146:recv_ack_seqno: 89
      eq_tapq:replication_ns_1@10.1.3.146:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.146:seqno_ack_requested: 89
      eq_tapq:replication_ns_1@10.1.3.146:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.146:suspended: false
      eq_tapq:replication_ns_1@10.1.3.146:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.146:total_noops: 91
      eq_tapq:replication_ns_1@10.1.3.146:type: producer
      eq_tapq:replication_ns_1@10.1.3.146:vb_filter: { [853,937] }
      eq_tapq:replication_ns_1@10.1.3.146:vb_filters: 85
      eq_tapq:replication_ns_1@10.1.3.147:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:ack_seqno: 91
      eq_tapq:replication_ns_1@10.1.3.147:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.147:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.147:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:connected: true
      eq_tapq:replication_ns_1@10.1.3.147:created: 2015
      eq_tapq:replication_ns_1@10.1.3.147:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.147:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.147:idle: true
      eq_tapq:replication_ns_1@10.1.3.147:paused: 1
      eq_tapq:replication_ns_1@10.1.3.147:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.147:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.147:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.147:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.147:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.147:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.147:rec_fetched: 88
      eq_tapq:replication_ns_1@10.1.3.147:recv_ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.147:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.147:seqno_ack_requested: 90
      eq_tapq:replication_ns_1@10.1.3.147:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.147:suspended: false
      eq_tapq:replication_ns_1@10.1.3.147:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:total_noops: 3
      eq_tapq:replication_ns_1@10.1.3.147:type: producer
      eq_tapq:replication_ns_1@10.1.3.147:vb_filter: { [938,1023] }
      eq_tapq:replication_ns_1@10.1.3.147:vb_filters: 86


      2) reboot .135 and .147 again
      .135 is auto failed over again -> lost items appeared again
      [root@localhost bin]# ./cbstats 10.1.3.146:11210 tap
      ep_tap_ack_grace_period: 300
      ep_tap_ack_interval: 1000
      ep_tap_ack_window_size: 10
      ep_tap_backoff_period: 5
      ep_tap_bg_fetch_requeued: 0
      ep_tap_bg_fetched: 3076107
      ep_tap_bg_max_pending: 500
      ep_tap_count: 5
      ep_tap_deletes: 0
      ep_tap_fg_fetched: 3257334
      ep_tap_noop_interval: 20
      ep_tap_queue_backfillremaining: 0
      ep_tap_queue_backoff: 0
      ep_tap_queue_drain: 0
      ep_tap_queue_fill: 0
      ep_tap_queue_itemondisk: 0
      ep_tap_throttle_queue_cap: 1000000
      ep_tap_throttle_threshold: 90
      ep_tap_throttled: 0
      ep_tap_total_backlog_size: 0
      ep_tap_total_fetched: 6343432
      ep_tap_total_queue: 0
      eq_tapq:anon_182:connected: true
      eq_tapq:anon_182:created: 20593
      eq_tapq:anon_182:num_checkpoint_end: 0
      eq_tapq:anon_182:num_checkpoint_end_failed: 0
      eq_tapq:anon_182:num_checkpoint_start: 85
      eq_tapq:anon_182:num_checkpoint_start_failed: 0
      eq_tapq:anon_182:num_delete: 0
      eq_tapq:anon_182:num_delete_failed: 0
      eq_tapq:anon_182:num_flush: 0
      eq_tapq:anon_182:num_flush_failed: 0
      eq_tapq:anon_182:num_mutation: 0
      eq_tapq:anon_182:num_mutation_failed: 0
      eq_tapq:anon_182:num_opaque: 2
      eq_tapq:anon_182:num_opaque_failed: 0
      eq_tapq:anon_182:num_unknown: 0
      eq_tapq:anon_182:num_vbucket_set: 0
      eq_tapq:anon_182:num_vbucket_set_failed: 0
      eq_tapq:anon_182:pending_disconnect: false
      eq_tapq:anon_182:reserved: 0
      eq_tapq:anon_182:supports_ack: true
      eq_tapq:anon_182:type: consumer
      eq_tapq:anon_191:connected: true
      eq_tapq:anon_191:created: 22859
      eq_tapq:anon_191:num_checkpoint_end: 0
      eq_tapq:anon_191:num_checkpoint_end_failed: 0
      eq_tapq:anon_191:num_checkpoint_start: 85
      eq_tapq:anon_191:num_checkpoint_start_failed: 0
      eq_tapq:anon_191:num_delete: 0
      eq_tapq:anon_191:num_delete_failed: 0
      eq_tapq:anon_191:num_flush: 0
      eq_tapq:anon_191:num_flush_failed: 0
      eq_tapq:anon_191:num_mutation: 0
      eq_tapq:anon_191:num_mutation_failed: 0
      eq_tapq:anon_191:num_opaque: 2
      eq_tapq:anon_191:num_opaque_failed: 0
      eq_tapq:anon_191:num_unknown: 0
      eq_tapq:anon_191:num_vbucket_set: 0
      eq_tapq:anon_191:num_vbucket_set_failed: 0
      eq_tapq:anon_191:pending_disconnect: false
      eq_tapq:anon_191:reserved: 0
      eq_tapq:anon_191:supports_ack: true
      eq_tapq:anon_191:type: consumer
      eq_tapq:replication_ns_1@10.1.3.135:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.135:ack_seqno: 91
      eq_tapq:replication_ns_1@10.1.3.135:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.135:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.135:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.135:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.135:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.135:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.135:connected: false
      eq_tapq:replication_ns_1@10.1.3.135:created: 22858
      eq_tapq:replication_ns_1@10.1.3.135:disconnects: 1
      eq_tapq:replication_ns_1@10.1.3.135:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.135:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.135:idle: true
      eq_tapq:replication_ns_1@10.1.3.135:paused: 1
      eq_tapq:replication_ns_1@10.1.3.135:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.135:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.135:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.135:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.135:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.135:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.135:rec_fetched: 88
      eq_tapq:replication_ns_1@10.1.3.135:recv_ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.135:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.135:seqno_ack_requested: 90
      eq_tapq:replication_ns_1@10.1.3.135:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.135:suspended: false
      eq_tapq:replication_ns_1@10.1.3.135:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.135:total_noops: 1
      eq_tapq:replication_ns_1@10.1.3.135:type: producer
      eq_tapq:replication_ns_1@10.1.3.135:vb_filter: { [256,341] }

      eq_tapq:replication_ns_1@10.1.3.135:vb_filters: 86
      eq_tapq:replication_ns_1@10.1.3.147:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.147:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.147:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.147:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:connected: true
      eq_tapq:replication_ns_1@10.1.3.147:created: 22859
      eq_tapq:replication_ns_1@10.1.3.147:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.147:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.147:idle: true
      eq_tapq:replication_ns_1@10.1.3.147:paused: 1
      eq_tapq:replication_ns_1@10.1.3.147:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.147:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.147:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.147:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.147:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.147:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.147:rec_fetched: 87
      eq_tapq:replication_ns_1@10.1.3.147:recv_ack_seqno: 89
      eq_tapq:replication_ns_1@10.1.3.147:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.147:seqno_ack_requested: 89
      eq_tapq:replication_ns_1@10.1.3.147:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.147:suspended: false
      eq_tapq:replication_ns_1@10.1.3.147:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:total_noops: 10
      eq_tapq:replication_ns_1@10.1.3.147:type: producer
      eq_tapq:replication_ns_1@10.1.3.147:vb_filter:

      { [342,426] }

      eq_tapq:replication_ns_1@10.1.3.147:vb_filters: 85
      eq_tapq:replication_ns_1@10.1.3.151:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.151:ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.151:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.151:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.151:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.151:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.151:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.151:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.151:connected: true
      eq_tapq:replication_ns_1@10.1.3.151:created: 20593
      eq_tapq:replication_ns_1@10.1.3.151:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.151:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.151:idle: true
      eq_tapq:replication_ns_1@10.1.3.151:paused: 1
      eq_tapq:replication_ns_1@10.1.3.151:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.151:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.151:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.151:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.151:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.151:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.151:rec_fetched: 87
      eq_tapq:replication_ns_1@10.1.3.151:recv_ack_seqno: 89
      eq_tapq:replication_ns_1@10.1.3.151:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.151:seqno_ack_requested: 89
      eq_tapq:replication_ns_1@10.1.3.151:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.151:suspended: false
      eq_tapq:replication_ns_1@10.1.3.151:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.151:total_noops: 114
      eq_tapq:replication_ns_1@10.1.3.151:type: producer
      eq_tapq:replication_ns_1@10.1.3.151:vb_filter:

      { [427,511] }

      eq_tapq:replication_ns_1@10.1.3.151:vb_filters: 85

      [root@localhost bin]# ./cbstats 10.1.3.147:11210 tap
      ep_tap_ack_grace_period: 300
      ep_tap_ack_interval: 1000
      ep_tap_ack_window_size: 10
      ep_tap_backoff_period: 5
      ep_tap_bg_fetch_requeued: 0
      ep_tap_bg_fetched: 0
      ep_tap_bg_max_pending: 500
      ep_tap_count: 4
      ep_tap_deletes: 0
      ep_tap_fg_fetched: 0
      ep_tap_noop_interval: 20
      ep_tap_queue_backfillremaining: 0
      ep_tap_queue_backoff: 0
      ep_tap_queue_drain: 0
      ep_tap_queue_fill: 0
      ep_tap_queue_itemondisk: 0
      ep_tap_throttle_queue_cap: 1000000
      ep_tap_throttle_threshold: 90
      ep_tap_throttled: 0
      ep_tap_total_backlog_size: 0
      ep_tap_total_fetched: 262
      ep_tap_total_queue: 0
      eq_tapq:anon_1:connected: true
      eq_tapq:anon_1:created: 124
      eq_tapq:anon_1:num_checkpoint_end: 0
      eq_tapq:anon_1:num_checkpoint_end_failed: 0
      eq_tapq:anon_1:num_checkpoint_start: 86
      eq_tapq:anon_1:num_checkpoint_start_failed: 0
      eq_tapq:anon_1:num_delete: 0
      eq_tapq:anon_1:num_delete_failed: 0
      eq_tapq:anon_1:num_flush: 0
      eq_tapq:anon_1:num_flush_failed: 0
      eq_tapq:anon_1:num_mutation: 0
      eq_tapq:anon_1:num_mutation_failed: 0
      eq_tapq:anon_1:num_opaque: 2
      eq_tapq:anon_1:num_opaque_failed: 0
      eq_tapq:anon_1:num_unknown: 0
      eq_tapq:anon_1:num_vbucket_set: 0
      eq_tapq:anon_1:num_vbucket_set_failed: 0
      eq_tapq:anon_1:pending_disconnect: false
      eq_tapq:anon_1:reserved: 0
      eq_tapq:anon_1:supports_ack: true
      eq_tapq:anon_1:type: consumer
      eq_tapq:anon_3:connected: true
      eq_tapq:anon_3:created: 124
      eq_tapq:anon_3:num_checkpoint_end: 0
      eq_tapq:anon_3:num_checkpoint_end_failed: 0
      eq_tapq:anon_3:num_checkpoint_start: 85
      eq_tapq:anon_3:num_checkpoint_start_failed: 0
      eq_tapq:anon_3:num_delete: 0
      eq_tapq:anon_3:num_delete_failed: 0
      eq_tapq:anon_3:num_flush: 0
      eq_tapq:anon_3:num_flush_failed: 0
      eq_tapq:anon_3:num_mutation: 0
      eq_tapq:anon_3:num_mutation_failed: 0
      eq_tapq:anon_3:num_opaque: 2
      eq_tapq:anon_3:num_opaque_failed: 0
      eq_tapq:anon_3:num_unknown: 0
      eq_tapq:anon_3:num_vbucket_set: 0
      eq_tapq:anon_3:num_vbucket_set_failed: 0
      eq_tapq:anon_3:pending_disconnect: false
      eq_tapq:anon_3:reserved: 0
      eq_tapq:anon_3:supports_ack: true
      eq_tapq:anon_3:type: consumer
      eq_tapq:replication_ns_1@10.1.3.146:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.146:ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.146:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.146:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.146:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.146:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.146:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.146:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.146:connected: true
      eq_tapq:replication_ns_1@10.1.3.146:created: 124
      eq_tapq:replication_ns_1@10.1.3.146:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.146:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.146:idle: true
      eq_tapq:replication_ns_1@10.1.3.146:paused: 1
      eq_tapq:replication_ns_1@10.1.3.146:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.146:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.146:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.146:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.146:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.146:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.146:rec_fetched: 87
      eq_tapq:replication_ns_1@10.1.3.146:recv_ack_seqno: 89
      eq_tapq:replication_ns_1@10.1.3.146:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.146:seqno_ack_requested: 89
      eq_tapq:replication_ns_1@10.1.3.146:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.146:suspended: false
      eq_tapq:replication_ns_1@10.1.3.146:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.146:total_noops: 44
      eq_tapq:replication_ns_1@10.1.3.146:type: producer
      eq_tapq:replication_ns_1@10.1.3.146:vb_filter:

      { [597,681] }

      eq_tapq:replication_ns_1@10.1.3.146:vb_filters: 85
      eq_tapq:replication_ns_1@10.1.3.151:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.151:ack_seqno: 91
      eq_tapq:replication_ns_1@10.1.3.151:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.151:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.151:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.151:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.151:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.151:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.151:connected: true
      eq_tapq:replication_ns_1@10.1.3.151:created: 123
      eq_tapq:replication_ns_1@10.1.3.151:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.151:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.151:idle: true
      eq_tapq:replication_ns_1@10.1.3.151:paused: 1
      eq_tapq:replication_ns_1@10.1.3.151:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.151:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.151:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.151:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.151:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.151:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.151:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.151:rec_fetched: 88
      eq_tapq:replication_ns_1@10.1.3.151:recv_ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.151:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.151:seqno_ack_requested: 90
      eq_tapq:replication_ns_1@10.1.3.151:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.151:suspended: false
      eq_tapq:replication_ns_1@10.1.3.151:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.151:total_noops: 44
      eq_tapq:replication_ns_1@10.1.3.151:type: producer
      eq_tapq:replication_ns_1@10.1.3.151:vb_filter:

      { [682,767] }

      eq_tapq:replication_ns_1@10.1.3.151:vb_filters: 86

      [root@localhost bin]# ./cbstats 10.1.3.151:11210 tap
      ep_tap_ack_grace_period: 300
      ep_tap_ack_interval: 1000
      ep_tap_ack_window_size: 10
      ep_tap_backoff_period: 5
      ep_tap_bg_fetch_requeued: 0
      ep_tap_bg_fetched: 0
      ep_tap_bg_max_pending: 500
      ep_tap_count: 4
      ep_tap_deletes: 0
      ep_tap_fg_fetched: 0
      ep_tap_noop_interval: 20
      ep_tap_queue_backfillremaining: 0
      ep_tap_queue_backoff: 0
      ep_tap_queue_drain: 0
      ep_tap_queue_fill: 0
      ep_tap_queue_itemondisk: 0
      ep_tap_throttle_queue_cap: 1000000
      ep_tap_throttle_threshold: 90
      ep_tap_throttled: 0
      ep_tap_total_backlog_size: 0
      ep_tap_total_fetched: 612
      ep_tap_total_queue: 0
      eq_tapq:anon_10:connected: true
      eq_tapq:anon_10:created: 2378
      eq_tapq:anon_10:num_checkpoint_end: 0
      eq_tapq:anon_10:num_checkpoint_end_failed: 0
      eq_tapq:anon_10:num_checkpoint_start: 86
      eq_tapq:anon_10:num_checkpoint_start_failed: 0
      eq_tapq:anon_10:num_delete: 0
      eq_tapq:anon_10:num_delete_failed: 0
      eq_tapq:anon_10:num_flush: 0
      eq_tapq:anon_10:num_flush_failed: 0
      eq_tapq:anon_10:num_mutation: 0
      eq_tapq:anon_10:num_mutation_failed: 0
      eq_tapq:anon_10:num_opaque: 2
      eq_tapq:anon_10:num_opaque_failed: 0
      eq_tapq:anon_10:num_unknown: 0
      eq_tapq:anon_10:num_vbucket_set: 0
      eq_tapq:anon_10:num_vbucket_set_failed: 0
      eq_tapq:anon_10:pending_disconnect: false
      eq_tapq:anon_10:reserved: 0
      eq_tapq:anon_10:supports_ack: true
      eq_tapq:anon_10:type: consumer
      eq_tapq:anon_1:connected: true
      eq_tapq:anon_1:created: 113
      eq_tapq:anon_1:num_checkpoint_end: 0
      eq_tapq:anon_1:num_checkpoint_end_failed: 0
      eq_tapq:anon_1:num_checkpoint_start: 85
      eq_tapq:anon_1:num_checkpoint_start_failed: 0
      eq_tapq:anon_1:num_delete: 0
      eq_tapq:anon_1:num_delete_failed: 0
      eq_tapq:anon_1:num_flush: 0
      eq_tapq:anon_1:num_flush_failed: 0
      eq_tapq:anon_1:num_mutation: 0
      eq_tapq:anon_1:num_mutation_failed: 0
      eq_tapq:anon_1:num_opaque: 2
      eq_tapq:anon_1:num_opaque_failed: 0
      eq_tapq:anon_1:num_unknown: 0
      eq_tapq:anon_1:num_vbucket_set: 0
      eq_tapq:anon_1:num_vbucket_set_failed: 0
      eq_tapq:anon_1:pending_disconnect: false
      eq_tapq:anon_1:reserved: 0
      eq_tapq:anon_1:supports_ack: true
      eq_tapq:anon_1:type: consumer
      eq_tapq:replication_ns_1@10.1.3.146:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.146:ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.146:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.146:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.146:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.146:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.146:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.146:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.146:connected: true
      eq_tapq:replication_ns_1@10.1.3.146:created: 113
      eq_tapq:replication_ns_1@10.1.3.146:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.146:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.146:idle: true
      eq_tapq:replication_ns_1@10.1.3.146:paused: 1
      eq_tapq:replication_ns_1@10.1.3.146:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.146:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.146:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.146:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.146:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.146:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.146:rec_fetched: 87
      eq_tapq:replication_ns_1@10.1.3.146:recv_ack_seqno: 89
      eq_tapq:replication_ns_1@10.1.3.146:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.146:seqno_ack_requested: 89
      eq_tapq:replication_ns_1@10.1.3.146:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.146:suspended: false
      eq_tapq:replication_ns_1@10.1.3.146:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.146:total_noops: 149
      eq_tapq:replication_ns_1@10.1.3.146:type: producer
      eq_tapq:replication_ns_1@10.1.3.146:vb_filter:

      { [853,937] }
      eq_tapq:replication_ns_1@10.1.3.146:vb_filters: 85
      eq_tapq:replication_ns_1@10.1.3.147:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:ack_seqno: 91
      eq_tapq:replication_ns_1@10.1.3.147:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.147:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.147:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:connected: true
      eq_tapq:replication_ns_1@10.1.3.147:created: 2379
      eq_tapq:replication_ns_1@10.1.3.147:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.147:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.147:idle: true
      eq_tapq:replication_ns_1@10.1.3.147:paused: 1
      eq_tapq:replication_ns_1@10.1.3.147:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.147:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.147:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.147:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.147:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.147:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.147:rec_fetched: 88
      eq_tapq:replication_ns_1@10.1.3.147:recv_ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.147:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.147:seqno_ack_requested: 90
      eq_tapq:replication_ns_1@10.1.3.147:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.147:suspended: false
      eq_tapq:replication_ns_1@10.1.3.147:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:total_noops: 46
      eq_tapq:replication_ns_1@10.1.3.147:type: producer
      eq_tapq:replication_ns_1@10.1.3.147:vb_filter: { [938,1023] }
      eq_tapq:replication_ns_1@10.1.3.147:vb_filters: 86

      [root@localhost bin]# ./cbstats 10.1.3.151:11210 tap
      ep_tap_ack_grace_period: 300
      ep_tap_ack_interval: 1000
      ep_tap_ack_window_size: 10
      ep_tap_backoff_period: 5
      ep_tap_bg_fetch_requeued: 0
      ep_tap_bg_fetched: 0
      ep_tap_bg_max_pending: 500
      ep_tap_count: 4
      ep_tap_deletes: 0
      ep_tap_fg_fetched: 0
      ep_tap_noop_interval: 20
      ep_tap_queue_backfillremaining: 0
      ep_tap_queue_backoff: 0
      ep_tap_queue_drain: 0
      ep_tap_queue_fill: 0
      ep_tap_queue_itemondisk: 0
      ep_tap_throttle_queue_cap: 1000000
      ep_tap_throttle_threshold: 90
      ep_tap_throttled: 0
      ep_tap_total_backlog_size: 0
      ep_tap_total_fetched: 612
      ep_tap_total_queue: 0
      eq_tapq:anon_10:connected: true
      eq_tapq:anon_10:created: 2378
      eq_tapq:anon_10:num_checkpoint_end: 0
      eq_tapq:anon_10:num_checkpoint_end_failed: 0
      eq_tapq:anon_10:num_checkpoint_start: 86
      eq_tapq:anon_10:num_checkpoint_start_failed: 0
      eq_tapq:anon_10:num_delete: 0
      eq_tapq:anon_10:num_delete_failed: 0
      eq_tapq:anon_10:num_flush: 0
      eq_tapq:anon_10:num_flush_failed: 0
      eq_tapq:anon_10:num_mutation: 0
      eq_tapq:anon_10:num_mutation_failed: 0
      eq_tapq:anon_10:num_opaque: 2
      eq_tapq:anon_10:num_opaque_failed: 0
      eq_tapq:anon_10:num_unknown: 0
      eq_tapq:anon_10:num_vbucket_set: 0
      eq_tapq:anon_10:num_vbucket_set_failed: 0
      eq_tapq:anon_10:pending_disconnect: false
      eq_tapq:anon_10:reserved: 0
      eq_tapq:anon_10:supports_ack: true
      eq_tapq:anon_10:type: consumer
      eq_tapq:anon_1:connected: true
      eq_tapq:anon_1:created: 113
      eq_tapq:anon_1:num_checkpoint_end: 0
      eq_tapq:anon_1:num_checkpoint_end_failed: 0
      eq_tapq:anon_1:num_checkpoint_start: 85
      eq_tapq:anon_1:num_checkpoint_start_failed: 0
      eq_tapq:anon_1:num_delete: 0
      eq_tapq:anon_1:num_delete_failed: 0
      eq_tapq:anon_1:num_flush: 0
      eq_tapq:anon_1:num_flush_failed: 0
      eq_tapq:anon_1:num_mutation: 0
      eq_tapq:anon_1:num_mutation_failed: 0
      eq_tapq:anon_1:num_opaque: 2
      eq_tapq:anon_1:num_opaque_failed: 0
      eq_tapq:anon_1:num_unknown: 0
      eq_tapq:anon_1:num_vbucket_set: 0
      eq_tapq:anon_1:num_vbucket_set_failed: 0
      eq_tapq:anon_1:pending_disconnect: false
      eq_tapq:anon_1:reserved: 0
      eq_tapq:anon_1:supports_ack: true
      eq_tapq:anon_1:type: consumer
      eq_tapq:replication_ns_1@10.1.3.146:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.146:ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.146:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.146:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.146:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.146:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.146:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.146:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.146:connected: true
      eq_tapq:replication_ns_1@10.1.3.146:created: 113
      eq_tapq:replication_ns_1@10.1.3.146:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.146:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.146:idle: true
      eq_tapq:replication_ns_1@10.1.3.146:paused: 1
      eq_tapq:replication_ns_1@10.1.3.146:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.146:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.146:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.146:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.146:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.146:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.146:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.146:rec_fetched: 87
      eq_tapq:replication_ns_1@10.1.3.146:recv_ack_seqno: 89
      eq_tapq:replication_ns_1@10.1.3.146:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.146:seqno_ack_requested: 89
      eq_tapq:replication_ns_1@10.1.3.146:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.146:suspended: false
      eq_tapq:replication_ns_1@10.1.3.146:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.146:total_noops: 166
      eq_tapq:replication_ns_1@10.1.3.146:type: producer
      eq_tapq:replication_ns_1@10.1.3.146:vb_filter: { [853,937] }

      eq_tapq:replication_ns_1@10.1.3.146:vb_filters: 85
      eq_tapq:replication_ns_1@10.1.3.147:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:ack_seqno: 91
      eq_tapq:replication_ns_1@10.1.3.147:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.147:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.147:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:connected: true
      eq_tapq:replication_ns_1@10.1.3.147:created: 2379
      eq_tapq:replication_ns_1@10.1.3.147:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.147:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.147:idle: true
      eq_tapq:replication_ns_1@10.1.3.147:paused: 1
      eq_tapq:replication_ns_1@10.1.3.147:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.147:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.147:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.147:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.147:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.147:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_drain: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.147:rec_fetched: 88
      eq_tapq:replication_ns_1@10.1.3.147:recv_ack_seqno: 90
      eq_tapq:replication_ns_1@10.1.3.147:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.147:seqno_ack_requested: 90
      eq_tapq:replication_ns_1@10.1.3.147:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.147:suspended: false
      eq_tapq:replication_ns_1@10.1.3.147:total_backlog_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:total_noops: 63
      eq_tapq:replication_ns_1@10.1.3.147:type: producer
      eq_tapq:replication_ns_1@10.1.3.147:vb_filter:

      { [938,1023] }

      eq_tapq:replication_ns_1@10.1.3.147:vb_filters: 86

      attaching cbcollect
      https://s3.amazonaws.com/bugdb/jira/MB-7214/c85a006a/collect-135.zip
      https://s3.amazonaws.com/bugdb/jira/MB-7214/c85a006a/collect-146.zip
      https://s3.amazonaws.com/bugdb/jira/MB-7214/c85a006a/collect-147.zip
      https://s3.amazonaws.com/bugdb/jira/MB-7214/c85a006a/collect-151.zip

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        steve Steve Yen added a comment -

        bug-scrub...

        possible candidate for release ship blocker if data loss is easily reproduced.

        Show
        steve Steve Yen added a comment - bug-scrub... possible candidate for release ship blocker if data loss is easily reproduced.
        Hide
        iryna iryna added a comment -

        flusher was completed before reboot

        Show
        iryna iryna added a comment - flusher was completed before reboot
        Hide
        iryna iryna added a comment -

        couldn't reproduce it for build 1965

        Show
        iryna iryna added a comment - couldn't reproduce it for build 1965
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        closing this bug as unable to reproduce

        Show
        farshid Farshid Ghods (Inactive) added a comment - closing this bug as unable to reproduce
        Hide
        iryna iryna added a comment -

        closed as not reproduced

        Show
        iryna iryna added a comment - closed as not reproduced

          People

          • Assignee:
            iryna iryna
            Reporter:
            iryna iryna
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes