Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7280

[windows] rebalance is hunging when load is going on

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Cannot Reproduce
    • Affects Version/s: 2.0
    • Fix Version/s: 2.0.1
    • Component/s: ns_server
    • Security Level: Public
    • Labels:
      None
    • Environment:

      Description

      rebalance is hunging when load is going on
      1 node, rebalance 1-> 2, 5 views (1 view per ddoc), load using mcsoda is going on
      once load is started rebalance hungs.
      Statistics all and tap:
      accepting_conns: 1
      auth_cmds: 4
      auth_errors: 0
      bucket_active_conns: 1
      bucket_conns: 26
      bytes: 755258232
      bytes_read: 2673284540
      bytes_written: 2824684578
      cas_badval: 0
      cas_hits: 0
      cas_misses: 0
      cmd_flush: 0
      cmd_get: 1280
      cmd_set: 4134710
      conn_yields: 113110
      connection_structures: 5000
      curr_connections: 26
      curr_conns_on_port_11209: 18
      curr_conns_on_port_11210: 6
      curr_items: 2027168
      curr_items_tot: 3618437
      curr_temp_items: 0
      daemon_connections: 4
      decr_hits: 0
      decr_misses: 0
      delete_hits: 0
      delete_misses: 0
      ep_access_scanner_last_runtime: 22
      ep_access_scanner_num_items: 3383889
      ep_access_scanner_task_time: 2012-11-29 10:00:00
      ep_allow_data_loss_during_shutdown: 1
      ep_alog_block_size: 4096
      ep_alog_path: c:/Program Files/Couchbase/Server/var/lib/couchbase/data/default/access.log
      ep_alog_sleep_time: 1440
      ep_alog_task_time: 10
      ep_backend: couchdb
      ep_bg_fetch_delay: 0
      ep_bg_fetched: 38
      ep_bg_load: 931092
      ep_bg_load_avg: 24502
      ep_bg_max_load: 618147
      ep_bg_max_wait: 3941
      ep_bg_meta_fetched: 0
      ep_bg_min_load: 397
      ep_bg_min_wait: 66
      ep_bg_num_samples: 38
      ep_bg_remaining_jobs: 0
      ep_bg_wait: 16623
      ep_bg_wait_avg: 437
      ep_chk_max_items: 5000
      ep_chk_period: 1800
      ep_chk_persistence_remains: 0
      ep_chk_persistence_timeout: 20
      ep_chk_remover_stime: 5
      ep_commit_num: 657
      ep_commit_time: 6759
      ep_commit_time_total: 235174
      ep_concurrentDB: 1
      ep_config_file:
      ep_couch_bucket: default
      ep_couch_host: localhost
      ep_couch_port: 11213
      ep_couch_reconnect_sleeptime: 250
      ep_couch_response_timeout: 180000
      ep_data_age: 2888
      ep_data_age_highwat: 3580
      ep_data_traffic_enabled: 0
      ep_dbinit: 1
      ep_dbname: c:/Program Files/Couchbase/Server/var/lib/couchbase/data/default
      ep_degraded_mode: 0
      ep_diskqueue_drain: 4935087
      ep_diskqueue_fill: 5721297
      ep_diskqueue_items: 824556
      ep_diskqueue_memory: 25198208
      ep_diskqueue_pending: 43177657
      ep_exp_pager_stime: 3600
      ep_expired_access: 0
      ep_expired_pager: 0
      ep_expiry_window: 3
      ep_failpartialwarmup: 0
      ep_flush_all: false
      ep_flush_duration_total: 8736
      ep_flushall_enabled: 0
      ep_flusher_state: running
      ep_flusher_todo: 579935
      ep_getl_default_timeout: 15
      ep_getl_max_timeout: 30
      ep_ht_locks: 5
      ep_ht_size: 3079
      ep_inconsistent_slave_chk: 0
      ep_initfile:
      ep_io_num_read: 38
      ep_io_num_write: 4842366
      ep_io_read_bytes: 10745
      ep_io_write_bytes: 1356419842
      ep_item_begin_failed: 0
      ep_item_commit_failed: 0
      ep_item_flush_expired: 0
      ep_item_flush_failed: 0
      ep_item_num_based_new_chk: 1
      ep_items_rm_from_checkpoints: 4644424
      ep_keep_closed_chks: 0
      ep_klog_block_size: 4096
      ep_klog_compactor_queue_cap: 500000
      ep_klog_compactor_stime: 3600
      ep_klog_flush: commit2
      ep_klog_max_entry_ratio: 10
      ep_klog_max_log_size: 2147483647
      ep_klog_path:
      ep_klog_sync: commit2
      ep_kv_size: 575124879
      ep_max_checkpoints: 2
      ep_max_data_size: 838860800
      ep_max_item_size: 20971520
      ep_max_size: 838860800
      ep_max_txn_size: 10000
      ep_max_vbuckets: 1024
      ep_mem_high_wat: 629145600
      ep_mem_low_wat: 503316480
      ep_mem_tracker_enabled: true
      ep_min_data_age: 0
      ep_mlog_compactor_runs: 0
      ep_mutation_mem_threshold: 0
      ep_num_access_scanner_runs: 1
      ep_num_eject_failures: 276090282
      ep_num_expiry_pager_runs: 3
      ep_num_non_resident: 2884986
      ep_num_not_my_vbuckets: 6
      ep_num_ops_del_meta: 0
      ep_num_ops_get_meta: 1
      ep_num_ops_set_meta: 0
      ep_num_pager_runs: 352
      ep_num_value_ejects: 3245555
      ep_oom_errors: 0
      ep_overhead: 93729316
      ep_pager_active_vb_pcnt: 40
      ep_pager_unbiased_period: 60
      ep_pending_ops: 0
      ep_pending_ops_max: 0
      ep_pending_ops_max_duration: 0
      ep_pending_ops_total: 0
      ep_postInitfile:
      ep_queue_age_cap: 900
      ep_queue_size: 244621
      ep_startup_time: 1354086142
      ep_storage_age: 2888
      ep_storage_age_highwat: 3579
      ep_store_max_concurrency: 10
      ep_store_max_readers: 9
      ep_store_max_readwrite: 1
      ep_stored_val_type:
      ep_tap_ack_grace_period: 300
      ep_tap_ack_initial_sequence_number: 1
      ep_tap_ack_interval: 1000
      ep_tap_ack_window_size: 10
      ep_tap_backfill_resident: 0.9
      ep_tap_backlog_limit: 5000
      ep_tap_backoff_period: 5
      ep_tap_bg_fetch_requeued: 0
      ep_tap_bg_fetched: 1978
      ep_tap_bg_max_pending: 500
      ep_tap_keepalive: 300
      ep_tap_noop_interval: 20
      ep_tap_requeue_sleep_time: 0.1
      ep_tap_throttle_cap_pcnt: 10
      ep_tap_throttle_queue_cap: 1000000
      ep_tap_throttle_threshold: 90
      ep_tmp_oom_errors: 408666
      ep_too_old: 245644
      ep_too_young: 0
      ep_total_cache_size: 531703635
      ep_total_del_items: 0
      ep_total_enqueued: 5943389
      ep_total_new_items: 3060472
      ep_total_persisted: 4842366
      ep_uncommitted_items: 4544
      ep_value_size: 246296364
      ep_vb0: 0
      ep_vb_snapshot_total: 602
      ep_vb_total: 922
      ep_vbucket_del: 102
      ep_vbucket_del_avg_walltime: 1184789
      ep_vbucket_del_fail: 0
      ep_vbucket_del_max_walltime: 6511942
      ep_version: 2.0.0r_140_gde42c8c
      ep_waitforwarmup: 0
      ep_warmup: 1
      ep_warmup_batch_size: 1000
      ep_warmup_dups: 0
      ep_warmup_min_items_threshold: 100
      ep_warmup_min_memory_threshold: 100
      ep_warmup_oom: 0
      ep_warmup_thread: complete
      ep_warmup_time: 2049
      get_hits: 1165
      get_misses: 115
      incr_hits: 0
      incr_misses: 0
      libevent: 2.0.11-stable
      limit_maxbytes: 67108864
      listen_disabled_num: 0
      max_conns_on_port_11209: 1000
      max_conns_on_port_11210: 9000
      mem_used: 755258232
      pid: 2940
      pointer_size: 64
      rejected_conns: 0
      tap_checkpoint_end_received: 6023
      tap_checkpoint_end_sent: 9611
      tap_checkpoint_start_received: 6778
      tap_checkpoint_start_sent: 10551
      tap_connect_received: 310
      tap_mutation_received: 3981711
      tap_mutation_sent: 3854114
      tap_opaque_received: 1877
      tap_opaque_sent: 2073
      tap_vbucket_set_received: 242
      tap_vbucket_set_sent: 204
      threads: 4
      time: 1354098953
      total_connections: 1606
      uptime: 13936
      vb_active_curr_items: 2027168
      vb_active_eject: 1930964
      vb_active_expired: 0
      vb_active_ht_memory: 13219776
      vb_active_itm_memory: 266946416
      vb_active_meta_data_memory: 171586408
      vb_active_num: 531
      vb_active_num_non_resident: 1736718
      vb_active_num_ref_ejects: 1734941
      vb_active_num_ref_items: 78790
      vb_active_ops_create: 1758886
      vb_active_ops_delete: 0
      vb_active_ops_reject: 0
      vb_active_ops_update: 1015722
      vb_active_perc_mem_resident: 14
      vb_active_queue_age: 703101627000
      vb_active_queue_drain: 2918662
      vb_active_queue_fill: 3212663
      vb_active_queue_memory: 9447520
      vb_active_queue_pending: 16159447
      vb_active_queue_size: 295235
      vb_dead_num: 0
      vb_pending_curr_items: 0
      vb_pending_eject: 0
      vb_pending_expired: 0
      vb_pending_ht_memory: 0
      vb_pending_itm_memory: 0
      vb_pending_meta_data_memory: 0
      vb_pending_num: 0
      vb_pending_num_non_resident: 0
      vb_pending_num_ref_ejects: 0
      vb_pending_num_ref_items: 0
      vb_pending_ops_create: 0
      vb_pending_ops_delete: 0
      vb_pending_ops_reject: 0
      vb_pending_ops_update: 0
      vb_pending_perc_mem_resident: 0
      vb_pending_queue_age: 0
      vb_pending_queue_drain: 0
      vb_pending_queue_fill: 0
      vb_pending_queue_memory: 0
      vb_pending_queue_pending: 0
      vb_pending_queue_size: 0
      vb_replica_curr_items: 1591269
      vb_replica_eject: 1313893
      vb_replica_expired: 0
      vb_replica_ht_memory: 9734336
      vb_replica_itm_memory: 264757219
      vb_replica_meta_data_memory: 134821945
      vb_replica_num: 391
      vb_replica_num_non_resident: 1148268
      vb_replica_num_ref_ejects: 1277577
      vb_replica_num_ref_items: 60822
      vb_replica_ops_create: 1172973
      vb_replica_ops_delete: 0
      vb_replica_ops_reject: 0
      vb_replica_ops_update: 679912
      vb_replica_perc_mem_resident: 27
      vb_replica_queue_age: 1161936223000
      vb_replica_queue_drain: 2016425
      vb_replica_queue_fill: 2508634
      vb_replica_queue_memory: 15750688
      vb_replica_queue_pending: 27018210
      vb_replica_queue_size: 492209
      version: 1.4.4_600_g7ea975a

      ep_tap_ack_grace_period: 300
      ep_tap_ack_interval: 1000
      ep_tap_ack_window_size: 10
      ep_tap_backoff_period: 5
      ep_tap_bg_fetch_requeued: 0
      ep_tap_bg_fetched: 1978
      ep_tap_bg_max_pending: 500
      ep_tap_count: 6
      ep_tap_deletes: 0
      ep_tap_fg_fetched: 3973157
      ep_tap_noop_interval: 20
      ep_tap_queue_backfillremaining: 0
      ep_tap_queue_backoff: 79439
      ep_tap_queue_drain: 3837591
      ep_tap_queue_fill: 0
      ep_tap_queue_itemondisk: 0
      ep_tap_throttle_queue_cap: 1000000
      ep_tap_throttle_threshold: 90
      ep_tap_throttled: 892516
      ep_tap_total_backlog_size: 8
      ep_tap_total_fetched: 3997574
      ep_tap_total_queue: 0
      eq_tapq:anon_1:connected: true
      eq_tapq:anon_1:created: 1127
      eq_tapq:anon_1:num_checkpoint_end: 6057
      eq_tapq:anon_1:num_checkpoint_end_failed: 0
      eq_tapq:anon_1:num_checkpoint_start: 6569
      eq_tapq:anon_1:num_checkpoint_start_failed: 0
      eq_tapq:anon_1:num_delete: 0
      eq_tapq:anon_1:num_delete_failed: 0
      eq_tapq:anon_1:num_flush: 0
      eq_tapq:anon_1:num_flush_failed: 0
      eq_tapq:anon_1:num_mutation: 3246722
      eq_tapq:anon_1:num_mutation_failed: 970546
      eq_tapq:anon_1:num_opaque: 1270
      eq_tapq:anon_1:num_opaque_failed: 0
      eq_tapq:anon_1:num_unknown: 0
      eq_tapq:anon_1:num_vbucket_set: 0
      eq_tapq:anon_1:num_vbucket_set_failed: 0
      eq_tapq:anon_1:pending_disconnect: false
      eq_tapq:anon_1:reserved: 0
      eq_tapq:anon_1:supports_ack: true
      eq_tapq:anon_1:type: consumer
      eq_tapq:anon_448:connected: true
      eq_tapq:anon_448:created: 13890
      eq_tapq:anon_448:num_checkpoint_end: 1
      eq_tapq:anon_448:num_checkpoint_end_failed: 0
      eq_tapq:anon_448:num_checkpoint_start: 2
      eq_tapq:anon_448:num_checkpoint_start_failed: 0
      eq_tapq:anon_448:num_delete: 0
      eq_tapq:anon_448:num_delete_failed: 0
      eq_tapq:anon_448:num_flush: 0
      eq_tapq:anon_448:num_flush_failed: 0
      eq_tapq:anon_448:num_mutation: 558
      eq_tapq:anon_448:num_mutation_failed: 6915
      eq_tapq:anon_448:num_opaque: 2
      eq_tapq:anon_448:num_opaque_failed: 0
      eq_tapq:anon_448:num_unknown: 0
      eq_tapq:anon_448:num_vbucket_set: 0
      eq_tapq:anon_448:num_vbucket_set_failed: 0
      eq_tapq:anon_448:pending_disconnect: false
      eq_tapq:anon_448:reserved: 0
      eq_tapq:anon_448:supports_ack: true
      eq_tapq:anon_448:type: consumer
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':ack_log_size: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':ack_seqno: 7386
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':ack_window_full: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':backfill_completed: true
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':backfill_start_timestamp: 1354092278
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':bg_jobs_completed: 1978
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':bg_jobs_issued: 1978
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':bg_result_size: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':connected: true
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':created: 7261
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':has_queued_item: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':idle: true
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':paused: 1
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':pending_backfill: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':pending_disconnect: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':pending_disk_backfill: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':qlen: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':qlen_high_pri: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':qlen_low_pri: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':queue_backfillremaining: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':queue_backoff: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':queue_drain: 7352
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':queue_fill: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':queue_itemondisk: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':queue_memory: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':rec_fetched: 5403
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':recv_ack_seqno: 7385
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':reserved: 1
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':seqno_ack_requested: 7385
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':supports_ack: true
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':suspended: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':total_backlog_size: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':total_noops: 103
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':type: producer
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':vb_filter:

      { 272 }
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':vb_filters: 1
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':ack_log_size: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':ack_seqno: 4205
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':ack_window_full: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':backfill_completed: true
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':backfill_start_timestamp: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':bg_jobs_completed: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':bg_jobs_issued: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':bg_result_size: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':connected: true
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':created: 7261
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':has_queued_item: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':idle: true
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':paused: 1
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':pending_backfill: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':pending_disconnect: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':pending_disk_backfill: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':qlen: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':qlen_high_pri: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':qlen_low_pri: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':queue_backfillremaining: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':queue_backoff: 154
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':queue_drain: 4175
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':queue_fill: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':queue_itemondisk: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':queue_memory: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':rec_fetched: 4202
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':recv_ack_seqno: 4204
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':reserved: 1
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':seqno_ack_requested: 4204
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':supports_ack: true
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':suspended: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':total_backlog_size: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':total_noops: 99
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':type: producer
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':vb_filter: { 272 }

      eq_tapq:replication_building_272_'ns_1@10.1.3.147':vb_filters: 1
      eq_tapq:replication_ns_1@10.1.3.135:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.135:ack_seqno: 561390
      eq_tapq:replication_ns_1@10.1.3.135:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.135:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.135:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.135:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.135:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.135:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.135:connected: true
      eq_tapq:replication_ns_1@10.1.3.135:created: 4873
      eq_tapq:replication_ns_1@10.1.3.135:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.135:has_queued_item: true
      eq_tapq:replication_ns_1@10.1.3.135:idle: false
      eq_tapq:replication_ns_1@10.1.3.135:paused: 1
      eq_tapq:replication_ns_1@10.1.3.135:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.135:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.135:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.135:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.135:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.135:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_drain: 557409
      eq_tapq:replication_ns_1@10.1.3.135:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.135:rec_fetched: 561267
      eq_tapq:replication_ns_1@10.1.3.135:recv_ack_seqno: 561389
      eq_tapq:replication_ns_1@10.1.3.135:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.135:seqno_ack_requested: 561389
      eq_tapq:replication_ns_1@10.1.3.135:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.135:suspended: false
      eq_tapq:replication_ns_1@10.1.3.135:total_backlog_size: 4
      eq_tapq:replication_ns_1@10.1.3.135:total_noops: 79
      eq_tapq:replication_ns_1@10.1.3.135:type: producer
      eq_tapq:replication_ns_1@10.1.3.135:vb_filter:

      { [682,802] }
      eq_tapq:replication_ns_1@10.1.3.135:vb_filters: 121
      eq_tapq:replication_ns_1@10.1.3.147:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:ack_seqno: 3286570
      eq_tapq:replication_ns_1@10.1.3.147:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.147:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.147:backfill_start_timestamp: 1354086144
      eq_tapq:replication_ns_1@10.1.3.147:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:connected: true
      eq_tapq:replication_ns_1@10.1.3.147:created: 1127
      eq_tapq:replication_ns_1@10.1.3.147:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.147:has_queued_item: true
      eq_tapq:replication_ns_1@10.1.3.147:idle: false
      eq_tapq:replication_ns_1@10.1.3.147:paused: 1
      eq_tapq:replication_ns_1@10.1.3.147:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.147:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.147:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.147:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.147:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.147:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_backoff: 79285
      eq_tapq:replication_ns_1@10.1.3.147:queue_drain: 3268655
      eq_tapq:replication_ns_1@10.1.3.147:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.147:rec_fetched: 3285440
      eq_tapq:replication_ns_1@10.1.3.147:recv_ack_seqno: 3286569
      eq_tapq:replication_ns_1@10.1.3.147:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.147:seqno_ack_requested: 3286569
      eq_tapq:replication_ns_1@10.1.3.147:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.147:suspended: false
      eq_tapq:replication_ns_1@10.1.3.147:total_backlog_size: 4
      eq_tapq:replication_ns_1@10.1.3.147:total_noops: 143
      eq_tapq:replication_ns_1@10.1.3.147:type: producer
      eq_tapq:replication_ns_1@10.1.3.147:vb_filter: { [0,169], [273,511] }
      eq_tapq:replication_ns_1@10.1.3.147:vb_filters: 409


      statistics after 30 min:
      accepting_conns: 1
      auth_cmds: 4
      auth_errors: 0
      bucket_active_conns: 1
      bucket_conns: 26
      bytes: 755127144
      bytes_read: 2885054305
      bytes_written: 2937374687
      cas_badval: 0
      cas_hits: 0
      cas_misses: 0
      cmd_flush: 0
      cmd_get: 1280
      cmd_set: 4523945
      conn_yields: 122744
      connection_structures: 5000
      curr_connections: 26
      curr_conns_on_port_11209: 18
      curr_conns_on_port_11210: 6
      curr_items: 2053924
      curr_items_tot: 3674443
      curr_temp_items: 0
      daemon_connections: 4
      decr_hits: 0
      decr_misses: 0
      delete_hits: 0
      delete_misses: 0
      ep_access_scanner_last_runtime: 22
      ep_access_scanner_num_items: 3383889
      ep_access_scanner_task_time: 2012-11-29 10:00:00
      ep_allow_data_loss_during_shutdown: 1
      ep_alog_block_size: 4096
      ep_alog_path: c:/Program Files/Couchbase/Server/var/lib/couchbase/data/default/access.log
      ep_alog_sleep_time: 1440
      ep_alog_task_time: 10
      ep_backend: couchdb
      ep_bg_fetch_delay: 0
      ep_bg_fetched: 38
      ep_bg_load: 931092
      ep_bg_load_avg: 24502
      ep_bg_max_load: 618147
      ep_bg_max_wait: 3941
      ep_bg_meta_fetched: 0
      ep_bg_min_load: 397
      ep_bg_min_wait: 66
      ep_bg_num_samples: 38
      ep_bg_remaining_jobs: 0
      ep_bg_wait: 16623
      ep_bg_wait_avg: 437
      ep_chk_max_items: 5000
      ep_chk_period: 1800
      ep_chk_persistence_remains: 0
      ep_chk_persistence_timeout: 20
      ep_chk_remover_stime: 5
      ep_commit_num: 666
      ep_commit_time: 5106
      ep_commit_time_total: 280469
      ep_concurrentDB: 1
      ep_config_file:
      ep_couch_bucket: default
      ep_couch_host: localhost
      ep_couch_port: 11213
      ep_couch_reconnect_sleeptime: 250
      ep_couch_response_timeout: 180000
      ep_data_age: 4156
      ep_data_age_highwat: 4200
      ep_data_traffic_enabled: 0
      ep_dbinit: 1
      ep_dbname: c:/Program Files/Couchbase/Server/var/lib/couchbase/data/default
      ep_degraded_mode: 0
      ep_diskqueue_drain: 5029279
      ep_diskqueue_fill: 5789725
      ep_diskqueue_items: 798546
      ep_diskqueue_memory: 24373760
      ep_diskqueue_pending: 41692209
      ep_exp_pager_stime: 3600
      ep_expired_access: 0
      ep_expired_pager: 0
      ep_expiry_window: 3
      ep_failpartialwarmup: 0
      ep_flush_all: false
      ep_flush_duration_total: 8736
      ep_flushall_enabled: 0
      ep_flusher_state: running
      ep_flusher_todo: 485497
      ep_getl_default_timeout: 15
      ep_getl_max_timeout: 30
      ep_ht_locks: 5
      ep_ht_size: 3079
      ep_inconsistent_slave_chk: 0
      ep_initfile:
      ep_io_num_read: 38
      ep_io_num_write: 4933340
      ep_io_read_bytes: 10745
      ep_io_write_bytes: 1382289598
      ep_item_begin_failed: 0
      ep_item_commit_failed: 0
      ep_item_flush_expired: 0
      ep_item_flush_failed: 0
      ep_item_num_based_new_chk: 1
      ep_items_rm_from_checkpoints: 4644424
      ep_keep_closed_chks: 0
      ep_klog_block_size: 4096
      ep_klog_compactor_queue_cap: 500000
      ep_klog_compactor_stime: 3600
      ep_klog_flush: commit2
      ep_klog_max_entry_ratio: 10
      ep_klog_max_log_size: 2147483647
      ep_klog_path:
      ep_klog_sync: commit2
      ep_kv_size: 571960225
      ep_max_checkpoints: 2
      ep_max_data_size: 838860800
      ep_max_item_size: 20971520
      ep_max_size: 838860800
      ep_max_txn_size: 10000
      ep_max_vbuckets: 1024
      ep_mem_high_wat: 629145600
      ep_mem_low_wat: 503316480
      ep_mem_tracker_enabled: true
      ep_min_data_age: 0
      ep_mlog_compactor_runs: 0
      ep_mutation_mem_threshold: 0
      ep_num_access_scanner_runs: 1
      ep_num_eject_failures: 347203302
      ep_num_expiry_pager_runs: 3
      ep_num_non_resident: 2976309
      ep_num_not_my_vbuckets: 6
      ep_num_ops_del_meta: 0
      ep_num_ops_get_meta: 1
      ep_num_ops_set_meta: 0
      ep_num_pager_runs: 412
      ep_num_value_ejects: 3336878
      ep_oom_errors: 0
      ep_overhead: 93967377
      ep_pager_active_vb_pcnt: 40
      ep_pager_unbiased_period: 60
      ep_pending_ops: 0
      ep_pending_ops_max: 0
      ep_pending_ops_max_duration: 0
      ep_pending_ops_total: 0
      ep_postInitfile:
      ep_queue_age_cap: 900
      ep_queue_size: 313049
      ep_startup_time: 1354086142
      ep_storage_age: 4154
      ep_storage_age_highwat: 4200
      ep_store_max_concurrency: 10
      ep_store_max_readers: 9
      ep_store_max_readwrite: 1
      ep_stored_val_type:
      ep_tap_ack_grace_period: 300
      ep_tap_ack_initial_sequence_number: 1
      ep_tap_ack_interval: 1000
      ep_tap_ack_window_size: 10
      ep_tap_backfill_resident: 0.9
      ep_tap_backlog_limit: 5000
      ep_tap_backoff_period: 5
      ep_tap_bg_fetch_requeued: 0
      ep_tap_bg_fetched: 1978
      ep_tap_bg_max_pending: 500
      ep_tap_keepalive: 300
      ep_tap_noop_interval: 20
      ep_tap_requeue_sleep_time: 0.1
      ep_tap_throttle_cap_pcnt: 10
      ep_tap_throttle_queue_cap: 1000000
      ep_tap_throttle_threshold: 90
      ep_tmp_oom_errors: 755084
      ep_too_old: 336634
      ep_too_young: 0
      ep_total_cache_size: 527866909
      ep_total_del_items: 0
      ep_total_enqueued: 6011817
      ep_total_new_items: 3142974
      ep_total_persisted: 4933340
      ep_uncommitted_items: 6277
      ep_value_size: 237995966
      ep_vb0: 0
      ep_vb_snapshot_total: 603
      ep_vb_total: 922
      ep_vbucket_del: 102
      ep_vbucket_del_avg_walltime: 1184789
      ep_vbucket_del_fail: 0
      ep_vbucket_del_max_walltime: 6511942
      ep_version: 2.0.0r_140_gde42c8c
      ep_waitforwarmup: 0
      ep_warmup: 1
      ep_warmup_batch_size: 1000
      ep_warmup_dups: 0
      ep_warmup_min_items_threshold: 100
      ep_warmup_min_memory_threshold: 100
      ep_warmup_oom: 0
      ep_warmup_thread: complete
      ep_warmup_time: 2049
      get_hits: 1165
      get_misses: 115
      incr_hits: 0
      incr_misses: 0
      libevent: 2.0.11-stable
      limit_maxbytes: 67108864
      listen_disabled_num: 0
      max_conns_on_port_11209: 1000
      max_conns_on_port_11210: 9000
      mem_used: 755127144
      pid: 2940
      pointer_size: 64
      rejected_conns: 0
      tap_checkpoint_end_received: 6135
      tap_checkpoint_end_sent: 9611
      tap_checkpoint_start_received: 6890
      tap_checkpoint_start_sent: 10551
      tap_connect_received: 310
      tap_mutation_received: 4241246
      tap_mutation_sent: 3973519
      tap_opaque_received: 1877
      tap_opaque_sent: 2073
      tap_vbucket_set_received: 242
      tap_vbucket_set_sent: 204
      threads: 4
      time: 1354099572
      total_connections: 1607
      uptime: 14555
      vb_active_curr_items: 2053924
      vb_active_eject: 2022287
      vb_active_expired: 0
      vb_active_ht_memory: 13219776
      vb_active_itm_memory: 252869508
      vb_active_meta_data_memory: 173860668
      vb_active_num: 531
      vb_active_num_non_resident: 1828041
      vb_active_num_ref_ejects: 1735016
      vb_active_num_ref_items: 105546
      vb_active_ops_create: 1841439
      vb_active_ops_delete: 0
      vb_active_ops_reject: 0
      vb_active_ops_update: 1024159
      vb_active_perc_mem_resident: 10
      vb_active_queue_age: 555566181000
      vb_active_queue_drain: 3012854
      vb_active_queue_fill: 3251701
      vb_active_queue_memory: 7682592
      vb_active_queue_pending: 13086939
      vb_active_queue_size: 240081
      vb_dead_num: 0
      vb_pending_curr_items: 0
      vb_pending_eject: 0
      vb_pending_expired: 0
      vb_pending_ht_memory: 0
      vb_pending_itm_memory: 0
      vb_pending_meta_data_memory: 0
      vb_pending_num: 0
      vb_pending_num_non_resident: 0
      vb_pending_num_ref_ejects: 0
      vb_pending_num_ref_items: 0
      vb_pending_ops_create: 0
      vb_pending_ops_delete: 0
      vb_pending_ops_reject: 0
      vb_pending_ops_update: 0
      vb_pending_perc_mem_resident: 0
      vb_pending_queue_age: 0
      vb_pending_queue_drain: 0
      vb_pending_queue_fill: 0
      vb_pending_queue_memory: 0
      vb_pending_queue_pending: 0
      vb_pending_queue_size: 0
      vb_replica_curr_items: 1620519
      vb_replica_eject: 1313893
      vb_replica_expired: 0
      vb_replica_ht_memory: 9734336
      vb_replica_itm_memory: 274997401
      vb_replica_meta_data_memory: 137308195
      vb_replica_num: 391
      vb_replica_num_non_resident: 1148268
      vb_replica_num_ref_ejects: 1277577
      vb_replica_num_ref_items: 90072
      vb_replica_ops_create: 1172973
      vb_replica_ops_delete: 0
      vb_replica_ops_reject: 0
      vb_replica_ops_update: 679912
      vb_replica_perc_mem_resident: 29
      vb_replica_queue_age: 1475291239000
      vb_replica_queue_drain: 2016425
      vb_replica_queue_fill: 2538024
      vb_replica_queue_memory: 16691168
      vb_replica_queue_pending: 28605270
      vb_replica_queue_size: 521599
      version: 1.4.4_600_g7ea975a


      ep_tap_ack_grace_period: 300
      ep_tap_ack_interval: 1000
      ep_tap_ack_window_size: 10
      ep_tap_backoff_period: 5
      ep_tap_bg_fetch_requeued: 0
      ep_tap_bg_fetched: 1978
      ep_tap_bg_max_pending: 500
      ep_tap_count: 6
      ep_tap_deletes: 0
      ep_tap_fg_fetched: 4042426
      ep_tap_noop_interval: 20
      ep_tap_queue_backfillremaining: 0
      ep_tap_queue_backoff: 118231
      ep_tap_queue_drain: 3906860
      ep_tap_queue_fill: 0
      ep_tap_queue_itemondisk: 0
      ep_tap_throttle_queue_cap: 1000000
      ep_tap_throttle_threshold: 90
      ep_tap_throttled: 1074548
      ep_tap_total_backlog_size: 181
      ep_tap_total_fetched: 4066843
      ep_tap_total_queue: 3359
      eq_tapq:anon_1:connected: true
      eq_tapq:anon_1:created: 1127
      eq_tapq:anon_1:num_checkpoint_end: 6057
      eq_tapq:anon_1:num_checkpoint_end_failed: 0
      eq_tapq:anon_1:num_checkpoint_start: 6569
      eq_tapq:anon_1:num_checkpoint_start_failed: 0
      eq_tapq:anon_1:num_delete: 0
      eq_tapq:anon_1:num_delete_failed: 0
      eq_tapq:anon_1:num_flush: 0
      eq_tapq:anon_1:num_flush_failed: 0
      eq_tapq:anon_1:num_mutation: 3257674
      eq_tapq:anon_1:num_mutation_failed: 1160954
      eq_tapq:anon_1:num_opaque: 1270
      eq_tapq:anon_1:num_opaque_failed: 0
      eq_tapq:anon_1:num_unknown: 0
      eq_tapq:anon_1:num_vbucket_set: 0
      eq_tapq:anon_1:num_vbucket_set_failed: 0
      eq_tapq:anon_1:pending_disconnect: false
      eq_tapq:anon_1:reserved: 0
      eq_tapq:anon_1:supports_ack: true
      eq_tapq:anon_1:type: consumer
      eq_tapq:anon_448:connected: true
      eq_tapq:anon_448:created: 13890
      eq_tapq:anon_448:num_checkpoint_end: 1
      eq_tapq:anon_448:num_checkpoint_end_failed: 0
      eq_tapq:anon_448:num_checkpoint_start: 2
      eq_tapq:anon_448:num_checkpoint_start_failed: 0
      eq_tapq:anon_448:num_delete: 0
      eq_tapq:anon_448:num_delete_failed: 0
      eq_tapq:anon_448:num_flush: 0
      eq_tapq:anon_448:num_flush_failed: 0
      eq_tapq:anon_448:num_mutation: 974
      eq_tapq:anon_448:num_mutation_failed: 8152
      eq_tapq:anon_448:num_opaque: 2
      eq_tapq:anon_448:num_opaque_failed: 0
      eq_tapq:anon_448:num_unknown: 0
      eq_tapq:anon_448:num_vbucket_set: 0
      eq_tapq:anon_448:num_vbucket_set_failed: 0
      eq_tapq:anon_448:pending_disconnect: false
      eq_tapq:anon_448:reserved: 0
      eq_tapq:anon_448:supports_ack: true
      eq_tapq:anon_448:type: consumer
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':ack_log_size: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':ack_seqno: 7446
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':ack_window_full: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':backfill_completed: true
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':backfill_start_timestamp: 1354092278
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':bg_jobs_completed: 1978
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':bg_jobs_issued: 1978
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':bg_result_size: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':connected: true
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':created: 7261
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':has_queued_item: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':idle: true
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':paused: 1
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':pending_backfill: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':pending_disconnect: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':pending_disk_backfill: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':qlen: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':qlen_high_pri: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':qlen_low_pri: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':queue_backfillremaining: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':queue_backoff: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':queue_drain: 7412
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':queue_fill: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':queue_itemondisk: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':queue_memory: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':rec_fetched: 5463
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':recv_ack_seqno: 7445
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':reserved: 1
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':seqno_ack_requested: 7445
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':supports_ack: true
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':suspended: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':total_backlog_size: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':total_noops: 106
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':type: producer
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':vb_filter: { 272 }
      eq_tapq:replication_building_272_'ns_1@10.1.3.135':vb_filters: 1
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':ack_log_size: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':ack_seqno: 4344
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':ack_window_full: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':backfill_completed: true
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':backfill_start_timestamp: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':bg_jobs_completed: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':bg_jobs_issued: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':bg_result_size: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':connected: true
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':created: 7261
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':has_queued_item: true
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':idle: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':paused: 1
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':pending_backfill: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':pending_disconnect: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':pending_disk_backfill: false
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':qlen: 1
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':qlen_high_pri: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':qlen_low_pri: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':queue_backfillremaining: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':queue_backoff: 234
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':queue_drain: 4314
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':queue_fill: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':queue_itemondisk: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':queue_memory: 8
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':rec_fetched: 4341
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':recv_ack_seqno: 4343
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':reserved: 1
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':seqno_ack_requested: 4343
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':supports_ack: true
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':suspended: true
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':total_backlog_size: 0
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':total_noops: 101
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':type: producer
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':vb_filter: { 272 }
      eq_tapq:replication_building_272_'ns_1@10.1.3.147':vb_filters: 1
      eq_tapq:replication_ns_1@10.1.3.135:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.135:ack_seqno: 569269
      eq_tapq:replication_ns_1@10.1.3.135:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.135:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.135:backfill_start_timestamp: 0
      eq_tapq:replication_ns_1@10.1.3.135:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.135:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.135:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.135:connected: true
      eq_tapq:replication_ns_1@10.1.3.135:created: 4873
      eq_tapq:replication_ns_1@10.1.3.135:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.135:has_queued_item: false
      eq_tapq:replication_ns_1@10.1.3.135:idle: true
      eq_tapq:replication_ns_1@10.1.3.135:paused: 1
      eq_tapq:replication_ns_1@10.1.3.135:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.135:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.135:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.135:qlen: 0
      eq_tapq:replication_ns_1@10.1.3.135:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.135:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_backoff: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_drain: 565288
      eq_tapq:replication_ns_1@10.1.3.135:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.135:queue_memory: 0
      eq_tapq:replication_ns_1@10.1.3.135:rec_fetched: 569146
      eq_tapq:replication_ns_1@10.1.3.135:recv_ack_seqno: 569268
      eq_tapq:replication_ns_1@10.1.3.135:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.135:seqno_ack_requested: 569268
      eq_tapq:replication_ns_1@10.1.3.135:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.135:suspended: false
      eq_tapq:replication_ns_1@10.1.3.135:total_backlog_size: 2
      eq_tapq:replication_ns_1@10.1.3.135:total_noops: 79
      eq_tapq:replication_ns_1@10.1.3.135:type: producer
      eq_tapq:replication_ns_1@10.1.3.135:vb_filter: { [682,802] }

      eq_tapq:replication_ns_1@10.1.3.135:vb_filters: 121
      eq_tapq:replication_ns_1@10.1.3.147:ack_log_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:ack_seqno: 3347761
      eq_tapq:replication_ns_1@10.1.3.147:ack_window_full: false
      eq_tapq:replication_ns_1@10.1.3.147:backfill_completed: true
      eq_tapq:replication_ns_1@10.1.3.147:backfill_start_timestamp: 1354086144
      eq_tapq:replication_ns_1@10.1.3.147:bg_jobs_completed: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_jobs_issued: 0
      eq_tapq:replication_ns_1@10.1.3.147:bg_result_size: 0
      eq_tapq:replication_ns_1@10.1.3.147:connected: true
      eq_tapq:replication_ns_1@10.1.3.147:created: 1127
      eq_tapq:replication_ns_1@10.1.3.147:flags: 85 (ack,backfill,vblist,checkpoints)
      eq_tapq:replication_ns_1@10.1.3.147:has_queued_item: true
      eq_tapq:replication_ns_1@10.1.3.147:idle: false
      eq_tapq:replication_ns_1@10.1.3.147:paused: 1
      eq_tapq:replication_ns_1@10.1.3.147:pending_backfill: false
      eq_tapq:replication_ns_1@10.1.3.147:pending_disconnect: false
      eq_tapq:replication_ns_1@10.1.3.147:pending_disk_backfill: false
      eq_tapq:replication_ns_1@10.1.3.147:qlen: 3358
      eq_tapq:replication_ns_1@10.1.3.147:qlen_high_pri: 0
      eq_tapq:replication_ns_1@10.1.3.147:qlen_low_pri: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_backfillremaining: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_backoff: 117997
      eq_tapq:replication_ns_1@10.1.3.147:queue_drain: 3329846
      eq_tapq:replication_ns_1@10.1.3.147:queue_fill: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_itemondisk: 0
      eq_tapq:replication_ns_1@10.1.3.147:queue_memory: 26864
      eq_tapq:replication_ns_1@10.1.3.147:rec_fetched: 3346631
      eq_tapq:replication_ns_1@10.1.3.147:recv_ack_seqno: 3347760
      eq_tapq:replication_ns_1@10.1.3.147:reserved: 1
      eq_tapq:replication_ns_1@10.1.3.147:seqno_ack_requested: 3347760
      eq_tapq:replication_ns_1@10.1.3.147:supports_ack: true
      eq_tapq:replication_ns_1@10.1.3.147:suspended: true
      eq_tapq:replication_ns_1@10.1.3.147:total_backlog_size: 179
      eq_tapq:replication_ns_1@10.1.3.147:total_noops: 143
      eq_tapq:replication_ns_1@10.1.3.147:type: producer
      eq_tapq:replication_ns_1@10.1.3.147:vb_filter:

      { [0,169], [273,511] }

      eq_tapq:replication_ns_1@10.1.3.147:vb_filters: 409

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        raju Raju Suravarjjala added a comment -

        Bulk closing all invalid bugs that are duplicate, user error, invalid. Please feel free to reopen them if you feel otherwise

        Show
        raju Raju Suravarjjala added a comment - Bulk closing all invalid bugs that are duplicate, user error, invalid. Please feel free to reopen them if you feel otherwise
        Hide
        maria Maria McDuff (Inactive) added a comment -

        tony: can you please try this use case and see if you are seeing the hanging in RC1 build 185? thanks.

        Show
        maria Maria McDuff (Inactive) added a comment - tony: can you please try this use case and see if you are seeing the hanging in RC1 build 185? thanks.
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        or closing this as unable to reproduce if we have not reproduced this with 2.0.1 builds

        Show
        farshid Farshid Ghods (Inactive) added a comment - or closing this as unable to reproduce if we have not reproduced this with 2.0.1 builds
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        given that this is not reproducible and can be worked around by rerunning rebalancing i recommend deferring this to 2.0.2

        Show
        farshid Farshid Ghods (Inactive) added a comment - given that this is not reproducible and can be worked around by rerunning rebalancing i recommend deferring this to 2.0.2
        Hide
        iryna iryna added a comment -

        tested with 2.0.1-141-rel, 2.0.1-145-rel. Not reproduced

        Show
        iryna iryna added a comment - tested with 2.0.1-141-rel, 2.0.1-145-rel. Not reproduced

          People

          • Assignee:
            thuan Thuan Nguyen
            Reporter:
            iryna iryna
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes