Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-3843

replica items missing in 4 node cluster( 2 items in 7 million items)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 1.7.0
    • 1.7 alpha 2
    • couchbase-bucket
    • Security Level: Public
    • None
    • basestar-260

    Description

      [2011-05-15 04:49:08,070] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:08,070] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:11,071] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:11,071] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:11,071] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:11,072] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:11,072] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:11,072] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:14,085] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:14,085] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:14,085] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:14,086] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:14,086] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:14,086] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:17,180] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:17,180] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:17,180] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:17,180] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:17,180] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:17,180] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:20,203] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:20,203] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:20,203] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:20,204] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:20,204] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:20,204] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:23,273] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:23,273] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:23,273] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:23,274] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:23,274] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:23,274] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:26,385] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:26,385] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:26,385] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:26,386] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:26,386] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:26,386] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:29,550] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:29,550] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:29,550] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:29,552] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:29,552] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:29,552] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:32,670] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:32,670] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:32,670] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:32,670] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:32,670] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:32,670] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:35,730] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:35,730] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:35,730] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:35,731] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:35,731] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:35,731] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:38,828] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:38,828] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:38,828] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:38,829] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:38,829] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:38,829] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:41,911] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:41,911] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:41,911] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:41,912] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:41,912] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:41,912] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:44,917] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:44,917] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:44,917] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:44,918] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:44,918] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:44,918] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:48,053] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:48,053] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:48,053] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:48,054] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:48,054] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:48,054] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:51,076] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:51,076] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:51,076] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:51,077] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:51,077] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:51,077] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:54,063] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:54,063] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:54,063] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:54,064] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:54,064] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:54,064] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:57,130] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:57,130] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:57,130] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:49:57,131] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:57,131] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:49:57,131] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:50:00,190] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:50:00,190] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:50:00,190] - [rebalance_helper] [47866199832992] - INFO - sum : 7943477
      [2011-05-15 04:50:00,191] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:50:00,191] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:50:00,191] - [rebalance_helper] [47866199832992] - INFO - master_stats : 23830429
      [2011-05-15 04:50:02,222] - [rebalance_helper] [47866199832992] - INFO - getting tap stats.. for 10.1.1.163
      [2011-05-15 04:50:02,222] - [rebalance_helper] [47866199832992] - INFO - getting tap stats.. for 10.1.1.163
      [2011-05-15 04:50:02,222] - [rebalance_helper] [47866199832992] - INFO - getting tap stats.. for 10.1.1.163
      [2011-05-15 04:50:02,780] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:has_item false
      [2011-05-15 04:50:02,780] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:has_item false
      [2011-05-15 04:50:02,780] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:has_item false
      [2011-05-15 04:50:02,781] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:pending_backfill false
      [2011-05-15 04:50:02,781] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:pending_backfill false
      [2011-05-15 04:50:02,781] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:pending_backfill false
      [2011-05-15 04:50:02,782] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:pending_backfill false
      [2011-05-15 04:50:02,782] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:pending_backfill false
      [2011-05-15 04:50:02,782] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:pending_backfill false
      [2011-05-15 04:50:02,783] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:has_queued_item true
      [2011-05-15 04:50:02,783] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:has_queued_item true
      [2011-05-15 04:50:02,783] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:has_queued_item true
      [2011-05-15 04:50:02,784] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:pending_backfill false
      [2011-05-15 04:50:02,784] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:pending_backfill false
      [2011-05-15 04:50:02,784] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:pending_backfill false
      [2011-05-15 04:50:02,784] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:ack_window_full false
      [2011-05-15 04:50:02,784] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:ack_window_full false
      [2011-05-15 04:50:02,784] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:ack_window_full false
      [2011-05-15 04:50:02,785] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:ack_window_full false
      [2011-05-15 04:50:02,785] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:ack_window_full false
      [2011-05-15 04:50:02,785] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:ack_window_full false
      [2011-05-15 04:50:02,786] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:ack_log_size 0
      [2011-05-15 04:50:02,786] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:ack_log_size 0
      [2011-05-15 04:50:02,786] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:ack_log_size 0
      [2011-05-15 04:50:02,787] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:has_queued_item true
      [2011-05-15 04:50:02,787] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:has_queued_item true
      [2011-05-15 04:50:02,787] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:has_queued_item true
      [2011-05-15 04:50:02,788] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:ack_seqno 3986084
      [2011-05-15 04:50:02,788] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:ack_seqno 3986084
      [2011-05-15 04:50:02,788] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:ack_seqno 3986084
      [2011-05-15 04:50:02,788] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:ack_window_full false
      [2011-05-15 04:50:02,788] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:ack_window_full false
      [2011-05-15 04:50:02,788] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:ack_window_full false
      [2011-05-15 04:50:02,789] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:ack_log_size 0
      [2011-05-15 04:50:02,789] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:ack_log_size 0
      [2011-05-15 04:50:02,789] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:ack_log_size 0
      [2011-05-15 04:50:02,790] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:paused 3
      [2011-05-15 04:50:02,790] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:paused 3
      [2011-05-15 04:50:02,790] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:paused 3
      [2011-05-15 04:50:02,791] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:has_item false
      [2011-05-15 04:50:02,791] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:has_item false
      [2011-05-15 04:50:02,791] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:has_item false
      [2011-05-15 04:50:02,791] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:paused 3
      [2011-05-15 04:50:02,791] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:paused 3
      [2011-05-15 04:50:02,791] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:paused 3
      [2011-05-15 04:50:02,792] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:idle false
      [2011-05-15 04:50:02,792] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:idle false
      [2011-05-15 04:50:02,792] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:idle false
      [2011-05-15 04:50:02,793] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:recv_ack_seqno 3985600
      [2011-05-15 04:50:02,793] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:recv_ack_seqno 3985600
      [2011-05-15 04:50:02,793] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:recv_ack_seqno 3985600
      [2011-05-15 04:50:02,794] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:idle false
      [2011-05-15 04:50:02,794] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:idle false
      [2011-05-15 04:50:02,794] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:idle false
      [2011-05-15 04:50:02,795] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:has_item false
      [2011-05-15 04:50:02,795] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:has_item false
      [2011-05-15 04:50:02,795] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:has_item false
      [2011-05-15 04:50:02,796] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:ack_seqno 3985571
      [2011-05-15 04:50:02,796] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:ack_seqno 3985571
      [2011-05-15 04:50:02,796] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:ack_seqno 3985571
      [2011-05-15 04:50:02,796] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:pending_disk_backfill false
      [2011-05-15 04:50:02,796] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:pending_disk_backfill false
      [2011-05-15 04:50:02,796] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:pending_disk_backfill false
      [2011-05-15 04:50:02,797] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:pending_backfill false
      [2011-05-15 04:50:02,797] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:pending_backfill false
      [2011-05-15 04:50:02,797] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:pending_backfill false
      [2011-05-15 04:50:02,798] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:ack_window_full false
      [2011-05-15 04:50:02,798] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:ack_window_full false
      [2011-05-15 04:50:02,798] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:ack_window_full false
      [2011-05-15 04:50:02,799] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:idle true
      [2011-05-15 04:50:02,799] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:idle true
      [2011-05-15 04:50:02,799] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:idle true
      [2011-05-15 04:50:02,800] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:pending_disk_backfill false
      [2011-05-15 04:50:02,800] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:pending_disk_backfill false
      [2011-05-15 04:50:02,800] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:pending_disk_backfill false
      [2011-05-15 04:50:02,801] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:idle true
      [2011-05-15 04:50:02,801] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:idle true
      [2011-05-15 04:50:02,801] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:idle true
      [2011-05-15 04:50:02,801] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:ack_seqno 3985603
      [2011-05-15 04:50:02,801] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:ack_seqno 3985603
      [2011-05-15 04:50:02,801] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:ack_seqno 3985603
      [2011-05-15 04:50:02,802] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:recv_ack_seqno 3986081
      [2011-05-15 04:50:02,802] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:recv_ack_seqno 3986081
      [2011-05-15 04:50:02,802] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:recv_ack_seqno 3986081
      [2011-05-15 04:50:02,803] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:has_item false
      [2011-05-15 04:50:02,803] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:has_item false
      [2011-05-15 04:50:02,803] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:has_item false
      [2011-05-15 04:50:02,804] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:has_queued_item false
      [2011-05-15 04:50:02,804] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:has_queued_item false
      [2011-05-15 04:50:02,804] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:has_queued_item false
      [2011-05-15 04:50:02,805] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:paused 3
      [2011-05-15 04:50:02,805] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:paused 3
      [2011-05-15 04:50:02,805] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:paused 3
      [2011-05-15 04:50:02,805] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:ack_seqno 3986080
      [2011-05-15 04:50:02,805] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:ack_seqno 3986080
      [2011-05-15 04:50:02,805] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:ack_seqno 3986080
      [2011-05-15 04:50:02,806] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:pending_disk_backfill false
      [2011-05-15 04:50:02,806] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:pending_disk_backfill false
      [2011-05-15 04:50:02,806] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:pending_disk_backfill false
      [2011-05-15 04:50:02,808] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:ack_log_size 0
      [2011-05-15 04:50:02,808] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:ack_log_size 0
      [2011-05-15 04:50:02,808] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:ack_log_size 0
      [2011-05-15 04:50:02,808] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:has_queued_item false
      [2011-05-15 04:50:02,808] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:has_queued_item false
      [2011-05-15 04:50:02,808] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:has_queued_item false
      [2011-05-15 04:50:02,809] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:ack_log_size 0
      [2011-05-15 04:50:02,809] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:ack_log_size 0
      [2011-05-15 04:50:02,809] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.152:ack_log_size 0
      [2011-05-15 04:50:02,810] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:paused 3
      [2011-05-15 04:50:02,810] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:paused 3
      [2011-05-15 04:50:02,810] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:paused 3
      [2011-05-15 04:50:02,811] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:recv_ack_seqno 3986077
      [2011-05-15 04:50:02,811] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:recv_ack_seqno 3986077
      [2011-05-15 04:50:02,811] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.162:recv_ack_seqno 3986077
      [2011-05-15 04:50:02,812] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:recv_ack_seqno 3985568
      [2011-05-15 04:50:02,812] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:recv_ack_seqno 3985568
      [2011-05-15 04:50:02,812] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.151:recv_ack_seqno 3985568
      [2011-05-15 04:50:02,812] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:pending_disk_backfill false
      [2011-05-15 04:50:02,812] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:pending_disk_backfill false
      [2011-05-15 04:50:02,812] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.163 :eq_tapq:replication_ns_1@10.1.1.163:pending_disk_backfill false
      [2011-05-15 04:50:02,819] - [rebalance_helper] [47866199832992] - INFO - getting tap stats.. for 10.1.1.162
      [2011-05-15 04:50:02,819] - [rebalance_helper] [47866199832992] - INFO - getting tap stats.. for 10.1.1.162
      [2011-05-15 04:50:02,819] - [rebalance_helper] [47866199832992] - INFO - getting tap stats.. for 10.1.1.162
      [2011-05-15 04:50:03,326] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:has_item false
      [2011-05-15 04:50:03,326] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:has_item false
      [2011-05-15 04:50:03,326] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:has_item false
      [2011-05-15 04:50:03,327] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:pending_backfill false
      [2011-05-15 04:50:03,327] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:pending_backfill false
      [2011-05-15 04:50:03,327] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:pending_backfill false
      [2011-05-15 04:50:03,328] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:pending_backfill false
      [2011-05-15 04:50:03,328] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:pending_backfill false
      [2011-05-15 04:50:03,328] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:pending_backfill false
      [2011-05-15 04:50:03,328] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:has_queued_item false
      [2011-05-15 04:50:03,328] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:has_queued_item false
      [2011-05-15 04:50:03,328] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:has_queued_item false
      [2011-05-15 04:50:03,329] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:pending_backfill false
      [2011-05-15 04:50:03,329] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:pending_backfill false
      [2011-05-15 04:50:03,329] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:pending_backfill false
      [2011-05-15 04:50:03,330] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:ack_window_full false
      [2011-05-15 04:50:03,330] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:ack_window_full false
      [2011-05-15 04:50:03,330] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:ack_window_full false
      [2011-05-15 04:50:03,331] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:ack_window_full false
      [2011-05-15 04:50:03,331] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:ack_window_full false
      [2011-05-15 04:50:03,331] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:ack_window_full false
      [2011-05-15 04:50:03,332] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:ack_log_size 0
      [2011-05-15 04:50:03,332] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:ack_log_size 0
      [2011-05-15 04:50:03,332] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:ack_log_size 0
      [2011-05-15 04:50:03,333] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:has_queued_item true
      [2011-05-15 04:50:03,333] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:has_queued_item true
      [2011-05-15 04:50:03,333] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:has_queued_item true
      [2011-05-15 04:50:03,333] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:ack_seqno 3986084
      [2011-05-15 04:50:03,333] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:ack_seqno 3986084
      [2011-05-15 04:50:03,333] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:ack_seqno 3986084
      [2011-05-15 04:50:03,334] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:ack_window_full false
      [2011-05-15 04:50:03,334] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:ack_window_full false
      [2011-05-15 04:50:03,334] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:ack_window_full false
      [2011-05-15 04:50:03,335] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:ack_log_size 0
      [2011-05-15 04:50:03,335] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:ack_log_size 0
      [2011-05-15 04:50:03,335] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:ack_log_size 0
      [2011-05-15 04:50:03,336] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:paused 3
      [2011-05-15 04:50:03,336] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:paused 3
      [2011-05-15 04:50:03,336] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:paused 3
      [2011-05-15 04:50:03,337] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:has_item false
      [2011-05-15 04:50:03,337] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:has_item false
      [2011-05-15 04:50:03,337] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:has_item false
      [2011-05-15 04:50:03,338] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:paused 3
      [2011-05-15 04:50:03,338] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:paused 3
      [2011-05-15 04:50:03,338] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:paused 3
      [2011-05-15 04:50:03,338] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:idle false
      [2011-05-15 04:50:03,338] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:idle false
      [2011-05-15 04:50:03,338] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:idle false
      [2011-05-15 04:50:03,339] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:recv_ack_seqno 3985600
      [2011-05-15 04:50:03,339] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:recv_ack_seqno 3985600
      [2011-05-15 04:50:03,339] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:recv_ack_seqno 3985600
      [2011-05-15 04:50:03,340] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:idle true
      [2011-05-15 04:50:03,340] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:idle true
      [2011-05-15 04:50:03,340] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:idle true
      [2011-05-15 04:50:03,341] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:has_item false
      [2011-05-15 04:50:03,341] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:has_item false
      [2011-05-15 04:50:03,341] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:has_item false
      [2011-05-15 04:50:03,342] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:ack_seqno 3985571
      [2011-05-15 04:50:03,342] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:ack_seqno 3985571
      [2011-05-15 04:50:03,342] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:ack_seqno 3985571
      [2011-05-15 04:50:03,343] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:pending_disk_backfill false
      [2011-05-15 04:50:03,343] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:pending_disk_backfill false
      [2011-05-15 04:50:03,343] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:pending_disk_backfill false
      [2011-05-15 04:50:03,343] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:pending_backfill false
      [2011-05-15 04:50:03,343] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:pending_backfill false
      [2011-05-15 04:50:03,343] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:pending_backfill false
      [2011-05-15 04:50:03,344] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:ack_window_full false
      [2011-05-15 04:50:03,344] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:ack_window_full false
      [2011-05-15 04:50:03,344] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:ack_window_full false
      [2011-05-15 04:50:03,345] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:idle true
      [2011-05-15 04:50:03,345] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:idle true
      [2011-05-15 04:50:03,345] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:idle true
      [2011-05-15 04:50:03,346] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:pending_disk_backfill false
      [2011-05-15 04:50:03,346] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:pending_disk_backfill false
      [2011-05-15 04:50:03,346] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:pending_disk_backfill false
      [2011-05-15 04:50:03,347] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:idle true
      [2011-05-15 04:50:03,347] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:idle true
      [2011-05-15 04:50:03,347] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:idle true
      [2011-05-15 04:50:03,348] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:ack_seqno 3985603
      [2011-05-15 04:50:03,348] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:ack_seqno 3985603
      [2011-05-15 04:50:03,348] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:ack_seqno 3985603
      [2011-05-15 04:50:03,348] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:recv_ack_seqno 3986081
      [2011-05-15 04:50:03,348] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:recv_ack_seqno 3986081
      [2011-05-15 04:50:03,348] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:recv_ack_seqno 3986081
      [2011-05-15 04:50:03,349] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:has_item false
      [2011-05-15 04:50:03,349] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:has_item false
      [2011-05-15 04:50:03,349] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:has_item false
      [2011-05-15 04:50:03,350] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:has_queued_item false
      [2011-05-15 04:50:03,350] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:has_queued_item false
      [2011-05-15 04:50:03,350] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:has_queued_item false
      [2011-05-15 04:50:03,351] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:paused 3
      [2011-05-15 04:50:03,351] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:paused 3
      [2011-05-15 04:50:03,351] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:paused 3
      [2011-05-15 04:50:03,352] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:ack_seqno 3986080
      [2011-05-15 04:50:03,352] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:ack_seqno 3986080
      [2011-05-15 04:50:03,352] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:ack_seqno 3986080
      [2011-05-15 04:50:03,352] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:pending_disk_backfill false
      [2011-05-15 04:50:03,352] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:pending_disk_backfill false
      [2011-05-15 04:50:03,352] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:pending_disk_backfill false
      [2011-05-15 04:50:03,353] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:ack_log_size 0
      [2011-05-15 04:50:03,353] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:ack_log_size 0
      [2011-05-15 04:50:03,353] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:ack_log_size 0
      [2011-05-15 04:50:03,354] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:has_queued_item false
      [2011-05-15 04:50:03,354] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:has_queued_item false
      [2011-05-15 04:50:03,354] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:has_queued_item false
      [2011-05-15 04:50:03,355] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:ack_log_size 0
      [2011-05-15 04:50:03,355] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:ack_log_size 0
      [2011-05-15 04:50:03,355] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.152:ack_log_size 0
      [2011-05-15 04:50:03,356] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:paused 3
      [2011-05-15 04:50:03,356] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:paused 3
      [2011-05-15 04:50:03,356] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:paused 3
      [2011-05-15 04:50:03,356] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:recv_ack_seqno 3986077
      [2011-05-15 04:50:03,356] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:recv_ack_seqno 3986077
      [2011-05-15 04:50:03,356] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.162:recv_ack_seqno 3986077
      [2011-05-15 04:50:03,357] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:recv_ack_seqno 3985568
      [2011-05-15 04:50:03,357] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:recv_ack_seqno 3985568
      [2011-05-15 04:50:03,357] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.151:recv_ack_seqno 3985568
      [2011-05-15 04:50:03,358] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:pending_disk_backfill false
      [2011-05-15 04:50:03,358] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:pending_disk_backfill false
      [2011-05-15 04:50:03,358] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.162 :eq_tapq:replication_ns_1@10.1.1.163:pending_disk_backfill false
      [2011-05-15 04:50:03,363] - [rebalance_helper] [47866199832992] - INFO - getting tap stats.. for 10.1.1.152
      [2011-05-15 04:50:03,363] - [rebalance_helper] [47866199832992] - INFO - getting tap stats.. for 10.1.1.152
      [2011-05-15 04:50:03,363] - [rebalance_helper] [47866199832992] - INFO - getting tap stats.. for 10.1.1.152
      [2011-05-15 04:50:03,514] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:has_item false
      [2011-05-15 04:50:03,514] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:has_item false
      [2011-05-15 04:50:03,514] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:has_item false
      [2011-05-15 04:50:03,515] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:pending_backfill false
      [2011-05-15 04:50:03,515] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:pending_backfill false
      [2011-05-15 04:50:03,515] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:pending_backfill false
      [2011-05-15 04:50:03,516] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:pending_backfill false
      [2011-05-15 04:50:03,516] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:pending_backfill false
      [2011-05-15 04:50:03,516] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:pending_backfill false
      [2011-05-15 04:50:03,517] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:has_queued_item true
      [2011-05-15 04:50:03,517] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:has_queued_item true
      [2011-05-15 04:50:03,517] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:has_queued_item true
      [2011-05-15 04:50:03,518] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:pending_backfill false
      [2011-05-15 04:50:03,518] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:pending_backfill false
      [2011-05-15 04:50:03,518] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:pending_backfill false
      [2011-05-15 04:50:03,519] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:ack_window_full false
      [2011-05-15 04:50:03,519] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:ack_window_full false
      [2011-05-15 04:50:03,519] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:ack_window_full false
      [2011-05-15 04:50:03,520] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:ack_window_full false
      [2011-05-15 04:50:03,520] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:ack_window_full false
      [2011-05-15 04:50:03,520] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:ack_window_full false
      [2011-05-15 04:50:03,521] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:ack_log_size 0
      [2011-05-15 04:50:03,521] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:ack_log_size 0
      [2011-05-15 04:50:03,521] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:ack_log_size 0
      [2011-05-15 04:50:03,522] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:has_queued_item false
      [2011-05-15 04:50:03,522] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:has_queued_item false
      [2011-05-15 04:50:03,522] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:has_queued_item false
      [2011-05-15 04:50:03,522] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:ack_seqno 3986084
      [2011-05-15 04:50:03,522] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:ack_seqno 3986084
      [2011-05-15 04:50:03,522] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:ack_seqno 3986084
      [2011-05-15 04:50:03,523] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:ack_window_full false
      [2011-05-15 04:50:03,523] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:ack_window_full false
      [2011-05-15 04:50:03,523] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:ack_window_full false
      [2011-05-15 04:50:03,524] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:ack_log_size 0
      [2011-05-15 04:50:03,524] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:ack_log_size 0
      [2011-05-15 04:50:03,524] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:ack_log_size 0
      [2011-05-15 04:50:03,525] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:paused 3
      [2011-05-15 04:50:03,525] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:paused 3
      [2011-05-15 04:50:03,525] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:paused 3
      [2011-05-15 04:50:03,526] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:has_item false
      [2011-05-15 04:50:03,526] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:has_item false
      [2011-05-15 04:50:03,526] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:has_item false
      [2011-05-15 04:50:03,527] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:paused 3
      [2011-05-15 04:50:03,527] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:paused 3
      [2011-05-15 04:50:03,527] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:paused 3
      [2011-05-15 04:50:03,528] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:idle true
      [2011-05-15 04:50:03,528] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:idle true
      [2011-05-15 04:50:03,528] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:idle true
      [2011-05-15 04:50:03,529] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:recv_ack_seqno 3985600
      [2011-05-15 04:50:03,529] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:recv_ack_seqno 3985600
      [2011-05-15 04:50:03,529] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:recv_ack_seqno 3985600
      [2011-05-15 04:50:03,529] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:idle false
      [2011-05-15 04:50:03,529] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:idle false
      [2011-05-15 04:50:03,529] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:idle false
      [2011-05-15 04:50:03,530] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:has_item false
      [2011-05-15 04:50:03,530] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:has_item false
      [2011-05-15 04:50:03,530] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:has_item false
      [2011-05-15 04:50:03,531] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:ack_seqno 3985571
      [2011-05-15 04:50:03,531] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:ack_seqno 3985571
      [2011-05-15 04:50:03,531] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:ack_seqno 3985571
      [2011-05-15 04:50:03,532] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:pending_disk_backfill false
      [2011-05-15 04:50:03,532] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:pending_disk_backfill false
      [2011-05-15 04:50:03,532] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:pending_disk_backfill false
      [2011-05-15 04:50:03,533] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:pending_backfill false
      [2011-05-15 04:50:03,533] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:pending_backfill false
      [2011-05-15 04:50:03,533] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:pending_backfill false
      [2011-05-15 04:50:03,534] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:ack_window_full false
      [2011-05-15 04:50:03,534] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:ack_window_full false
      [2011-05-15 04:50:03,534] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:ack_window_full false
      [2011-05-15 04:50:03,535] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:idle false
      [2011-05-15 04:50:03,535] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:idle false
      [2011-05-15 04:50:03,535] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:idle false
      [2011-05-15 04:50:03,536] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:pending_disk_backfill false
      [2011-05-15 04:50:03,536] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:pending_disk_backfill false
      [2011-05-15 04:50:03,536] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:pending_disk_backfill false
      [2011-05-15 04:50:03,537] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:idle true
      [2011-05-15 04:50:03,537] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:idle true
      [2011-05-15 04:50:03,537] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:idle true
      [2011-05-15 04:50:03,538] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:ack_seqno 3985603
      [2011-05-15 04:50:03,538] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:ack_seqno 3985603
      [2011-05-15 04:50:03,538] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:ack_seqno 3985603
      [2011-05-15 04:50:03,538] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:recv_ack_seqno 3986081
      [2011-05-15 04:50:03,538] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:recv_ack_seqno 3986081
      [2011-05-15 04:50:03,538] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:recv_ack_seqno 3986081
      [2011-05-15 04:50:03,539] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:has_item false
      [2011-05-15 04:50:03,539] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:has_item false
      [2011-05-15 04:50:03,539] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:has_item false
      [2011-05-15 04:50:03,540] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:has_queued_item true
      [2011-05-15 04:50:03,540] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:has_queued_item true
      [2011-05-15 04:50:03,540] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:has_queued_item true
      [2011-05-15 04:50:03,541] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:paused 3
      [2011-05-15 04:50:03,541] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:paused 3
      [2011-05-15 04:50:03,541] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:paused 3
      [2011-05-15 04:50:03,542] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:ack_seqno 3986080
      [2011-05-15 04:50:03,542] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:ack_seqno 3986080
      [2011-05-15 04:50:03,542] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:ack_seqno 3986080
      [2011-05-15 04:50:03,543] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:pending_disk_backfill false
      [2011-05-15 04:50:03,543] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:pending_disk_backfill false
      [2011-05-15 04:50:03,543] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:pending_disk_backfill false
      [2011-05-15 04:50:03,544] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:ack_log_size 0
      [2011-05-15 04:50:03,544] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:ack_log_size 0
      [2011-05-15 04:50:03,544] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:ack_log_size 0
      [2011-05-15 04:50:03,544] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:has_queued_item false
      [2011-05-15 04:50:03,544] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:has_queued_item false
      [2011-05-15 04:50:03,544] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:has_queued_item false
      [2011-05-15 04:50:03,545] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:ack_log_size 0
      [2011-05-15 04:50:03,545] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:ack_log_size 0
      [2011-05-15 04:50:03,545] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.152:ack_log_size 0
      [2011-05-15 04:50:03,546] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:paused 3
      [2011-05-15 04:50:03,546] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:paused 3
      [2011-05-15 04:50:03,546] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:paused 3
      [2011-05-15 04:50:03,547] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:recv_ack_seqno 3986077
      [2011-05-15 04:50:03,547] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:recv_ack_seqno 3986077
      [2011-05-15 04:50:03,547] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.162:recv_ack_seqno 3986077
      [2011-05-15 04:50:03,548] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:recv_ack_seqno 3985568
      [2011-05-15 04:50:03,548] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:recv_ack_seqno 3985568
      [2011-05-15 04:50:03,548] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.151:recv_ack_seqno 3985568
      [2011-05-15 04:50:03,549] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:pending_disk_backfill false
      [2011-05-15 04:50:03,549] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:pending_disk_backfill false
      [2011-05-15 04:50:03,549] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.152 :eq_tapq:replication_ns_1@10.1.1.163:pending_disk_backfill false
      [2011-05-15 04:50:03,552] - [rebalance_helper] [47866199832992] - INFO - getting tap stats.. for 10.1.1.151
      [2011-05-15 04:50:03,552] - [rebalance_helper] [47866199832992] - INFO - getting tap stats.. for 10.1.1.151
      [2011-05-15 04:50:03,552] - [rebalance_helper] [47866199832992] - INFO - getting tap stats.. for 10.1.1.151
      [2011-05-15 04:50:03,676] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:has_item false
      [2011-05-15 04:50:03,676] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:has_item false
      [2011-05-15 04:50:03,676] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:has_item false
      [2011-05-15 04:50:03,677] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:pending_backfill false
      [2011-05-15 04:50:03,677] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:pending_backfill false
      [2011-05-15 04:50:03,677] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:pending_backfill false
      [2011-05-15 04:50:03,678] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:idle true
      [2011-05-15 04:50:03,678] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:idle true
      [2011-05-15 04:50:03,678] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:idle true
      [2011-05-15 04:50:03,678] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:pending_backfill false
      [2011-05-15 04:50:03,678] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:pending_backfill false
      [2011-05-15 04:50:03,678] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:pending_backfill false
      [2011-05-15 04:50:03,679] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:has_queued_item true
      [2011-05-15 04:50:03,679] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:has_queued_item true
      [2011-05-15 04:50:03,679] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:has_queued_item true
      [2011-05-15 04:50:03,680] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:pending_backfill false
      [2011-05-15 04:50:03,680] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:pending_backfill false
      [2011-05-15 04:50:03,680] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:pending_backfill false
      [2011-05-15 04:50:03,681] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:ack_window_full false
      [2011-05-15 04:50:03,681] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:ack_window_full false
      [2011-05-15 04:50:03,681] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:ack_window_full false
      [2011-05-15 04:50:03,682] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:ack_window_full false
      [2011-05-15 04:50:03,682] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:ack_window_full false
      [2011-05-15 04:50:03,682] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:ack_window_full false
      [2011-05-15 04:50:03,683] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:ack_log_size 0
      [2011-05-15 04:50:03,683] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:ack_log_size 0
      [2011-05-15 04:50:03,683] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:ack_log_size 0
      [2011-05-15 04:50:03,683] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:has_queued_item false
      [2011-05-15 04:50:03,683] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:has_queued_item false
      [2011-05-15 04:50:03,683] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:has_queued_item false
      [2011-05-15 04:50:03,684] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:ack_seqno 3986182
      [2011-05-15 04:50:03,684] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:ack_seqno 3986182
      [2011-05-15 04:50:03,684] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:ack_seqno 3986182
      [2011-05-15 04:50:03,685] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:ack_window_full false
      [2011-05-15 04:50:03,685] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:ack_window_full false
      [2011-05-15 04:50:03,685] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:ack_window_full false
      [2011-05-15 04:50:03,686] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:pending_disk_backfill false
      [2011-05-15 04:50:03,686] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:pending_disk_backfill false
      [2011-05-15 04:50:03,686] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:pending_disk_backfill false
      [2011-05-15 04:50:03,687] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:ack_log_size 43
      [2011-05-15 04:50:03,687] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:ack_log_size 43
      [2011-05-15 04:50:03,687] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:ack_log_size 43
      [2011-05-15 04:50:03,687] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:paused 3
      [2011-05-15 04:50:03,687] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:paused 3
      [2011-05-15 04:50:03,687] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:paused 3
      [2011-05-15 04:50:03,688] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:has_item false
      [2011-05-15 04:50:03,688] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:has_item false
      [2011-05-15 04:50:03,688] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:has_item false
      [2011-05-15 04:50:03,689] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:paused 3
      [2011-05-15 04:50:03,689] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:paused 3
      [2011-05-15 04:50:03,689] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:paused 3
      [2011-05-15 04:50:03,690] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:idle false
      [2011-05-15 04:50:03,690] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:idle false
      [2011-05-15 04:50:03,690] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:idle false
      [2011-05-15 04:50:03,691] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:recv_ack_seqno 3985600
      [2011-05-15 04:50:03,691] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:recv_ack_seqno 3985600
      [2011-05-15 04:50:03,691] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:recv_ack_seqno 3985600
      [2011-05-15 04:50:03,691] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:idle false
      [2011-05-15 04:50:03,691] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:idle false
      [2011-05-15 04:50:03,691] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:idle false
      [2011-05-15 04:50:03,692] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:has_item false
      [2011-05-15 04:50:03,692] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:has_item false
      [2011-05-15 04:50:03,692] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:has_item false
      [2011-05-15 04:50:03,693] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:ack_seqno 3985571
      [2011-05-15 04:50:03,693] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:ack_seqno 3985571
      [2011-05-15 04:50:03,693] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:ack_seqno 3985571
      [2011-05-15 04:50:03,694] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:pending_disk_backfill false
      [2011-05-15 04:50:03,694] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:pending_disk_backfill false
      [2011-05-15 04:50:03,694] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:pending_disk_backfill false
      [2011-05-15 04:50:03,695] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:pending_backfill false
      [2011-05-15 04:50:03,695] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:pending_backfill false
      [2011-05-15 04:50:03,695] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:pending_backfill false
      [2011-05-15 04:50:03,696] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:idle true
      [2011-05-15 04:50:03,696] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:idle true
      [2011-05-15 04:50:03,696] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:idle true
      [2011-05-15 04:50:03,697] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:ack_seqno 3985603
      [2011-05-15 04:50:03,697] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:ack_seqno 3985603
      [2011-05-15 04:50:03,697] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:ack_seqno 3985603
      [2011-05-15 04:50:03,698] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:recv_ack_seqno 3986136
      [2011-05-15 04:50:03,698] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:recv_ack_seqno 3986136
      [2011-05-15 04:50:03,698] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:recv_ack_seqno 3986136
      [2011-05-15 04:50:03,698] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:has_item false
      [2011-05-15 04:50:03,698] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:has_item false
      [2011-05-15 04:50:03,698] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:has_item false
      [2011-05-15 04:50:03,699] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:has_queued_item false
      [2011-05-15 04:50:03,699] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:has_queued_item false
      [2011-05-15 04:50:03,699] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:has_queued_item false
      [2011-05-15 04:50:03,700] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:paused 3
      [2011-05-15 04:50:03,700] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:paused 3
      [2011-05-15 04:50:03,700] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:paused 3
      [2011-05-15 04:50:03,701] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:ack_seqno 3986084
      [2011-05-15 04:50:03,701] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:ack_seqno 3986084
      [2011-05-15 04:50:03,701] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:ack_seqno 3986084
      [2011-05-15 04:50:03,701] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:pending_disk_backfill false
      [2011-05-15 04:50:03,701] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:pending_disk_backfill false
      [2011-05-15 04:50:03,701] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:pending_disk_backfill false
      [2011-05-15 04:50:03,702] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:ack_log_size 0
      [2011-05-15 04:50:03,702] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:ack_log_size 0
      [2011-05-15 04:50:03,702] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:ack_log_size 0
      [2011-05-15 04:50:03,703] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:has_queued_item false
      [2011-05-15 04:50:03,703] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:has_queued_item false
      [2011-05-15 04:50:03,703] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:has_queued_item false
      [2011-05-15 04:50:03,704] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:ack_log_size 0
      [2011-05-15 04:50:03,704] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:ack_log_size 0
      [2011-05-15 04:50:03,704] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.152:ack_log_size 0
      [2011-05-15 04:50:03,705] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:paused 3
      [2011-05-15 04:50:03,705] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:paused 3
      [2011-05-15 04:50:03,705] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:paused 3
      [2011-05-15 04:50:03,705] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:recv_ack_seqno 3986081
      [2011-05-15 04:50:03,705] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:recv_ack_seqno 3986081
      [2011-05-15 04:50:03,705] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.162:recv_ack_seqno 3986081
      [2011-05-15 04:50:03,706] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:ack_window_full false
      [2011-05-15 04:50:03,706] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:ack_window_full false
      [2011-05-15 04:50:03,706] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:ack_window_full false
      [2011-05-15 04:50:03,707] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:recv_ack_seqno 3985568
      [2011-05-15 04:50:03,707] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:recv_ack_seqno 3985568
      [2011-05-15 04:50:03,707] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.151:recv_ack_seqno 3985568
      [2011-05-15 04:50:03,708] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:pending_disk_backfill false
      [2011-05-15 04:50:03,708] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:pending_disk_backfill false
      [2011-05-15 04:50:03,708] - [rebalance_helper] [47866199832992] - INFO - TAP ns_1@10.1.1.151 :eq_tapq:replication_ns_1@10.1.1.163:pending_disk_backfill false
      FAIL
      [2011-05-15 04:50:03,710] - [bucket_helper] [47866199832992] - INFO - deleting existing buckets on [ip:10.1.1.152 ssh_username:root]
      [2011-05-15 04:50:04,381] - [bucket_helper] [47866199832992] - INFO - deleted bucket : replica-2-ram-50-0dce53d3-165f-45d1-82cd-b649c4854c13 from 10.1.1.152
      [2011-05-15 04:50:04,381] - [bucket_helper] [47866199832992] - INFO - waiting for bucket deletion to complete....
      [2011-05-15 04:50:04,386] - [rest_client] [47866199832992] - INFO - existing buckets : []
      [2011-05-15 04:50:04,386] - [bucket_helper] [47866199832992] - INFO - sleeping for 10 seconds because we want to
      [2011-05-15 04:50:14,392] - [cluster_helper] [47866199832992] - INFO - rebalancing all nodes in order to remove nodes
      [2011-05-15 04:50:14,393] - [rest_client] [47866199832992] - INFO - rebalanace params : password=password&ejectedNodes=ns_1%4010.1.1.163%2Cns_1%4010.1.1.162%2Cns_1%4010.1.1.151&user=Administrator&knownNodes=ns_1%4010.1.1.163%2Cns_1%4010.1.1.162%2Cns_1%4010.1.1.152%2Cns_1%4010.1.1.151
      [2011-05-15 04:50:14,504] - [rest_client] [47866199832992] - INFO - rebalance:

      {'date': 'Sun, 15 May 2011 11:50:14 GMT', 'status': '200', 'content-length': '0', 'server': 'MochiWeb/1.0 (Any of you quaids got a smint?)'}

      [2011-05-15 04:50:14,505] - [rest_client] [47866199832992] - INFO - rebalance operation started
      [2011-05-15 04:50:16,515] - [rest_client] [47866199832992] - INFO - rebalance progress took 2.01015806198 seconds
      [2011-05-15 04:50:16,516] - [cluster_helper] [47866199832992] - INFO - removed all the nodes from this cluster ?
      [2011-05-15 04:50:39,412] - [bucket_helper] [47866199832992] - INFO - deleting existing buckets on [ip:10.1.1.152 ssh_username:root, ip:10.1.1.163 ssh_username:root, ip:10.1.1.162 ssh_username:root, ip:10.1.1.151 ssh_username:root]
      [2011-05-15 04:50:41,033] - [bucket_helper] [47866199832992] - INFO - sleeping for 10 seconds because we want to

      ======================================================================
      FAIL: test_replication_2_replica_50_percent (management.replication.replicationtests.ReplicationTests)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
      File "pytests/management/replication/replicationtests.py", line 76, in test_replication_2_replica_50_percent
      self._test_body(fill_ram_percentage=50, number_of_replicas=2)
      File "pytests/management/replication/replicationtests.py", line 283, in _test_body
      msg="replication was completed but sum(curr_items) dont match the curr_items_total")
      AssertionError: replication was completed but sum(curr_items) dont match the curr_items_total

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            chiyoung Chiyoung Seo (Inactive)
            farshid Farshid Ghods (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty