Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-14184

Stat issue: num_items for replica vbuckets when front end load with deletions

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Major
    • 4.0.0
    • 4.0.0
    • couchbase-bucket
    • Security Level: Public
    • None
    • Untriaged
    • Centos 64-bit
    • Yes
    • Mar 9 - Mar 27

    Description

      Build: 4.0.0-1650

      test:
      python27 testrunner.py -i /tmp/xdcr.ini -t xdcr.uniXDCR.unidirectional.load_with_async_ops,items=100000,rdirection=unidirection,ctopology=chain,update=C1,delete=C1,sasl_buckets=1,replication_type=xmem

      1. C1 [.24,.25] --> C2 [.26,.27]
      2. Load 100000 keys on C1
      3. Update and delete 30% keys on C1.

      C1:

      2015-03-27 13:31:08 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_active_curr_items 70000 == 70000 expected on '172.23.106.24:8091''172.23.106.25:8091',default bucket

      2015-03-27 13:31:09 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_active_curr_items 70000 == 70000 expected on '172.23.106.24:8091''172.23.106.25:8091',sasl_bucket_1 bucket

      2015-03-27 13:31:10 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_replica_curr_items 70000 == 70000 expected on '172.23.106.24:8091''172.23.106.25:8091',default bucket

      2015-03-27 13:31:11 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_replica_curr_items 70000 == 70000 expected on '172.23.106.24:8091''172.23.106.25:8091',sasl_bucket_1 bucket

      C2:

      Though this is an XDCR test, I observed data loss during the intra-cluster replication:
      2015-03-27 13:31:12 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_active_curr_items 70000 == 70000 expected on '172.23.106.26:8091''172.23.106.27:8091',sasl_bucket_1 bucket

      2015-03-27 13:31:12 | INFO | MainProcess | Cluster_Thread | [task.check] Saw vb_active_curr_items 70000 == 70000 expected on '172.23.106.26:8091''172.23.106.27:8091',default bucket

      2015-03-27 13:34:11 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 70370 == 70000 expected on '172.23.106.26:8091''172.23.106.27:8091', sasl_bucket_1 bucket

      2015-03-27 13:34:11 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 70008 == 70000 expected on '172.23.106.26:8091''172.23.106.27:8091', default bucket

      /opt/couchbase/bin/cbvdiff -b default 172.23.106.26:11210,172.23.106.27:11210

      2015-03-27 13:34:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
      2015-03-27 13:34:17 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 15: active count 80 != 81 replica count
      2015-03-27 13:34:17 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:17 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 23: active count 68 != 69 replica count
      2015-03-27 13:34:17 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:17 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 27: active count 74 != 75 replica count
      2015-03-27 13:34:17 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:17 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 55: active count 73 != 74 replica count
      2015-03-27 13:34:17 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:17 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 79: active count 72 != 73 replica count
      2015-03-27 13:34:17 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:17 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 168: active count 81 != 82 replica count
      2015-03-27 13:34:17 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:17 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 473: active count 74 != 75 replica count
      2015-03-27 13:34:17 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:17 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 478: active count 77 != 78 replica count
      2015-03-27 13:34:17 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:17 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Active item count = 70000

      running command.raw on 172.23.106.26: /opt/couchbase/bin/cbvdiff -b sasl_bucket_1 -p password 172.23.106.26:11210,172.23.106.27:11210
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 0: active count 70 != 72 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 4: active count 59 != 60 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 12: active count 65 != 66 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 15: active count 80 != 81 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 16: active count 64 != 65 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 20: active count 79 != 81 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 23: active count 68 != 70 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 24: active count 60 != 61 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 36: active count 73 != 74 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 44: active count 74 != 76 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 56: active count 73 != 74 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 64: active count 79 != 81 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 72: active count 70 != 72 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 80: active count 62 != 63 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 84: active count 71 != 73 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 96: active count 65 != 66 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 104: active count 60 != 61 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 108: active count 75 != 76 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 112: active count 75 != 76 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 116: active count 57 != 58 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 119: active count 75 != 76 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 120: active count 72 != 73 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 123: active count 61 != 62 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 132: active count 73 != 75 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 135: active count 62 != 63 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 136: active count 65 != 66 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 139: active count 72 != 75 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 140: active count 74 != 75 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 143: active count 66 != 67 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 144: active count 76 != 78 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 147: active count 66 != 67 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 150: active count 72 != 73 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 151: active count 77 != 78 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 152: active count 71 != 72 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 155: active count 59 != 60 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 159: active count 71 != 74 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 162: active count 61 != 62 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 163: active count 63 != 64 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 167: active count 72 != 75 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 168: active count 81 != 82 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 171: active count 65 != 66 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 175: active count 77 != 78 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 179: active count 77 != 79 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 181: active count 76 != 77 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 183: active count 63 != 64 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 186: active count 80 != 83 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 187: active count 75 != 76 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 190: active count 64 != 65 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 194: active count 74 != 76 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 197: active count 77 != 78 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 199: active count 58 != 59 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 203: active count 72 != 73 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 206: active count 68 != 70 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 207: active count 62 != 63 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 211: active count 66 != 67 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 214: active count 80 != 82 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 218: active count 60 != 61 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 223: active count 79 != 81 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 230: active count 70 != 73 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 231: active count 74 != 77 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 234: active count 66 != 67 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 235: active count 59 != 60 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 238: active count 73 != 76 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 239: active count 75 != 76 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 242: active count 75 != 76 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 243: active count 71 != 74 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 246: active count 68 != 69 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 247: active count 60 != 61 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 250: active count 74 != 78 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 254: active count 56 != 57 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 255: active count 69 != 70 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 257: active count 80 != 81 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 258: active count 67 != 68 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 259: active count 61 != 62 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 261: active count 58 != 59 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 262: active count 74 != 77 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 266: active count 57 != 58 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 267: active count 64 != 65 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 270: active count 72 != 73 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 271: active count 79 != 82 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 274: active count 77 != 78 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 279: active count 65 != 66 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 281: active count 64 != 65 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 282: active count 76 != 78 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 283: active count 70 != 72 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 285: active count 78 != 79 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 286: active count 66 != 68 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 287: active count 59 != 60 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 290: active count 72 != 74 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 291: active count 73 != 75 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 293: active count 75 != 77 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 294: active count 66 != 67 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 298: active count 73 != 76 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 301: active count 76 != 78 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 302: active count 60 != 61 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 305: active count 72 != 74 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 306: active count 62 != 63 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 309: active count 59 != 60 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 310: active count 70 != 72 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 311: active count 74 != 76 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 314: active count 62 != 63 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 315: active count 64 != 65 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 317: active count 69 != 70 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 318: active count 79 != 84 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 319: active count 73 != 75 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 321: active count 71 != 73 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 325: active count 61 != 62 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 326: active count 73 != 76 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 327: active count 74 != 75 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 329: active count 73 != 75 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 330: active count 65 != 66 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 331: active count 62 != 63 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 334: active count 74 != 75 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 335: active count 71 != 73 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 337: active count 66 != 67 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 338: active count 76 != 81 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 339: active count 70 != 72 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 341: active count 77 != 78 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 342: active count 66 != 67 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 343: active count 64 != 65 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 345: active count 58 != 59 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 346: active count 71 != 73 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 347: active count 79 != 80 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 349: active count 72 != 74 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 350: active count 63 != 64 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 353: active count 64 != 65 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 354: active count 75 != 77 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 355: active count 72 != 74 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 357: active count 72 != 75 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 358: active count 59 != 60 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 362: active count 81 != 83 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 363: active count 75 != 76 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 366: active count 65 != 66 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 367: active count 57 != 58 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 369: active count 78 != 79 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 370: active count 66 != 68 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 371: active count 60 != 61 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 374: active count 71 != 72 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 375: active count 75 != 78 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 377: active count 75 != 76 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 378: active count 58 != 59 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 379: active count 65 != 66 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 381: active count 61 != 63 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 382: active count 72 != 74 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 385: active count 70 != 72 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 386: active count 78 != 80 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 387: active count 71 != 72 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 389: active count 68 != 71 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 391: active count 63 != 64 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 393: active count 58 != 59 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 394: active count 70 != 71 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 395: active count 76 != 80 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 397: active count 72 != 75 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 398: active count 64 != 66 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 399: active count 66 != 67 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 401: active count 73 != 75 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 402: active count 63 != 64 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 403: active count 65 != 66 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 406: active count 70 != 72 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 407: active count 74 != 77 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 409: active count 77 != 81 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 414: active count 74 != 75 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 415: active count 73 != 76 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 417: active count 80 != 83 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 418: active count 65 != 66 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 419: active count 58 != 59 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 420: active count 61 != 62 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 421: active count 64 != 65 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 423: active count 72 != 74 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 425: active count 71 != 72 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 429: active count 65 != 66 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 430: active count 75 != 78 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 431: active count 71 != 72 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 433: active count 58 != 59 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 435: active count 81 != 84 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 437: active count 75 != 77 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 438: active count 58 != 59 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 441: active count 61 != 62 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 442: active count 71 != 73 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 443: active count 75 != 76 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 444: active count 73 != 74 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 445: active count 78 != 79 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 447: active count 59 != 60 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 450: active count 69 != 71 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 451: active count 76 != 77 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 453: active count 73 != 74 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 454: active count 60 != 61 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 455: active count 66 != 67 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 459: active count 77 != 80 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 461: active count 80 != 83 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 462: active count 65 != 67 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 465: active count 76 != 77 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 467: active count 57 != 58 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 469: active count 68 != 69 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 470: active count 80 != 82 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 471: active count 72 != 73 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 473: active count 74 != 78 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 474: active count 60 != 61 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 477: active count 60 != 61 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 478: active count 77 != 79 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 479: active count 74 != 77 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 481: active count 74 != 79 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 482: active count 64 != 66 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 483: active count 62 != 63 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 485: active count 56 != 57 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 486: active count 73 != 74 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 487: active count 79 != 81 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 488: active count 71 != 72 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 489: active count 75 != 76 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 491: active count 62 != 63 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 492: active count 60 != 61 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 493: active count 68 != 69 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 494: active count 73 != 75 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 495: active count 70 != 72 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 497: active count 66 != 67 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 499: active count 73 != 77 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 500: active count 75 != 76 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 501: active count 73 != 76 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 503: active count 60 != 61 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 505: active count 61 != 62 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 506: active count 72 != 73 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 507: active count 72 != 73 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 508: active count 74 != 77 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 509: active count 70 != 73 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 510: active count 60 != 61 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 511: active count 66 != 67 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 699: active count 73 != 74 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 791: active count 64 != 65 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 943: active count 81 != 82 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 957: active count 80 != 82 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 962: active count 77 != 78 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 965: active count 74 != 75 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 969: active count 68 != 69 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 977: active count 80 != 81 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] VBucket 1001: active count 73 != 74 replica count
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output]
      2015-03-27 13:34:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Active item count = 70000

      The Jenkins run is here http://qa.sc.couchbase.com/job/erics-test-area/4/console.
      This problem is consistently reproducible.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            abhinav Abhi Dangeti
            ericcooper Eric Cooper (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty