Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-14918

GoXDCR: No replication for more than 20 mins after target node is warmed up. "Failed to resend document" errors.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 4.0.0
    • 4.0.0
    • XDCR
    • Security Level: Public
    • centOS 6.x

    Description

      Build


      4.0.0-2093

      Testcase
      --------
      1. C1 [.38,.42] --> C2 [.205,.206]
      2. 2 buckets
      3. warmup .206
      4. perform updates and deletes on C1.

       
      ./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,GROUP=CHAIN -t xdcr.uniXDCR.unidirectional.load_with_async_ops_with_warmup,items=100000,rdirection=unidirection,ctopology=chain,standard_buckets=1,update=C1,delete=C1,warm=C2,GROUP=CHAIN;P1
       
      Test Input params:
      {'logs_folder': '/data/workspace/cen006-p0-xxdcr-vset08-00-goxdcr-unixdcr/logs/testrunner-15-May-11_10-55-15/test_6', 'GROUP': 'CHAIN', 'items': '100000', 'warm': 'C2', 'standard_buckets': '1', 'update': 'C1', 'conf_file': 'conf/py-xdcr-unidirectional.conf', 'num_nodes': 6, 'cluster_name': 'INI_FILE', 'ctopology': 'chain', 'rdirection': 'unidirection', 'stop-on-failure': 'False', 'ini': 'INI_FILE.ini', 'case_number': 6, 'get-logs': 'False', 'get-cbcollect-info': 'True', 'spec': 'py-xdcr-unidirectional', 'delete': 'C1'}
      [2015-05-11 12:27:59,699] - [remote_util:158] INFO - connecting to 172.23.106.38 with username : root password : couchbase ssh_key: 
      [2015-05-11 12:27:59,930] - [remote_util:191] INFO - Connected to 172.23.106.38
      [2015-05-11 12:28:00,107] - [remote_util:1896] INFO - running command.raw on 172.23.106.38: cat /proc/cpuinfo
      [2015-05-11 12:28:00,192] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:00,193] - [remote_util:1896] INFO - running command.raw on 172.23.106.38: df -Th
      [2015-05-11 12:28:00,277] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:00,278] - [remote_util:1896] INFO - running command.raw on 172.23.106.38: cat /proc/meminfo
      [2015-05-11 12:28:00,360] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:00,361] - [remote_util:1896] INFO - running command.raw on 172.23.106.38: hostname
      [2015-05-11 12:28:00,475] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:00,476] - [remote_util:1896] INFO - running command.raw on 172.23.106.38: hostname -d
      [2015-05-11 12:28:00,558] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:00,675] - [xdcrnewbasetests:380] INFO - 0
      [2015-05-11 12:28:00,678] - [remote_util:158] INFO - connecting to 172.23.106.42 with username : root password : couchbase ssh_key: 
      [2015-05-11 12:28:00,846] - [remote_util:191] INFO - Connected to 172.23.106.42
      [2015-05-11 12:28:01,020] - [remote_util:1896] INFO - running command.raw on 172.23.106.42: cat /proc/cpuinfo
      [2015-05-11 12:28:01,103] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:01,104] - [remote_util:1896] INFO - running command.raw on 172.23.106.42: df -Th
      [2015-05-11 12:28:01,187] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:01,187] - [remote_util:1896] INFO - running command.raw on 172.23.106.42: cat /proc/meminfo
      [2015-05-11 12:28:01,244] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:01,245] - [remote_util:1896] INFO - running command.raw on 172.23.106.42: hostname
      [2015-05-11 12:28:01,303] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:01,304] - [remote_util:1896] INFO - running command.raw on 172.23.106.42: hostname -d
      [2015-05-11 12:28:01,386] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:01,434] - [xdcrnewbasetests:380] INFO - 0
      [2015-05-11 12:28:01,436] - [remote_util:158] INFO - connecting to 172.23.107.205 with username : root password : couchbase ssh_key: 
      [2015-05-11 12:28:01,611] - [remote_util:191] INFO - Connected to 172.23.107.205
      [2015-05-11 12:28:01,783] - [remote_util:1896] INFO - running command.raw on 172.23.107.205: cat /proc/cpuinfo
      [2015-05-11 12:28:01,869] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:01,869] - [remote_util:1896] INFO - running command.raw on 172.23.107.205: df -Th
      [2015-05-11 12:28:01,953] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:01,954] - [remote_util:1896] INFO - running command.raw on 172.23.107.205: cat /proc/meminfo
      [2015-05-11 12:28:02,038] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:02,039] - [remote_util:1896] INFO - running command.raw on 172.23.107.205: hostname
      [2015-05-11 12:28:02,123] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:02,123] - [remote_util:1896] INFO - running command.raw on 172.23.107.205: hostname -d
      [2015-05-11 12:28:02,182] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:02,215] - [xdcrnewbasetests:380] INFO - 0
      [2015-05-11 12:28:02,217] - [remote_util:158] INFO - connecting to 172.23.107.206 with username : root password : couchbase ssh_key: 
      [2015-05-11 12:28:02,385] - [remote_util:191] INFO - Connected to 172.23.107.206
      [2015-05-11 12:28:02,580] - [remote_util:1896] INFO - running command.raw on 172.23.107.206: cat /proc/cpuinfo
      [2015-05-11 12:28:02,663] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:02,664] - [remote_util:1896] INFO - running command.raw on 172.23.107.206: df -Th
      [2015-05-11 12:28:02,745] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:02,746] - [remote_util:1896] INFO - running command.raw on 172.23.107.206: cat /proc/meminfo
      [2015-05-11 12:28:02,826] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:02,827] - [remote_util:1896] INFO - running command.raw on 172.23.107.206: hostname
      [2015-05-11 12:28:02,908] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:02,909] - [remote_util:1896] INFO - running command.raw on 172.23.107.206: hostname -d
      [2015-05-11 12:28:02,992] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:03,031] - [xdcrnewbasetests:380] INFO - 1
      [2015-05-11 12:28:03,146] - [remote_util:158] INFO - connecting to 172.23.106.44 with username : root password : couchbase ssh_key: 
      [2015-05-11 12:28:03,313] - [remote_util:191] INFO - Connected to 172.23.106.44
      [2015-05-11 12:28:03,479] - [remote_util:1896] INFO - running command.raw on 172.23.106.44: cat /proc/cpuinfo
      [2015-05-11 12:28:03,563] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:03,564] - [remote_util:1896] INFO - running command.raw on 172.23.106.44: df -Th
      [2015-05-11 12:28:03,645] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:03,646] - [remote_util:1896] INFO - running command.raw on 172.23.106.44: cat /proc/meminfo
      [2015-05-11 12:28:03,704] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:03,705] - [remote_util:1896] INFO - running command.raw on 172.23.106.44: hostname
      [2015-05-11 12:28:03,786] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:03,787] - [remote_util:1896] INFO - running command.raw on 172.23.106.44: hostname -d
      [2015-05-11 12:28:03,870] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:03,966] - [xdcrnewbasetests:380] INFO - 0
      [2015-05-11 12:28:04,081] - [remote_util:158] INFO - connecting to 172.23.107.207 with username : root password : couchbase ssh_key: 
      [2015-05-11 12:28:04,248] - [remote_util:191] INFO - Connected to 172.23.107.207
      [2015-05-11 12:28:04,421] - [remote_util:1896] INFO - running command.raw on 172.23.107.207: cat /proc/cpuinfo
      [2015-05-11 12:28:04,502] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:04,503] - [remote_util:1896] INFO - running command.raw on 172.23.107.207: df -Th
      [2015-05-11 12:28:04,586] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:04,587] - [remote_util:1896] INFO - running command.raw on 172.23.107.207: cat /proc/meminfo
      [2015-05-11 12:28:04,668] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:04,669] - [remote_util:1896] INFO - running command.raw on 172.23.107.207: hostname
      [2015-05-11 12:28:04,749] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:04,750] - [remote_util:1896] INFO - running command.raw on 172.23.107.207: hostname -d
      [2015-05-11 12:28:04,811] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:28:04,840] - [xdcrnewbasetests:380] INFO - 0
      [2015-05-11 12:28:04,842] - [xdcrnewbasetests:2409] INFO - {'172.23.106.38': {'panic:': 0}, '172.23.107.206': {'panic:': 1}, '172.23.106.44': {'panic:': 0}, '172.23.107.207': {'panic:': 0}, '172.23.106.42': {'panic:': 0}, '172.23.107.205': {'panic:': 0}}
      [2015-05-11 12:28:04,843] - [xdcrnewbasetests:2239] INFO - ==== XDCRNewbasetests setup is started for test #6 load_with_async_ops_with_warmup ====
      [2015-05-11 12:28:04,845] - [xdcrnewbasetests:1054] INFO - removing xdcr/nodes settings
      [2015-05-11 12:28:04,879] - [xdcrnewbasetests:1064] INFO - cleanup [ip:172.23.106.38 port:8091 ssh_username:root, ip:172.23.106.42 port:8091 ssh_username:root]
      [2015-05-11 12:28:04,888] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.106.38
      [2015-05-11 12:28:04,912] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.106.38:8091
      [2015-05-11 12:28:04,917] - [cluster_helper:80] INFO - ns_server @ 172.23.106.38:8091 is running
      [2015-05-11 12:28:04,927] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.106.42
      [2015-05-11 12:28:04,957] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.106.42:8091
      [2015-05-11 12:28:04,963] - [cluster_helper:80] INFO - ns_server @ 172.23.106.42:8091 is running
      [2015-05-11 12:28:04,963] - [xdcrnewbasetests:1054] INFO - removing xdcr/nodes settings
      [2015-05-11 12:28:04,995] - [xdcrnewbasetests:1064] INFO - cleanup [ip:172.23.107.205 port:8091 ssh_username:root, ip:172.23.107.206 port:8091 ssh_username:root]
      [2015-05-11 12:28:05,003] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.107.205
      [2015-05-11 12:28:05,028] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.205:8091
      [2015-05-11 12:28:05,033] - [cluster_helper:80] INFO - ns_server @ 172.23.107.205:8091 is running
      [2015-05-11 12:28:05,045] - [bucket_helper:137] INFO - deleting existing buckets [] on 172.23.107.206
      [2015-05-11 12:28:05,070] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.107.206:8091
      [2015-05-11 12:28:05,074] - [cluster_helper:80] INFO - ns_server @ 172.23.107.206:8091 is running
      [2015-05-11 12:28:05,075] - [xdcrnewbasetests:2419] INFO - Initializing all clusters...
      [2015-05-11 12:28:05,086] - [rest_client:1182] INFO - /diag/eval status on 172.23.106.38:8091: True content: [4,0] command: cluster_compat_mode:get_compat_version().
      [2015-05-11 12:28:05,860] - [rest_client:748] INFO - settings/web params on 172.23.106.38:8091:username=Administrator&password=password&port=8091
      [2015-05-11 12:28:05,877] - [rest_client:785] INFO - pools/default params : memoryQuota=2031
      [2015-05-11 12:28:05,896] - [rest_client:748] INFO - settings/web params on 172.23.106.42:8091:username=Administrator&password=password&port=8091
      [2015-05-11 12:28:05,924] - [rest_client:785] INFO - pools/default params : memoryQuota=2031
      [2015-05-11 12:28:06,932] - [task:369] INFO - adding node 172.23.106.42:8091 to cluster
      [2015-05-11 12:28:06,932] - [rest_client:948] INFO - adding remote node @172.23.106.42:8091 to this cluster @172.23.106.38:8091
      [2015-05-11 12:28:09,699] - [rest_client:1166] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.106.38%2Cns_1%40172.23.106.42
      [2015-05-11 12:28:09,706] - [rest_client:1170] INFO - rebalance operation started
      [2015-05-11 12:28:09,713] - [rest_client:1288] INFO - rebalance percentage : 0.00 %
      [2015-05-11 12:28:19,732] - [task:485] INFO - rebalancing was completed with progress: 100% in 10.0257878304 sec
      [2015-05-11 12:28:19,744] - [rest_client:1182] INFO - /diag/eval status on 172.23.107.205:8091: True content: [4,0] command: cluster_compat_mode:get_compat_version().
      [2015-05-11 12:28:19,877] - [rest_client:748] INFO - settings/web params on 172.23.107.205:8091:username=Administrator&password=password&port=8091
      [2015-05-11 12:28:19,896] - [rest_client:785] INFO - pools/default params : memoryQuota=2031
      [2015-05-11 12:28:19,916] - [rest_client:748] INFO - settings/web params on 172.23.107.206:8091:username=Administrator&password=password&port=8091
      [2015-05-11 12:28:19,935] - [rest_client:785] INFO - pools/default params : memoryQuota=2031
      [2015-05-11 12:28:20,941] - [task:369] INFO - adding node 172.23.107.206:8091 to cluster
      [2015-05-11 12:28:20,942] - [rest_client:948] INFO - adding remote node @172.23.107.206:8091 to this cluster @172.23.107.205:8091
      [2015-05-11 12:28:23,959] - [rest_client:1166] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.107.206%2Cns_1%40172.23.107.205
      [2015-05-11 12:28:23,965] - [rest_client:1170] INFO - rebalance operation started
      [2015-05-11 12:28:23,974] - [rest_client:1288] INFO - rebalance percentage : 0.00 %
      [2015-05-11 12:28:33,993] - [task:485] INFO - rebalancing was completed with progress: 100% in 10.0278241634 sec
      [2015-05-11 12:28:34,771] - [rest_client:1689] INFO - http://172.23.106.38:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=765&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None
      [2015-05-11 12:28:34,782] - [rest_client:1711] INFO - 0.01 seconds to create bucket default
      [2015-05-11 12:28:34,783] - [bucket_helper:305] INFO - waiting for memcached bucket : default in 172.23.106.38 to accept set ops
      [2015-05-11 12:28:36,124] - [data_helper:295] INFO - creating direct client 172.23.106.38:11210 default
      [2015-05-11 12:28:36,212] - [data_helper:295] INFO - creating direct client 172.23.106.42:11210 default
      [2015-05-11 12:28:36,343] - [data_helper:295] INFO - creating direct client 172.23.106.38:11210 default
      [2015-05-11 12:28:36,429] - [data_helper:295] INFO - creating direct client 172.23.106.42:11210 default
      [2015-05-11 12:28:36,994] - [task:253] INFO - bucket 'default' was created with per node RAM quota: 765
      [2015-05-11 12:28:38,090] - [rest_client:1689] INFO - http://172.23.106.38:8091/pools/default/buckets with param: proxyPort=11217&bucketType=membase&authType=none&replicaIndex=1&name=standard_bucket_1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&replicaNumber=1&ramQuotaMB=765
      [2015-05-11 12:28:38,103] - [rest_client:1711] INFO - 0.01 seconds to create bucket standard_bucket_1
      [2015-05-11 12:28:38,103] - [bucket_helper:305] INFO - waiting for memcached bucket : standard_bucket_1 in 172.23.106.38 to accept set ops
      [2015-05-11 12:28:39,340] - [data_helper:295] INFO - creating direct client 172.23.106.38:11210 standard_bucket_1
      [2015-05-11 12:28:39,575] - [data_helper:295] INFO - creating direct client 172.23.106.42:11210 standard_bucket_1
      [2015-05-11 12:28:39,718] - [data_helper:295] INFO - creating direct client 172.23.106.38:11210 standard_bucket_1
      [2015-05-11 12:28:39,839] - [data_helper:295] INFO - creating direct client 172.23.106.42:11210 standard_bucket_1
      [2015-05-11 12:28:40,424] - [task:253] INFO - bucket 'standard_bucket_1' was created with per node RAM quota: 765
      [2015-05-11 12:28:41,060] - [rest_client:1689] INFO - http://172.23.107.205:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=765&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None
      [2015-05-11 12:28:41,071] - [rest_client:1711] INFO - 0.01 seconds to create bucket default
      [2015-05-11 12:28:41,071] - [bucket_helper:305] INFO - waiting for memcached bucket : default in 172.23.107.205 to accept set ops
      [2015-05-11 12:28:42,356] - [data_helper:295] INFO - creating direct client 172.23.107.205:11210 default
      [2015-05-11 12:28:42,439] - [data_helper:295] INFO - creating direct client 172.23.107.206:11210 default
      [2015-05-11 12:28:42,715] - [data_helper:295] INFO - creating direct client 172.23.107.205:11210 default
      [2015-05-11 12:28:42,822] - [data_helper:295] INFO - creating direct client 172.23.107.206:11210 default
      [2015-05-11 12:28:43,384] - [task:253] INFO - bucket 'default' was created with per node RAM quota: 765
      [2015-05-11 12:28:44,409] - [rest_client:1689] INFO - http://172.23.107.205:8091/pools/default/buckets with param: proxyPort=11217&bucketType=membase&authType=none&replicaIndex=1&name=standard_bucket_1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&replicaNumber=1&ramQuotaMB=765
      [2015-05-11 12:28:44,420] - [rest_client:1711] INFO - 0.01 seconds to create bucket standard_bucket_1
      [2015-05-11 12:28:44,420] - [bucket_helper:305] INFO - waiting for memcached bucket : standard_bucket_1 in 172.23.107.205 to accept set ops
      [2015-05-11 12:28:45,598] - [data_helper:295] INFO - creating direct client 172.23.107.205:11210 standard_bucket_1
      [2015-05-11 12:28:45,692] - [data_helper:295] INFO - creating direct client 172.23.107.206:11210 standard_bucket_1
      [2015-05-11 12:28:45,820] - [data_helper:295] INFO - creating direct client 172.23.107.205:11210 standard_bucket_1
      [2015-05-11 12:28:45,915] - [data_helper:295] INFO - creating direct client 172.23.107.206:11210 standard_bucket_1
      [2015-05-11 12:28:46,621] - [task:253] INFO - bucket 'standard_bucket_1' was created with per node RAM quota: 765
      [2015-05-11 12:28:46,635] - [rest_client:1912] INFO - Update internal setting xdcrCheckpointInterval=60
      [2015-05-11 12:28:46,651] - [rest_client:1912] INFO - Update internal setting xdcrCheckpointInterval=60
      [2015-05-11 12:28:46,652] - [xdcrnewbasetests:2245] INFO - ==== XDCRNewbasetests setup is finished for test #6 load_with_async_ops_with_warmup ====
      [2015-05-11 12:28:46,658] - [rest_client:838] INFO - adding remote cluster hostname:172.23.107.205:8091 with username:password Administrator:password name:remote_cluster_C1-C2 to source node: 172.23.106.38:8091
      [2015-05-11 12:28:46,723] - [rest_client:885] INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C1-C2 with settings {}
      [2015-05-11 12:28:46,810] - [rest_client:898] INFO - Replication created with id: 449fcf5248457209d6f8eb8f800fe6e7/default/default
      [2015-05-11 12:28:46,819] - [rest_client:885] INFO - starting continuous replication type:xmem from standard_bucket_1 to standard_bucket_1 in the remote cluster remote_cluster_C1-C2 with settings {}
      [2015-05-11 12:28:46,911] - [rest_client:898] INFO - Replication created with id: 449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1
      [2015-05-11 12:28:46,975] - [data_helper:295] INFO - creating direct client 172.23.106.38:11210 default
      [2015-05-11 12:28:47,053] - [data_helper:295] INFO - creating direct client 172.23.106.42:11210 default
      [2015-05-11 12:28:47,174] - [data_helper:295] INFO - creating direct client 172.23.106.38:11210 standard_bucket_1
      [2015-05-11 12:28:47,250] - [data_helper:295] INFO - creating direct client 172.23.106.42:11210 standard_bucket_1
      [2015-05-11 12:31:39,559] - [xdcrnewbasetests:2943] INFO - sleep for 10 secs.  ...
      [2015-05-11 12:31:49,570] - [remote_util:158] INFO - connecting to 172.23.107.206 with username : root password : couchbase ssh_key: 
      [2015-05-11 12:31:49,739] - [remote_util:191] INFO - Connected to 172.23.107.206
      [2015-05-11 12:31:49,946] - [remote_util:1896] INFO - running command.raw on 172.23.107.206: cat /proc/cpuinfo
      [2015-05-11 12:31:50,031] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:31:50,033] - [remote_util:1896] INFO - running command.raw on 172.23.107.206: df -Th
      [2015-05-11 12:31:50,117] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:31:50,118] - [remote_util:1896] INFO - running command.raw on 172.23.107.206: cat /proc/meminfo
      [2015-05-11 12:31:50,212] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:31:50,212] - [remote_util:1896] INFO - running command.raw on 172.23.107.206: hostname
      [2015-05-11 12:31:50,273] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:31:50,274] - [remote_util:1896] INFO - running command.raw on 172.23.107.206: hostname -d
      [2015-05-11 12:31:50,359] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:31:50,361] - [remote_util:1896] INFO - running command.raw on 172.23.107.206: /etc/init.d/couchbase-server stop
      [2015-05-11 12:32:03,398] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:32:03,399] - [remote_util:1835] INFO - Stopping couchbase-server
      [2015-05-11 12:32:03,399] - [remote_util:1835] INFO - [  OK  ]
      [2015-05-11 12:32:03,399] - [remote_util:1835] INFO - 
      [2015-05-11 12:32:08,405] - [remote_util:1896] INFO - running command.raw on 172.23.107.206: /etc/init.d/couchbase-server start
      [2015-05-11 12:32:09,733] - [remote_util:1933] INFO - command executed successfully
      [2015-05-11 12:32:09,734] - [remote_util:1835] INFO - Starting couchbase-server
      [2015-05-11 12:32:09,734] - [remote_util:1835] INFO - 
      [2015-05-11 12:32:09,848] - [xdcrnewbasetests:2943] INFO - sleep for 60 secs.  ...
      [2015-05-11 12:33:09,868] - [xdcrnewbasetests:2703] INFO - Updating keys @ C1
      [2015-05-11 12:33:09,869] - [xdcrnewbasetests:1499] INFO - At bucket 'default' @ C1: operation: update, key range 0 - 30000
      [2015-05-11 12:33:09,935] - [data_helper:295] INFO - creating direct client 172.23.106.38:11210 default
      [2015-05-11 12:33:10,243] - [data_helper:295] INFO - creating direct client 172.23.106.42:11210 default
      [2015-05-11 12:33:10,321] - [xdcrnewbasetests:1499] INFO - At bucket 'standard_bucket_1' @ C1: operation: update, key range 0 - 30000
      [2015-05-11 12:33:10,387] - [data_helper:295] INFO - creating direct client 172.23.106.38:11210 standard_bucket_1
      [2015-05-11 12:33:10,490] - [data_helper:295] INFO - creating direct client 172.23.106.42:11210 standard_bucket_1
      [2015-05-11 12:35:08,373] - [xdcrnewbasetests:2710] INFO - Batched updates loaded to cluster(s)
      [2015-05-11 12:35:08,375] - [xdcrnewbasetests:2716] INFO - Deleting keys @ C1
      [2015-05-11 12:35:08,375] - [xdcrnewbasetests:1499] INFO - At bucket 'default' @ C1: operation: delete, key range 70000 - 100000
      [2015-05-11 12:35:08,468] - [data_helper:295] INFO - creating direct client 172.23.106.38:11210 default
      [2015-05-11 12:35:08,578] - [data_helper:295] INFO - creating direct client 172.23.106.42:11210 default
      [2015-05-11 12:35:08,662] - [xdcrnewbasetests:1499] INFO - At bucket 'standard_bucket_1' @ C1: operation: delete, key range 70000 - 100000
      [2015-05-11 12:35:08,763] - [data_helper:295] INFO - creating direct client 172.23.106.38:11210 standard_bucket_1
      [2015-05-11 12:35:08,857] - [data_helper:295] INFO - creating direct client 172.23.106.42:11210 standard_bucket_1
      [2015-05-11 12:36:03,527] - [xdcrnewbasetests:2723] INFO - Batched deletes sent to cluster(s)
      [2015-05-11 12:36:03,528] - [xdcrnewbasetests:2943] INFO - sleep for 30 secs.  ...
      [2015-05-11 12:36:33,576] - [data_helper:295] INFO - creating direct client 172.23.107.206:11210 default
      [2015-05-11 12:36:33,678] - [xdcrnewbasetests:310] INFO - Warmed up: 100000 items on default on ip:172.23.107.206 port:8091 ssh_username:root
      [2015-05-11 12:36:43,698] - [xdcrnewbasetests:2881] INFO - Merging keys for replication Replication C1:default -> C2:default
      [2015-05-11 12:36:43,882] - [xdcrnewbasetests:2820] INFO - src_kvstore has 70000 valid and 30000 deleted keys
      [2015-05-11 12:36:43,882] - [xdcrnewbasetests:2822] INFO - dest kvstore has 0 valid and 0 deleted keys
      [2015-05-11 12:36:45,223] - [xdcrnewbasetests:2871] INFO - After merging: destination bucket's kv_store now has 70000 valid keys and 0 deleted keys
      [2015-05-11 12:36:45,228] - [xdcrnewbasetests:2881] INFO - Merging keys for replication Replication C1:standard_bucket_1 -> C2:standard_bucket_1
      [2015-05-11 12:36:45,399] - [xdcrnewbasetests:2820] INFO - src_kvstore has 70000 valid and 30000 deleted keys
      [2015-05-11 12:36:45,399] - [xdcrnewbasetests:2822] INFO - dest kvstore has 0 valid and 0 deleted keys
      [2015-05-11 12:36:46,910] - [xdcrnewbasetests:2871] INFO - After merging: destination bucket's kv_store now has 70000 valid keys and 0 deleted keys
      [2015-05-11 12:36:46,953] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.106.38', 'password': 'password', 'port': u'8091'}
      [2015-05-11 12:36:46,953] - [data_helper:293] INFO - creating direct client 172.23.106.38:11210 default
      [2015-05-11 12:36:47,011] - [cluster_helper:345] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.106.38', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default
      setting param: exp_pager_stime 10
      [2015-05-11 12:36:47,012] - [cluster_helper:359] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.106.38', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (2501429715, 0, '')
      [2015-05-11 12:36:47,033] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.106.42', 'password': 'password', 'port': u'8091'}
      [2015-05-11 12:36:47,034] - [data_helper:293] INFO - creating direct client 172.23.106.42:11210 default
      [2015-05-11 12:36:47,084] - [cluster_helper:345] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.106.42', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default
      setting param: exp_pager_stime 10
      [2015-05-11 12:36:47,085] - [cluster_helper:359] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.106.42', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (4236025040, 0, '')
      [2015-05-11 12:36:47,086] - [xdcrnewbasetests:1541] INFO - wait for expiry pager to run on all these nodes
      [2015-05-11 12:36:47,122] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.106.38', 'password': 'password', 'port': u'8091'}
      [2015-05-11 12:36:47,122] - [data_helper:293] INFO - creating direct client 172.23.106.38:11210 standard_bucket_1
      [2015-05-11 12:36:47,352] - [cluster_helper:345] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.106.38', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on standard_bucket_1
      setting param: exp_pager_stime 10
      [2015-05-11 12:36:47,353] - [cluster_helper:359] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.106.38', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (1858969027, 0, '')
      [2015-05-11 12:36:47,379] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.106.42', 'password': 'password', 'port': u'8091'}
      [2015-05-11 12:36:47,379] - [data_helper:293] INFO - creating direct client 172.23.106.42:11210 standard_bucket_1
      [2015-05-11 12:36:47,457] - [cluster_helper:345] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.106.42', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on standard_bucket_1
      setting param: exp_pager_stime 10
      [2015-05-11 12:36:47,458] - [cluster_helper:359] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.106.42', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (2581336305, 0, '')
      [2015-05-11 12:36:47,459] - [xdcrnewbasetests:1541] INFO - wait for expiry pager to run on all these nodes
      [2015-05-11 12:36:57,522] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.205', 'password': 'password', 'port': u'8091'}
      [2015-05-11 12:36:57,523] - [data_helper:293] INFO - creating direct client 172.23.107.205:11210 default
      [2015-05-11 12:36:57,592] - [cluster_helper:345] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.205', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default
      setting param: exp_pager_stime 10
      [2015-05-11 12:36:57,593] - [cluster_helper:359] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.205', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (876606101, 0, '')
      [2015-05-11 12:36:57,620] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.206', 'password': 'password', 'port': u'8091'}
      [2015-05-11 12:36:57,621] - [data_helper:293] INFO - creating direct client 172.23.107.206:11210 default
      [2015-05-11 12:36:57,710] - [cluster_helper:345] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.206', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on default
      setting param: exp_pager_stime 10
      [2015-05-11 12:36:57,711] - [cluster_helper:359] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.206', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (2408870720, 0, '')
      [2015-05-11 12:36:57,711] - [xdcrnewbasetests:1541] INFO - wait for expiry pager to run on all these nodes
      [2015-05-11 12:36:57,758] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.205', 'password': 'password', 'port': u'8091'}
      [2015-05-11 12:36:57,759] - [data_helper:293] INFO - creating direct client 172.23.107.205:11210 standard_bucket_1
      [2015-05-11 12:36:57,825] - [cluster_helper:345] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.205', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on standard_bucket_1
      setting param: exp_pager_stime 10
      [2015-05-11 12:36:57,826] - [cluster_helper:359] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.205', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (3662245328, 0, '')
      [2015-05-11 12:36:57,858] - [data_helper:292] INFO - dict:{'username': 'Administrator', 'ip': u'172.23.107.206', 'password': 'password', 'port': u'8091'}
      [2015-05-11 12:36:57,858] - [data_helper:293] INFO - creating direct client 172.23.107.206:11210 standard_bucket_1
      [2015-05-11 12:36:57,927] - [cluster_helper:345] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.206', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10 on standard_bucket_1
      setting param: exp_pager_stime 10
      [2015-05-11 12:36:57,928] - [cluster_helper:359] INFO - Setting flush param on server {'username': 'Administrator', 'ip': u'172.23.107.206', 'password': 'password', 'port': u'8091'}, exp_pager_stime to 10, result: (4032249625, 0, '')
      [2015-05-11 12:36:57,928] - [xdcrnewbasetests:1541] INFO - wait for expiry pager to run on all these nodes
      [2015-05-11 12:37:08,367] - [data_helper:295] INFO - creating direct client 172.23.106.38:11210 default
      [2015-05-11 12:37:08,449] - [task:537] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.38:8091',default bucket
      [2015-05-11 12:37:08,461] - [data_helper:295] INFO - creating direct client 172.23.106.38:11210 standard_bucket_1
      [2015-05-11 12:37:08,531] - [task:537] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.38:8091',standard_bucket_1 bucket
      [2015-05-11 12:37:08,544] - [data_helper:295] INFO - creating direct client 172.23.106.42:11210 default
      [2015-05-11 12:37:08,622] - [task:537] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.42:8091',default bucket
      [2015-05-11 12:37:08,634] - [data_helper:295] INFO - creating direct client 172.23.106.42:11210 standard_bucket_1
      [2015-05-11 12:37:08,923] - [task:537] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.106.42:8091',standard_bucket_1 bucket
      [2015-05-11 12:37:09,369] - [data_helper:295] INFO - creating direct client 172.23.107.205:11210 default
      [2015-05-11 12:37:09,432] - [task:537] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.205:8091',default bucket
      [2015-05-11 12:37:09,451] - [data_helper:295] INFO - creating direct client 172.23.107.205:11210 standard_bucket_1
      [2015-05-11 12:37:09,520] - [task:537] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.205:8091',standard_bucket_1 bucket
      [2015-05-11 12:37:09,533] - [data_helper:295] INFO - creating direct client 172.23.107.206:11210 default
      [2015-05-11 12:37:09,618] - [task:537] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.206:8091',default bucket
      [2015-05-11 12:37:09,632] - [data_helper:295] INFO - creating direct client 172.23.107.206:11210 standard_bucket_1
      [2015-05-11 12:37:09,723] - [task:537] INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.206:8091',standard_bucket_1 bucket
      [2015-05-11 12:37:09,753] - [xdcrnewbasetests:2128] INFO - Waiting for dcp queue to drain on cluster node: 172.23.106.38
      [2015-05-11 12:37:09,805] - [xdcrnewbasetests:2138] INFO - Current dcp queue size on C1 for default is 0
      [2015-05-11 12:37:09,806] - [xdcrnewbasetests:2128] INFO - Waiting for dcp queue to drain on cluster node: 172.23.107.205
      [2015-05-11 12:37:09,862] - [xdcrnewbasetests:2138] INFO - Current dcp queue size on C2 for default is 0
      [2015-05-11 12:37:09,862] - [xdcrnewbasetests:2158] INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.106.38
      [2015-05-11 12:37:09,895] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:37:09,917] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:37:14,941] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:37:14,965] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:37:19,998] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:37:20,025] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:37:25,051] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:37:25,072] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:37:30,101] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:37:30,122] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:37:35,147] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:37:35,169] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:37:40,192] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:37:40,212] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:37:45,243] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:37:45,269] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:37:50,300] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:37:50,328] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:37:55,355] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:37:55,382] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:38:00,415] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:38:00,439] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:38:05,467] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:38:05,488] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:38:10,518] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:38:10,541] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:38:15,570] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:38:15,598] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:38:20,635] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:38:20,666] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:38:25,694] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:38:25,720] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:38:30,749] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:38:30,779] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:38:35,807] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:38:35,829] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:38:40,856] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:38:40,882] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:38:45,950] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:38:45,994] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:38:51,025] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:38:51,052] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:38:56,089] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:38:56,113] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:39:01,137] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:39:01,156] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:39:06,178] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:39:06,196] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:39:11,221] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:39:11,244] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:39:16,271] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:39:16,301] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:39:21,348] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:39:21,378] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:39:26,402] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:39:26,428] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:39:31,510] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:39:31,722] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:39:36,746] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:39:36,769] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:39:41,800] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:39:41,825] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:39:46,850] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:39:46,873] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:39:51,902] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:39:51,922] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:39:56,951] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:39:56,973] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:40:02,000] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:40:02,030] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:40:07,062] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket default is 16525
      [2015-05-11 12:40:07,084] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C1 for bucket standard_bucket_1 is 19018
      [2015-05-11 12:40:12,090] - [xdcrnewbasetests:2184] ERROR - Timeout occurs while waiting for mutations to be replicated
      [2015-05-11 12:40:12,090] - [xdcrnewbasetests:2158] INFO - Waiting for Outbound mutation to be zero on cluster node: 172.23.107.205
      [2015-05-11 12:40:12,103] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C2 for bucket default is 0
      [2015-05-11 12:40:12,111] - [xdcrnewbasetests:2172] INFO - Current Outbound mutations on cluster node: C2 for bucket standard_bucket_1 is 0
      [2015-05-11 12:40:12,260] - [xdcrnewbasetests:2032] INFO - Saw: vb_active_curr_items 70000 == 70000 expected on C1, default bucket
      [2015-05-11 12:40:12,411] - [xdcrnewbasetests:2032] INFO - Saw: vb_active_curr_items 70000 == 70000 expected on C1, standard_bucket_1 bucket
      [2015-05-11 12:40:12,565] - [xdcrnewbasetests:2066] INFO - Saw: vb_replica_curr_items 70000 == 70000 expected on C1, default bucket
      [2015-05-11 12:40:12,709] - [xdcrnewbasetests:2066] INFO - Saw: vb_replica_curr_items 70000 == 70000 expected on C1, standard_bucket_1 bucket
      [2015-05-11 12:40:12,853] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:40:17,924] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:40:22,991] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:40:28,053] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:40:33,115] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:40:38,170] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:40:43,233] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:40:48,325] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:40:53,376] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:40:58,429] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:41:03,494] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:41:08,550] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:41:13,621] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:41:18,679] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:41:23,740] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:41:28,800] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:41:33,859] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:41:38,911] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:41:43,961] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:41:49,028] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:41:54,083] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:41:59,141] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:42:04,208] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:42:09,282] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:42:14,342] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:42:19,405] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:42:24,462] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:42:29,529] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:42:34,587] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:42:39,659] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:42:44,725] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:42:49,778] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:42:54,873] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:42:59,935] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:43:04,986] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:43:10,053] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:43:15,098] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:43:20,173] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:43:25,242] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:43:30,313] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:43:35,359] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:43:40,409] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:43:45,472] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:43:50,531] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:43:55,589] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:44:00,654] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:44:05,716] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:44:10,775] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:44:15,831] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:44:20,893] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:44:25,957] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:44:31,013] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:44:36,075] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:44:41,129] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:44:46,185] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:44:51,239] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:44:56,316] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:45:01,369] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:45:06,417] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:45:11,473] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:45:16,479] - [xdcrnewbasetests:2025] ERROR - ERROR: Timed-out waiting for active item count to match
      [2015-05-11 12:45:16,612] - [xdcrnewbasetests:2021] WARNING - Not Ready: vb_active_curr_items 77500 == 70000 expected on C2, standard_bucket_1 bucket
      [2015-05-11 12:45:21,617] - [xdcrnewbasetests:2025] ERROR - ERROR: Timed-out waiting for active item count to match
      [2015-05-11 12:45:21,754] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:45:24,823] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:45:27,896] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:45:30,957] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:45:34,010] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:45:37,065] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:45:40,124] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:45:43,176] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:45:46,230] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:45:49,278] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:45:52,328] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:45:55,388] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:45:58,442] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:01,497] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:04,543] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:07,597] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:10,657] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:13,723] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:16,785] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:19,835] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77502 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:22,888] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:25,942] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:29,020] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:32,067] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:35,125] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:38,189] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:41,245] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:44,297] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:47,351] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:50,408] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:53,460] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:56,507] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:46:59,556] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:47:02,607] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:47:05,663] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:47:08,734] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:47:11,795] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:47:14,847] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:47:17,906] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:47:20,956] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:47:24,008] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:47:27,054] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:47:30,109] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:47:33,169] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:47:36,219] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:47:39,267] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:47:42,313] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:47:45,358] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:47:48,402] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:47:51,464] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:47:54,520] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:47:57,573] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:00,629] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:03,677] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:06,729] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:09,778] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:12,822] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:15,871] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:18,925] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:21,980] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:25,030] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:28,091] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:31,150] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:34,206] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:37,264] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:40,311] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:43,362] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:46,415] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:49,469] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:52,532] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:55,583] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:48:58,651] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:01,702] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:04,754] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:07,808] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:10,858] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:13,922] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:16,983] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:20,062] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:23,116] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:26,169] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:29,248] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:32,326] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:35,385] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:38,442] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:41,493] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:44,555] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:47,605] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:50,656] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:53,706] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:56,765] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:49:59,826] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:50:02,896] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:50:05,945] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:50:09,001] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:50:12,052] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:50:15,109] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:50:18,162] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:50:21,234] - [xdcrnewbasetests:2054] WARNING - Not Ready: vb_replica_curr_items 77500 == 70000 expected on C2, default bucket
      [2015-05-11 12:50:24,237] - [xdcrnewbasetests:2058] ERROR - ERROR: Timed-out waiting for replica item count to match
      

      .38 has replicated all its docs but .42 hasn't.

      StatisticsManager 2015-05-11T12:35:53.894-07:00 [INFO] total_doc=80000, docs_processed=60964, changes_left=19036
      StatisticsManager 2015-05-11T12:35:56.297-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:36:03.871-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:36:06.306-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:36:13.859-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:36:16.293-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:36:23.854-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:36:26.296-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:36:33.866-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:36:36.294-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:36:43.876-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:36:46.302-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:36:53.880-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:36:56.291-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:37:03.857-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:37:06.304-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:37:13.861-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:37:16.311-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:37:23.860-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:37:26.299-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:37:33.864-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:37:36.307-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:37:43.867-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:37:46.315-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:37:53.864-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:37:56.301-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:38:03.864-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:38:06.304-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:38:13.869-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:38:16.322-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:38:23.860-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:38:26.297-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:38:33.869-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:38:36.306-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:38:43.852-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:38:46.310-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:38:53.860-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:38:56.296-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:39:03.889-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:39:06.305-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:39:13.869-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:39:16.310-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:39:23.859-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:39:26.311-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:39:33.866-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:39:36.314-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:39:43.854-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:39:46.306-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:39:53.867-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:39:56.304-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:40:03.867-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:40:06.298-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:40:13.874-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:40:16.309-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:40:23.863-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:40:26.298-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:40:33.867-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:40:36.303-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:40:43.864-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:40:46.304-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:40:53.857-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:40:56.293-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:41:03.862-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:41:06.305-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:41:13.883-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:41:16.302-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:41:23.865-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:41:26.302-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:41:33.857-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:41:36.343-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:41:43.867-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:41:46.297-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:41:53.868-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:41:56.307-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:42:03.865-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:42:06.302-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:42:13.863-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:42:16.298-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:42:23.862-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:42:26.312-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:42:33.861-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:42:36.302-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:42:43.866-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:42:46.327-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:42:53.859-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:42:56.296-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:43:03.868-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:43:06.305-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:43:13.867-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:43:16.301-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:43:23.869-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:43:26.304-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:43:33.870-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:43:36.301-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:43:43.854-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:43:46.323-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:43:53.857-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:43:56.296-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:44:03.855-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:44:06.292-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:44:13.868-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:44:16.297-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:44:23.881-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:44:26.290-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:44:33.852-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:44:36.304-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:44:43.868-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:44:46.299-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:44:53.860-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:44:56.291-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:45:03.852-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:45:06.296-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:45:13.862-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:45:16.298-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:45:23.870-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:45:26.292-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:45:33.866-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:45:36.299-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:45:43.876-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:45:46.300-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:45:53.855-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:45:56.313-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:46:03.850-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:46:06.297-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:46:13.868-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:46:16.310-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:46:23.864-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:46:26.300-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:46:33.884-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:46:36.291-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:46:43.855-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:46:46.308-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:46:53.866-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:46:56.299-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:47:03.872-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:47:06.296-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:47:13.853-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:47:16.299-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:47:23.855-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:47:26.294-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:47:33.869-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:47:36.290-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:47:43.855-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:47:46.291-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:47:53.858-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:47:56.302-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:48:03.856-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:48:06.297-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:48:13.859-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:48:16.297-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:48:23.865-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:48:26.290-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:48:33.851-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:48:36.291-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:48:43.860-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:48:46.296-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:48:53.857-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:48:56.302-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:49:03.848-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:49:06.305-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:49:13.865-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:49:16.296-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:49:23.864-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:49:26.304-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:49:33.851-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:49:36.295-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:49:43.875-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:49:46.294-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:49:53.872-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:49:56.300-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:50:03.859-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:50:06.296-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:50:13.850-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:50:16.298-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:50:23.849-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:50:26.292-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:50:33.854-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:50:36.305-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:50:43.885-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:50:46.292-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:50:53.857-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:50:56.292-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:51:03.863-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:51:06.296-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:51:13.866-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:51:16.298-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:51:23.865-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:51:26.302-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:51:33.849-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:51:36.290-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:51:43.877-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:51:46.292-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:51:53.857-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:51:56.305-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:52:03.851-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:52:06.298-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:52:13.865-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:52:16.298-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:52:23.859-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:52:26.304-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:52:33.868-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:52:36.290-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:52:43.872-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:52:46.307-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:52:53.867-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:52:56.290-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:53:03.862-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:53:06.336-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:53:13.859-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:53:16.294-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:53:23.863-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:53:26.294-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:53:33.852-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:53:36.293-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:53:43.880-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:53:46.312-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:53:53.858-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:53:56.296-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:54:03.852-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:54:06.307-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:54:13.854-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:54:16.295-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:54:23.851-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:54:26.293-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:54:33.863-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:54:36.305-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:54:43.885-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:54:46.306-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:54:53.855-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:54:56.308-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:55:03.875-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:55:06.298-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:55:13.857-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:55:16.292-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:55:23.852-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:55:26.296-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:55:33.857-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:55:36.301-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:55:43.888-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:55:46.289-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:55:53.856-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:55:56.301-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:56:03.878-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:56:06.318-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:56:13.868-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:56:16.293-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:56:23.862-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:56:26.302-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:56:33.853-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:56:36.294-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:56:43.858-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:56:46.292-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:56:53.855-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:56:56.304-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:57:03.853-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:57:06.300-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:57:13.864-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:57:16.289-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:57:23.855-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:57:26.290-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:57:33.859-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:57:36.301-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:57:43.867-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:57:46.287-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:57:53.856-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:57:56.312-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:58:03.857-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:58:06.288-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:58:13.857-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:58:16.291-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:58:23.858-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:58:26.295-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      StatisticsManager 2015-05-11T12:58:33.875-07:00 [INFO] total_doc=80000, docs_processed=60982, changes_left=19018
      StatisticsManager 2015-05-11T12:58:36.294-07:00 [INFO] total_doc=80000, docs_processed=63477, changes_left=16523
      

      XmemNozzle 2015-05-11T12:34:09.846-07:00 [ERROR] Failed to resend document C1-key-84403, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:34:09.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-84403, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:34:12.291-07:00 [ERROR] Failed to resend document C1-key-93779, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:34:12.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-93779, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:34:21.846-07:00 [ERROR] Failed to resend document C1-key-87426, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:34:21.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-87426, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:34:24.292-07:00 [ERROR] Failed to resend document C1-key-98028, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:34:24.292-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-98028, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:34:33.846-07:00 [ERROR] Failed to resend document C1-key-84593, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:34:33.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-84593, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:34:36.291-07:00 [ERROR] Failed to resend document C1-key-94286, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:34:36.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-94286, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:34:45.846-07:00 [ERROR] Failed to resend document C1-key-89201, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:34:45.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-89201, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:34:48.291-07:00 [ERROR] Failed to resend document C1-key-71990, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:34:48.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-71990, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:34:57.848-07:00 [ERROR] Failed to resend document C1-key-86854, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:34:57.848-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-86854, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:35:00.291-07:00 [ERROR] Failed to resend document C1-key-74113, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:35:00.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-74113, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:35:09.847-07:00 [ERROR] Failed to resend document C1-key-89391, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:35:09.847-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-89391, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:35:12.291-07:00 [ERROR] Failed to resend document C1-key-95449, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:35:12.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-95449, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:35:21.846-07:00 [ERROR] Failed to resend document C1-key-87150, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:35:21.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-87150, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:35:24.297-07:00 [ERROR] Failed to resend document C1-key-94316, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:35:24.297-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-94316, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:35:33.846-07:00 [ERROR] Failed to resend document C1-key-89573, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:35:33.847-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-89573, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:35:36.293-07:00 [ERROR] Failed to resend document C1-key-73325, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:35:36.293-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-73325, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:35:45.846-07:00 [ERROR] Failed to resend document C1-key-87622, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:35:45.847-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-87622, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:35:48.292-07:00 [ERROR] Failed to resend document C1-key-74661, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:35:48.292-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-74661, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:35:57.846-07:00 [ERROR] Failed to resend document C1-key-91969, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:35:57.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-91969, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:36:00.291-07:00 [ERROR] Failed to resend document C1-key-96668, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:36:00.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-96668, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:36:09.846-07:00 [ERROR] Failed to resend document C1-key-88891, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:36:09.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-88891, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:36:12.291-07:00 [ERROR] Failed to resend document C1-key-94464, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:36:12.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-94464, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:36:21.846-07:00 [ERROR] Failed to resend document C1-key-94878, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:36:21.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-94878, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:36:24.291-07:00 [ERROR] Failed to resend document C1-key-73457, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:36:24.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-73457, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:36:33.846-07:00 [ERROR] Failed to resend document C1-key-88901, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:36:33.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-88901, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:36:36.291-07:00 [ERROR] Failed to resend document C1-key-75817, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:36:36.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-75817, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:36:45.846-07:00 [ERROR] Failed to resend document C1-key-98208, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:36:45.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-98208, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:36:48.291-07:00 [ERROR] Failed to resend document C1-key-96833, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:36:48.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-96833, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:36:57.846-07:00 [ERROR] Failed to resend document C1-key-89005, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:36:57.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-89005, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:37:00.291-07:00 [ERROR] Failed to resend document C1-key-74881, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:37:00.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-74881, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:37:09.846-07:00 [ERROR] Failed to resend document C1-key-98398, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:37:09.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-98398, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:37:12.291-07:00 [ERROR] Failed to resend document C1-key-75987, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:37:12.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-75987, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:37:21.846-07:00 [ERROR] Failed to resend document C1-key-89195, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:37:21.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-89195, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:37:24.291-07:00 [ERROR] Failed to resend document C1-key-97137, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:37:24.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-97137, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:37:33.846-07:00 [ERROR] Failed to resend document C1-key-89777, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:37:33.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-89777, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:37:36.291-07:00 [ERROR] Failed to resend document C1-key-74911, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:37:36.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-74911, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:37:45.846-07:00 [ERROR] Failed to resend document C1-key-90269, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:37:45.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-90269, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:37:48.291-07:00 [ERROR] Failed to resend document C1-key-77332, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:37:48.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-77332, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:37:57.846-07:00 [ERROR] Failed to resend document C1-key-93048, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:37:57.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-93048, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:38:00.291-07:00 [ERROR] Failed to resend document C1-key-97645, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:38:00.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-97645, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:38:09.846-07:00 [ERROR] Failed to resend document C1-key-95378, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:38:09.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-95378, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:38:12.291-07:00 [ERROR] Failed to resend document C1-key-75015, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:38:12.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-75015, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:38:21.846-07:00 [ERROR] Failed to resend document C1-key-96159, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:38:21.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-96159, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:38:24.291-07:00 [ERROR] Failed to resend document C1-key-77440, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:38:24.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-77440, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:38:33.846-07:00 [ERROR] Failed to resend document C1-key-99898, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:38:33.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-99898, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:38:36.291-07:00 [ERROR] Failed to resend document C1-key-98966, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:38:36.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-98966, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:38:45.846-07:00 [ERROR] Failed to resend document C1-key-99908, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:38:45.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-99908, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:38:48.291-07:00 [ERROR] Failed to resend document C1-key-75185, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:38:48.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-75185, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:38:57.846-07:00 [ERROR] Failed to resend document C1-key-81167, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:38:57.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-81167, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:39:00.291-07:00 [ERROR] Failed to resend document C1-key-79267, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:39:00.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-79267, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:39:09.846-07:00 [ERROR] Failed to resend document C1-key-81615, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:39:09.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-81615, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:39:12.291-07:00 [ERROR] Failed to resend document C1-key-99062, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:39:12.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-99062, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:39:21.846-07:00 [ERROR] Failed to resend document C1-key-81785, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:39:21.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-81785, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:39:24.291-07:00 [ERROR] Failed to resend document C1-key-75767, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:39:24.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-75767, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:39:33.846-07:00 [ERROR] Failed to resend document C1-key-82346, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:39:33.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-82346, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:39:36.291-07:00 [ERROR] Failed to resend document C1-key-79485, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:39:36.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-79485, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:39:45.846-07:00 [ERROR] Failed to resend document C1-key-82434, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:39:45.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-82434, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:39:48.291-07:00 [ERROR] Failed to resend document C1-key-99680, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:39:48.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-99680, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:39:57.846-07:00 [ERROR] Failed to resend document C1-key-84076, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:39:57.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-84076, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:40:00.291-07:00 [ERROR] Failed to resend document C1-key-76234, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:40:00.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-76234, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:40:09.846-07:00 [ERROR] Failed to resend document C1-key-84694, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:40:09.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-84694, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:40:12.291-07:00 [ERROR] Failed to resend document C1-key-79515, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:40:12.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-79515, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:40:21.846-07:00 [ERROR] Failed to resend document C1-key-84704, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:40:21.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-84704, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:40:24.291-07:00 [ERROR] Failed to resend document C1-key-99710, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:40:24.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-99710, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:40:33.846-07:00 [ERROR] Failed to resend document C1-key-85972, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:40:33.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-85972, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:40:36.291-07:00 [ERROR] Failed to resend document C1-key-76546, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:40:36.292-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-76546, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:40:45.846-07:00 [ERROR] Failed to resend document C1-key-87257, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:40:45.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-87257, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:40:48.291-07:00 [ERROR] Failed to resend document C1-key-80839, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:40:48.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-80839, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:40:57.846-07:00 [ERROR] Failed to resend document C1-key-87525, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:40:57.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-87525, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:41:00.291-07:00 [ERROR] Failed to resend document C1-key-78361, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:41:00.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-78361, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:41:09.846-07:00 [ERROR] Failed to resend document C1-key-89292, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:41:09.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-89292, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:41:12.291-07:00 [ERROR] Failed to resend document C1-key-85928, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:41:12.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-85928, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:41:21.846-07:00 [ERROR] Failed to resend document C1-key-89302, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:41:21.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-89302, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:41:24.291-07:00 [ERROR] Failed to resend document C1-key-78413, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:41:24.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-78413, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:41:33.846-07:00 [ERROR] Failed to resend document C1-key-89470, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:41:33.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-89470, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:41:36.291-07:00 [ERROR] Failed to resend document C1-key-89358, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:41:36.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-89358, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:41:45.846-07:00 [ERROR] Failed to resend document C1-key-98479, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:41:45.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-98479, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:41:48.291-07:00 [ERROR] Failed to resend document C1-key-78583, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:41:48.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-78583, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:41:57.846-07:00 [ERROR] Failed to resend document C1-key-85048, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:41:57.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-85048, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:42:00.291-07:00 [ERROR] Failed to resend document C1-key-90134, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:42:00.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-90134, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:42:09.846-07:00 [ERROR] Failed to resend document C1-key-86269, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:42:09.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-86269, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:42:12.291-07:00 [ERROR] Failed to resend document C1-key-80749, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:42:12.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-80749, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:42:21.846-07:00 [ERROR] Failed to resend document C1-key-90854, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:42:21.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-90854, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:42:24.291-07:00 [ERROR] Failed to resend document C1-key-90646, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:42:24.292-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-90646, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:42:33.846-07:00 [ERROR] Failed to resend document C1-key-91150, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:42:33.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-91150, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:42:36.291-07:00 [ERROR] Failed to resend document C1-key-83568, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:42:36.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-83568, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:42:45.846-07:00 [ERROR] Failed to resend document C1-key-91622, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:42:45.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-91622, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:42:48.291-07:00 [ERROR] Failed to resend document C1-key-91830, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:42:48.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-91830, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:42:57.846-07:00 [ERROR] Failed to resend document C1-key-92371, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:42:57.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-92371, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:43:00.291-07:00 [ERROR] Failed to resend document C1-key-85658, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:43:00.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-85658, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:43:09.846-07:00 [ERROR] Failed to resend document C1-key-92403, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:43:09.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-92403, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:43:12.291-07:00 [ERROR] Failed to resend document C1-key-93285, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:43:12.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-93285, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:43:21.846-07:00 [ERROR] Failed to resend document C1-key-92593, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:43:21.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-92593, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:43:24.291-07:00 [ERROR] Failed to resend document C1-key-86479, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:43:24.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-86479, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:43:33.846-07:00 [ERROR] Failed to resend document C1-key-94041, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:43:33.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-94041, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:43:36.291-07:00 [ERROR] Failed to resend document C1-key-93315, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:43:36.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-93315, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:43:45.846-07:00 [ERROR] Failed to resend document C1-key-94733, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:43:45.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-94733, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:43:48.291-07:00 [ERROR] Failed to resend document C1-key-90936, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:43:48.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-90936, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:43:57.846-07:00 [ERROR] Failed to resend document C1-key-95945, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:43:57.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-95945, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:44:00.291-07:00 [ERROR] Failed to resend document C1-key-93467, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:44:00.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-93467, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:44:09.846-07:00 [ERROR] Failed to resend document C1-key-97260, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:44:09.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-97260, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:44:12.291-07:00 [ERROR] Failed to resend document C1-key-91032, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:44:12.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-91032, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:44:21.846-07:00 [ERROR] Failed to resend document C1-key-97482, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:44:21.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-97482, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:44:24.291-07:00 [ERROR] Failed to resend document C1-key-94921, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:44:24.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-94921, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:44:33.846-07:00 [ERROR] Failed to resend document C1-key-97512, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:44:33.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-97512, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:44:36.291-07:00 [ERROR] Failed to resend document C1-key-91740, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:44:36.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-91740, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:44:45.846-07:00 [ERROR] Failed to resend document C1-key-99335, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:44:45.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-99335, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:44:48.291-07:00 [ERROR] Failed to resend document C1-key-95025, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:44:48.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-95025, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:44:57.846-07:00 [ERROR] Failed to resend document C1-key-99447, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:44:57.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-99447, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:45:00.291-07:00 [ERROR] Failed to resend document C1-key-92213, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:45:00.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-92213, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:45:09.846-07:00 [ERROR] Failed to resend document C1-key-83809, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:45:09.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-83809, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:45:12.291-07:00 [ERROR] Failed to resend document C1-key-95757, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:45:12.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-95757, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:45:21.846-07:00 [ERROR] Failed to resend document C1-key-83999, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:45:21.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-83999, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:45:24.291-07:00 [ERROR] Failed to resend document C1-key-92383, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:45:24.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-92383, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:45:33.846-07:00 [ERROR] Failed to resend document C1-key-84002, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:45:33.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-84002, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:45:36.291-07:00 [ERROR] Failed to resend document C1-key-96204, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:45:36.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-96204, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:45:45.846-07:00 [ERROR] Failed to resend document C1-key-87619, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:45:45.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-87619, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:45:48.291-07:00 [ERROR] Failed to resend document C1-key-92561, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:45:48.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-92561, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:45:57.846-07:00 [ERROR] Failed to resend document C1-key-86888, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:45:57.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-86888, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:46:00.291-07:00 [ERROR] Failed to resend document C1-key-96394, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:46:00.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-96394, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:46:09.846-07:00 [ERROR] Failed to resend document C1-key-84192, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:46:09.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-84192, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:46:12.291-07:00 [ERROR] Failed to resend document C1-key-94123, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:46:12.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-94123, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:46:21.846-07:00 [ERROR] Failed to resend document C1-key-87789, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:46:21.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-87789, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:46:24.291-07:00 [ERROR] Failed to resend document C1-key-96576, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:46:24.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-96576, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:46:33.856-07:00 [ERROR] Failed to resend document C1-key-86918, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:46:33.856-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-86918, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:46:36.291-07:00 [ERROR] Failed to resend document C1-key-94651, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:46:36.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-94651, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:46:45.847-07:00 [ERROR] Failed to resend document C1-key-84770, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:46:45.847-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-84770, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:46:48.291-07:00 [ERROR] Failed to resend document C1-key-98351, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:46:48.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-98351, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:46:57.846-07:00 [ERROR] Failed to resend document C1-key-90252, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:46:57.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-90252, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:47:00.291-07:00 [ERROR] Failed to resend document C1-key-95827, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:47:00.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-95827, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:47:09.846-07:00 [ERROR] Failed to resend document C1-key-89149, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:47:09.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-89149, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:47:12.291-07:00 [ERROR] Failed to resend document C1-key-98423, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:47:12.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-98423, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:47:21.846-07:00 [ERROR] Failed to resend document C1-key-85896, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:47:21.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-85896, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:47:24.291-07:00 [ERROR] Failed to resend document C1-key-97292, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:47:24.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-97292, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:47:33.846-07:00 [ERROR] Failed to resend document C1-key-90520, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:47:33.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-90520, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:47:36.291-07:00 [ERROR] Failed to resend document C1-key-97302, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:47:36.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-97302, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:47:45.846-07:00 [ERROR] Failed to resend document C1-key-90325, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:47:45.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-90325, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:47:48.291-07:00 [ERROR] Failed to resend document C1-key-97470, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:47:48.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-97470, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:47:57.846-07:00 [ERROR] Failed to resend document C1-key-85906, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:47:57.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-85906, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:48:00.291-07:00 [ERROR] Failed to resend document C1-key-99257, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:48:00.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-99257, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:48:09.846-07:00 [ERROR] Failed to resend document C1-key-92977, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:48:09.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-92977, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:48:12.291-07:00 [ERROR] Failed to resend document C1-key-99525, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:48:12.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-99525, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:48:21.846-07:00 [ERROR] Failed to resend document C1-key-90457, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:48:21.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-90457, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:48:24.291-07:00 [ERROR] Failed to resend document C1-key-96929, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:48:24.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-96929, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:48:33.846-07:00 [ERROR] Failed to resend document C1-key-87223, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:48:33.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-87223, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:48:36.291-07:00 [ERROR] Failed to resend document C1-key-99178, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:48:36.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-99178, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:48:45.846-07:00 [ERROR] Failed to resend document C1-key-93073, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:48:45.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-93073, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:48:48.291-07:00 [ERROR] Failed to resend document C1-key-72848, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:48:48.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-72848, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:48:57.846-07:00 [ERROR] Failed to resend document C1-key-92800, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:48:57.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-92800, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:49:00.291-07:00 [ERROR] Failed to resend document C1-key-77959, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:49:00.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-77959, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:49:09.846-07:00 [ERROR] Failed to resend document C1-key-87551, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:49:09.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-87551, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:49:12.291-07:00 [ERROR] Failed to resend document C1-key-78098, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:49:12.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-78098, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:49:21.846-07:00 [ERROR] Failed to resend document C1-key-93691, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:49:21.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-93691, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:49:24.291-07:00 [ERROR] Failed to resend document C1-key-78108, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:49:24.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-78108, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:49:33.846-07:00 [ERROR] Failed to resend document C1-key-92990, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:49:33.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-92990, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:49:36.291-07:00 [ERROR] Failed to resend document C1-key-80252, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:49:36.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-80252, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:49:45.846-07:00 [ERROR] Failed to resend document C1-key-89376, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:49:45.846-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/standard_bucket_1/standard_bucket_1_172.23.107.206:11210_0 connection client_setMeta is broken due to Failed to resend document C1-key-89376, has tried to resend it 6, maximum retry 5 reached, try to repair...
      XmemNozzle 2015-05-11T12:49:48.291-07:00 [ERROR] Failed to resend document C1-key-80520, has tried to resend it 6, maximum retry 5 reached
      XmemNozzle 2015-05-11T12:49:48.291-07:00 [ERROR] xmem_449fcf5248457209d6f8eb8f800fe6e7/default/default_172.23.107.206:11210_1 connection client_setMeta is broken due to Failed to resend document C1-key-80520, has tried to resend it 6, maximum retry 5 reached, try to repair...
      :
      :
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            apiravi Aruna Piravi (Inactive)
            apiravi Aruna Piravi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty