CLONE - [XDCR] data items mismatch between source and dest cluster

2017-11-09 15:18:51 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://172.23.123.245:8091/pools/default/buckets/default/stats?zoom=minute

2017-11-09 15:18:51 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3500 == 7000 expected on C2, default bucket

2017-11-09 15:18:54 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://172.23.123.245:8091/pools/default/buckets/default/stats?zoom=minute

2017-11-09 15:18:54 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3500 == 7000 expected on C2, default bucket

2017-11-09 15:18:57 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://172.23.123.245:8091/pools/default/buckets/default/stats?zoom=minute

2017-11-09 15:18:57 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3500 == 7000 expected on C2, default bucket

2017-11-09 15:19:00 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://172.23.123.245:8091/pools/default/buckets/default/stats?zoom=minute

2017-11-09 15:19:00 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3500 == 7000 expected on C2, default bucket

2017-11-09 15:19:03 | INFO | MainProcess | test_thread | [rest_client.fetch_bucket_stats] http://172.23.123.245:8091/pools/default/buckets/default/stats?zoom=minute

2017-11-09 15:19:03 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3500 == 7000 expected on C2, default bucket

2017-11-09 15:19:06 | ERROR | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] ERROR: Timed-out waiting for replica item count to match

2017-11-09 15:19:06 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.123.245 with username:root password:couchbase ssh_key:

2017-11-09 15:19:06 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.123.245

2017-11-09 15:19:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] It could be other operating systyem.  Go to check at other location

2017-11-09 15:19:07 | INFO | MainProcess | test_thread | [xdcrnewbasetests.run_cbvdiff] Executing cbvdiff for bucket default

2017-11-09 15:19:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.123.245: /opt/couchbase/bin/cbvdiff -b default 172.23.123.245:11210,172.23.127.31:11210 

2017-11-09 15:19:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully

2017-11-09 15:19:07 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Authentication error for default

2017-11-09 15:19:07 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.121.211:11210 default

2017-11-09 15:19:07 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.123.244:11210 default

2017-11-09 15:19:07 | INFO | MainProcess | test_thread | [task.__init__] 10000 items will be verified on default bucket

2017-11-09 15:19:07 | INFO | MainProcess | load_gen_task | [task.has_next] 0 items were verified

2017-11-09 15:19:13 | INFO | MainProcess | load_gen_task | [task.has_next] 10000 items were verified

2017-11-09 15:19:13 | INFO | MainProcess | load_gen_task | [task.has_next] 10000 items were verified in 5.25594186783 sec.the average number of ops                - 1902.60778542 per second

2017-11-09 15:19:13 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.123.245:11210 default

2017-11-09 15:19:13 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.127.31:11210 default

2017-11-09 15:19:13 | INFO | MainProcess | test_thread | [task.__init__] 7000 items will be verified on default bucket

2017-11-09 15:19:13 | INFO | MainProcess | load_gen_task | [task.has_next] 0 items were verified

[('/usr/lib64/python2.7/threading.py', 784, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 811, '__bootstrap_inner', 'self.run()'), ('./testrunner.py', 297, 'run', '**self._Thread__kwargs)'), ('/usr/lib64/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib64/python2.7/unittest/case.py'[('/usr/lib64/python2.7/threading.py', 784, '__bootstrap', 'self.__bootstrap_inner(), ')433, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib64/python2.7/unittest/case.py', 369, 'run', 'testMethod()'), ('pytests/xdcr/rebalanceXDCR.py', 139, 'swap_rebalance', 'self.verify_results()'), ('pytests/xdcr/xdcrnewbasetests.py', 3519, 'verify_results', 'dest_cluster.verify_data(max_verify=self._max_verify, skip=skip_verify_data, )'(), ('pytests/xdcr/xdcrnewbasetests.py', 2352, 'verify_data', 'task.result(timeout)'), ('lib/tasks/future.py', 160, 'result', ''/ruetursrn/ lsib64/python2.7/threading.py', 811, '__bootstrap_inneelf.__get_result()'), ('lib/tasks/future.py', 111, '__get_result', 'print traceback.extract_stack()')]

r'2017-11-09 15:19:14 | ERROR | MainProcess | test_thread | [xdcrnewbasetests.verify_results] Not able to get values for following keys - set(['C1-key-5142', 'C1-key-5410', 'C1-key-1911', 'C1-key-4940', 'C1-key-1520', 'C1-key-4416', 'C1-key-794', 'C1-key-5961', 'C1-key-4861', 'C1-key-3748', 'C1-key-3030', 'C1-key-3747', 'C1-key-4781', 'C1-key-3422', 'C1-key-3580', 'C1-key-2913', 'C1-key-5091', 'C1-key-4652', 'C1-key-322', 'C1-key-4468', 'C1-key-5806', 'C1-key-280', 'C1-key-5567', 'C1-key-2424', 'C1-key-6061', 'C1-key-2036', 'C1-key-2784', 'C1-key-2039', 'C1-key-1351', 'C1-key-1457', 'C1-key-3345', 'C1-key-6991', 'C1-key-766', 'C1-key-5895', 'C1-key-3739', 'C1-key-3922', 'C1-key-4556', 'C1-key-3736', 'C1-key-647', 'C1-key-769', 'C1-key-2730', 'C1-key-2047', 'C1-key-5787', 'C1-key-5788', 'C1-key-5025', 'C1-key-2322', 'C1-key-2048', 'C1-key-2255', 'C1-key-6610', 'C1-key-3324', 'C1-key-12', 'C1-key-1186', 'C1-key-6195', 'C1-key-1672', 'C1-key-424', 'C1-key-3813', 'C1-key-6291', 'C1-key-6842', 'C1-key-6109', 'C1-key-2130', 'C1-key-63', 'C1-key-4633', 'C1-key-4221', 'C1-key-864', 'C1-key-6444', 'C1-key-6333', 'C1-key-3499', 'C1-key-396', 'C1-key-3253', 'C1-key-6721', 'C1-key-6683', 'C1-key-899', 'C1-key-1174', 'C1-key-5321', 'C1-key-1270', 'C1-key-4642', 'C1-key-4135', 'C1-key-4528', 'C1-key-4527', 'C1-key-4985', 'C1-key-4359', 'C1-key-5516', 'C1-key-5928', 'C1-key-4828', 'C1-key-3782', 'C1-key-5927', 'C1-key-692', 'C1-key-6740', 'C1-key-5691', 'C1-key-3076', 'C1-key-4278', 'C1-key-3079', 'C1-key-6435', 'C1-key-5197', 'C1-key-2815', 'C1-key-5210', 'C1-key-5602', 'C1-key-5568', 'C1-key-5073', 'C1-key-2374', 'C1-key-1820', 'C1-key-5175', 'C1-key-5850', 'C1-key-1551', 'C1-key-3679', 'C1-key-6898', 'C1-key-4919', 'C1-key-4856', 'C1-key-2779', 'C1-key-3770', 'C1-key-176', 'C1-key-2776', 'C1-key-3474', 'C1-key-5459', 'C1-key-1957', 'C1-key-1958', 'C1-key-5104', 'C1-key-374', 'C1-key-54', 'C1-key-3986', 'C1-key-2107', 'C1-key-2455', 'C1-key-6093', 'C1-key-1732', 'C1-key-3989', 'C1-key-3862', 'C1-key-1307', 'C1-key-3', 'C1-key-3855', 'C1-key-2589', 'C1-key-730', 'C1-key-6804', 'C1-key-3607', 'C1-key-108', 'C1-key-6028', 'C1-key-1789', 'C1-key-1786', 'C1-key-6027', 'C1-key-5775', 'C1-key-490', 'C1-key-2070', 'C1-key-6375', 'C1-key-2203', 'C1-key-6273', 'C1-key-3355', 'C1-key-515', 'C1-key-2194', 'C1-key-1603', 'C1-key-455', 'C1-key-1236', 'C1-key-1239', 'C1-key-4173', 'C1-key-4075', 'C1-key-2290', 'C1-key-6659', 'C1-key-2586', 'C1-key-3915', 'C1-key-6875', 'C1-key-5385', 'C1-key-4310', 'C1-key-955', 'C1-key-6575', 'C1-key-4250', 'C1-key-3136', 'C1-key-2990', 'C1-key-815', 'C1-key-853', 'C1-key-6473', 'C1-key-2853', 'C1-key-6777', 'C1-key-5256', 'C1-key-5316', 'C1-key-1122', 'C1-key-4383', 'C1-key-5259', 'C1-key-822', 'C1-key-4773', 'C1-key-4675', 'C1-key-3182', 'C1-key-5133', 'C1-key-1517', 'C1-key-1518', 'C1-key-3630', 'C1-key-4950', 'C1-key-5956', 'C1-key-3008', 'C1-key-4791', 'C1-key-3007', 'C1-key-5959', 'C1-key-3432', 'C1-key-5550', 'C1-key-5227', 'C1-key-4495', 'C1-key-332', 'C1-key-2179', 'C1-key-5654', 'C1-key-6056', 'C1-key-2026', 'C1-key-4571', 'C1-key-2029', 'C1-key-6059', 'C1-key-4921', 'C1-key-3729', 'C1-key-776', 'C1-key-1926', 'C1-key-779', 'C1-key-3726', 'C1-key-2707', 'C1-key-1901', 'C1-key-2708', 'C1-key-2332', 'C1-key-3313', 'C1-key-1645', 'C1-key-413', 'C1-key-1530', 'C1-key-1884', 'C1-key-6281', 'C1-key-6833', 'C1-key-2766', 'C1-key-2413', 'C1-key-4623', 'C1-key-6304', 'C1-key-1080', 'C1-key-6731', 'C1-key-1164', 'C1-key-1248', 'C1-key-1247', 'C1-key-4538', 'C1-key-4102', 'C1-key-4537', 'C1-key-5663', 'C1-key-5910', 'C1-key-2553', 'C1-key-3041', 'C1-key-682', 'C1-key-6425', 'C1-key-2805', 'C1-key-5261', 'C1-key-4219', 'C1-key-5612', 'C1-key-4216', 'C1-key-2692', 'C1-key-2343', 'C1-key-1939', 'C1-key-1541', 'C1-key-5165', 'C1-key-6795', 'C1-key-4967', 'C1-key-3684', 'C1-key-4968', 'C1-key-2741', 'C1-key-3760', 'C1-key-184', 'C1-key-3443', 'C1-key-1947', 'C1-key-5509', 'C1-key-5506', 'C1-key-1948', 'C1-key-343', 'C1-key-44', 'C1-key-2445', 'C1-key-3824', 'C1-key-5483', 'C1-key-708', 'C1-key-2599', 'C1-key-315', 'C1-key-2596', 'C1-key-707', 'C1-key-6010', 'C1-key-2060', 'C1-key-4097', 'C1-key-4546', 'C1-key-4549', 'C1-key-4098', 'C1-key-1613', 'C1-key-1201', 'C1-key-445', 'C1-key-4144', 'C1-key-2280', 'C1-key-3834', 'C1-key-6865', 'C1-key-4712', 'C1-key-6587', 'C1-key-6588', 'C1-key-805', 'C1-key-6463', 'C1-key-2843', 'C1-key-6778', 'C1-key-1115', 'C1-key-4393', 'C1-key-3503', 'C1-key-4665', 'C1-key-6342', 'C1-key-1507', 'C1-key-5531', 'C1-key-5949', 'C1-key-5123', 'C1-key-5946', 'C1-key-1508', 'C1-key-3017', 'C1-key-3018', 'C1-key-3405', 'C1-key-2832', 'C1-key-305', 'C1-key-4485', 'C1-key-5540', 'C1-key-5237', 'C1-key-5238', 'C1-key-5625', 'C1-key-4893', 'C1-key-3555', 'C1-key-2011', 'C1-key-1576', 'C1-key-4931', 'C1-key-1579', 'C1-key-2718', 'C1-key-980', 'C1-key-3711', 'C1-key-2717', 'C1-key-2305', 'C1-key-1970', 'C1-key-3303', 'C1-key-2472', 'C1-key-35', 'C1-key-1655', 'C1-key-6365', 'C1-key-403', 'C1-key-741', 'C1-key-1894', 'C1-key-1936', 'C1-key-1482', 'C1-key-3872', 'C1-key-6049', 'C1-key-6823', 'C1-key-6046', 'C1-key-2403', 'C1-key-6709', 'C1-key-4614', 'C1-key-843', 'C1-key-2057', 'C1-key-6314', 'C1-key-6706', 'C1-key-1090', 'C1-key-3372', 'C1-key-2058', 'C1-key-1153', 'C1-key-1624', 'C1-key-472', 'C1-key-1257', 'C1-key-4112', 'C1-key-1258', 'C1-key-1394', 'C1-key-4500', 'C1-key-5673', 'C1-key-5900', 'C1-key-4277', 'C1-key-3051', 'C1-key-832', 'C1-key-6412', 'C1-key-3286', 'C1-key-2874', 'C1-key-5271', 'C1-key-3289', 'C1-key-586', 'C1-key-4209', 'C1-key-4206', 'C1-key-589', 'C1-key-2682', 'C1-key-1929', 'C1-key-5152', 'C1-key-3758', 'C1-key-5971', 'C1-key-4977', 'C1-key-3694', 'C1-key-3020', 'C1-key-4978', 'C1-key-3757', 'C1-key-3453', 'C1-key-194', 'C1-key-2751', 'C1-key-3590', 'C1-key-5081', 'C1-key-353', 'C1-key-5578', 'C1-key-5577', 'C1-key-290', 'C1-key-6071', 'C1-key-2434', 'C1-key-4163', 'C1-key-5493', 'C1-key-6981', 'C1-key-5885', 'C1-key-717', 'C1-key-718', 'C1-key-2720', 'C1-key-5797', 'C1-key-6000', 'C1-key-2353', 'C1-key-5798', 'C1-key-4087', 'C1-key-5519', 'C1-key-3334', 'C1-key-4088', 'C1-key-6185', 'C1-key-1662', 'C1-key-434', 'C1-key-1211', 'C1-key-4154', 'C1-key-3803', 'C1-key-73', 'C1-key-6852', 'C1-key-4231', 'C1-key-6597', 'C1-key-6598', 'C1-key-874', 'C1-key-6454', 'C1-key-6750', 'C1-key-6693', 'C1-key-1105', 'C1-key-1260', 'C1-key-4559', 'C1-key-2601', 'C1-key-4125', 'C1-key-6352', 'C1-key-5521', 'C1-key-5938', 'C1-key-5937', 'C1-key-3066', 'C1-key-3415', 'C1-key-3069', 'C1-key-5228', 'C1-key-2822', 'C1-key-1003', 'C1-key-6768', 'C1-key-5200', 'C1-key-5635', 'C1-key-4883', 'C1-key-2364', 'C1-key-4909', 'C1-key-1566', 'C1-key-1569', 'C1-key-4906', 'C1-key-751', 'C1-key-3701', 'C1-key-4240', 'C1-key-2769', 'C1-key-990', 'C1-key-3464', 'C1-key-6323', 'C1-key-2315', 'C1-key-1960', 'C1-key-5114', 'C1-key-364', 'C1-key-3996', 'C1-key-25', 'C1-key-2462', 'C1-key-3999', 'C1-key-3845', 'C1-key-1492', 'C1-key-720', 'C1-key-6814', 'C1-key-1799', 'C1-key-6038', 'C1-key-6037', 'C1-key-1796', 'C1-key-2001', 'C1-key-4561', 'C1-key-6719', 'C1-key-6716', 'C1-key-3362', 'C1-key-1634', 'C1-key-1143', 'C1-key-2184', 'C1-key-1226', 'C1-key-462', 'C1-key-1384', 'C1-key-1229', 'C1-key-4510', 'C1-key-4268', 'C1-key-5644', 'C1-key-5395', 'C1-key-4267', 'C1-key-6402', 'C1-key-2980', 'C1-key-6767', 'C1-key-2864', 'C1-key-3296', 'C1-key-5246', 'C1-key-5249', 'C1-key-1132', 'C1-key-3299', 'C1-key-596', 'C1-key-4916', 'C1-key-599', 'C1-key-3192', 'C1-key-4604'])

, 'self.run()'), ('lib/tasks/task.py', 697, 'run', 'self.next()'), ('lib/tasks/task.py', 1776, 'next', 'self._check_valid_keys(keys_batch)'), ('lib/tasks/task.py', 1788, '_check_valid_keys', 'self.set_exception(error)'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]

Thu Nov  9 15:19:14 2017

2017-11-09 15:19:14 | INFO | MainProcess | test_thread | [xdcrnewbasetests.verify_rev_ids] Verifying RevIds for Couchbase Cluster: C1, Master Ip: 172.23.121.211 -> Couchbase Cluster: C2, Master Ip: 172.23.123.245, bucket default

2017-11-09 15:19:14 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.121.211:11210 default

2017-11-09 15:19:14 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.123.244:11210 default

2017-11-09 15:19:14 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.121.211:11210 default

2017-11-09 15:19:14 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.123.244:11210 default

2017-11-09 15:19:14 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.123.245:11210 default

2017-11-09 15:19:14 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.127.31:11210 default

2017-11-09 15:19:15 | INFO | MainProcess | test_thread | [task.__init__] RevID verification: in progress for default ...

2017-11-09 15:19:32 | INFO | MainProcess | load_gen_task | [task.has_next] RevId Verification : 7000 existing items have been verified

2017-11-09 15:19:32 | INFO | MainProcess | load_gen_task | [task.has_next] RevId Verification : 3000 deleted items have been verified

2017-11-09 15:19:32 | INFO | MainProcess | load_gen_task | [task.has_next] RevId Verification : 1500 keys were apparently filtered out and not found in target bucket

2017-11-09 15:19:32 | INFO | MainProcess | load_gen_task | [rest_client.fetch_bucket_stats] http://172.23.121.211:8091/pools/default/buckets/default/stats?zoom=minute

2017-11-09 15:19:32 | ERROR | MainProcess | test_thread | [xdcrnewbasetests.verify_rev_ids] 3500 keys not found on 172.23.123.245, printing first 100 keys: [('key: C1-key-4097', 'vbucket: 202'), ('key: C1-key-5142', 'vbucket: 112'), ('key: C1-key-4546', 'vbucket: 168'), ('key: C1-key-6071', 'vbucket: 291'), ('key: C1-key-5961', 'vbucket: 348'), ('key: C1-key-3748', 'vbucket: 323'), ('key: C1-key-3030', 'vbucket: 406'), ('key: C1-key-1911', 'vbucket: 383'), ('key: C1-key-445', 'vbucket: 215'), ('key: C1-key-3580', 'vbucket: 169'), ('key: C1-key-2280', 'vbucket: 346'), ('key: C1-key-322', 'vbucket: 422'), ('key: C1-key-6865', 'vbucket: 326'), ('key: C1-key-4240', 'vbucket: 132'), ('key: C1-key-2036', 'vbucket: 73'), ('key: C1-key-6588', 'vbucket: 428'), ('key: C1-key-805', 'vbucket: 417'), ('key: C1-key-2843', 'vbucket: 369'), ('key: C1-key-2779', 'vbucket: 469'), ('key: C1-key-3736', 'vbucket: 442'), ('key: C1-key-4393', 'vbucket: 481'), ('key: C1-key-2730', 'vbucket: 101'), ('key: C1-key-4623', 'vbucket: 222'), ('key: C1-key-6342', 'vbucket: 321'), ('key: C1-key-2048', 'vbucket: 176'), ('key: C1-key-1507', 'vbucket: 29'), ('key: C1-key-5949', 'vbucket: 433'), ('key: C1-key-5123', 'vbucket: 301'), ('key: C1-key-1201', 'vbucket: 49'), ('key: C1-key-424', 'vbucket: 394'), ('key: C1-key-3018', 'vbucket: 379'), ('key: C1-key-3405', 'vbucket: 472'), ('key: C1-key-3813', 'vbucket: 186'), ('key: C1-key-6291', 'vbucket: 36'), ('key: C1-key-5540', 'vbucket: 375'), ('key: C1-key-5238', 'vbucket: 418'), ('key: C1-key-5625', 'vbucket: 257'), ('key: C1-key-4221', 'vbucket: 473'), ('key: C1-key-864', 'vbucket: 252'), ('key: C1-key-6444', 'vbucket: 365'), ('key: C1-key-4931', 'vbucket: 151'), ('key: C1-key-1579', 'vbucket: 228'), ('key: C1-key-4144', 'vbucket: 431'), ('key: C1-key-2718', 'vbucket: 136'), ('key: C1-key-980', 'vbucket: 208'), ('key: C1-key-4642', 'vbucket: 387'), ('key: C1-key-4527', 'vbucket: 501'), ('key: C1-key-1970', 'vbucket: 34'), ('key: C1-key-5516', 'vbucket: 355'), ('key: C1-key-5928', 'vbucket: 236'), ('key: C1-key-3303', 'vbucket: 500'), ('key: C1-key-1655', 'vbucket: 290'), ('key: C1-key-692', 'vbucket: 153'), ('key: C1-key-5091', 'vbucket: 277'), ('key: C1-key-741', 'vbucket: 508'), ('key: C1-key-1894', 'vbucket: 14'), ('key: C1-key-3079', 'vbucket: 38'), ('key: C1-key-1482', 'vbucket: 364'), ('key: C1-key-2815', 'vbucket: 357'), ('key: C1-key-5210', 'vbucket: 335'), ('key: C1-key-3872', 'vbucket: 487'), ('key: C1-key-5797', 'vbucket: 313'), ('key: C1-key-6046', 'vbucket: 106'), ('key: C1-key-2403', 'vbucket: 7'), ('key: C1-key-6709', 'vbucket: 502'), ('key: C1-key-4614', 'vbucket: 407'), ('key: C1-key-2057', 'vbucket: 276'), ('key: C1-key-6314', 'vbucket: 341'), ('key: C1-key-5175', 'vbucket: 313'), ('key: C1-key-1551', 'vbucket: 9'), ('key: C1-key-1153', 'vbucket: 270'), ('key: C1-key-4919', 'vbucket: 122'), ('key: C1-key-472', 'vbucket: 414'), ('key: C1-key-1257', 'vbucket: 37'), ('key: C1-key-4112', 'vbucket: 443'), ('key: C1-key-6375', 'vbucket: 8'), ('key: C1-key-3362', 'vbucket: 169'), ('key: C1-key-1958', 'vbucket: 207'), ('key: C1-key-374', 'vbucket: 434'), ('key: C1-key-5673', 'vbucket: 277'), ('key: C1-key-2455', 'vbucket: 19'), ('key: C1-key-4277', 'vbucket: 461'), ('key: C1-key-3989', 'vbucket: 111'), ('key: C1-key-3051', 'vbucket: 203'), ('key: C1-key-832', 'vbucket: 232'), ('key: C1-key-6412', 'vbucket: 377'), ('key: C1-key-3286', 'vbucket: 133'), ('key: C1-key-2874', 'vbucket: 56'), ('key: C1-key-6804', 'vbucket: 27'), ('key: C1-key-5271', 'vbucket: 18'), ('key: C1-key-5900', 'vbucket: 1'), ('key: C1-key-2305', 'vbucket: 43'), ('key: C1-key-1786', 'vbucket: 71'), ('key: C1-key-6027', 'vbucket: 311'), ('key: C1-key-589', 'vbucket: 22'), ('key: C1-key-4571', 'vbucket: 481'), ('key: C1-key-1926', 'vbucket: 54'), ('key: C1-key-1530', 'vbucket: 340'), ('key: C1-key-3355', 'vbucket: 480'), ('key: C1-key-4978', 'vbucket: 295')]

FAIL

2017-11-09 15:19:32 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'5.1.0-1368-enterprise', u'5.1.0-1368-enterprise']

2017-11-09 15:19:32 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'5.1.0-1368-enterprise', u'5.1.0-1368-enterprise']

2017-11-09 15:19:32 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.tearDown] CLEANUP WAS SKIPPED

======================================================================

FAIL: swap_rebalance (xdcr.rebalanceXDCR.Rebalance)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "pytests/xdcr/rebalanceXDCR.py", line 139, in swap_rebalance

    self.verify_results()

  File "pytests/xdcr/xdcrnewbasetests.py", line 3529, in verify_results

    self.fail("Incomplete replication: Active key count is incorrect")

AssertionError: Incomplete replication: Active key count is incorrect

----------------------------------------------------------------------