Details
-
Bug
-
Resolution: Fixed
-
Test Blocker
-
3.0
-
Security Level: Public
-
None
-
Untriaged
-
Yes
-
June 30 - July 18
Description
Build
--------
3.0.0-741
Scenario (internal repl = UPR, XDCR on UPR)
-------------
1. Uni-xdcr between 2 clusters
2. Load 1000 items on source
3. Warmup non-master source nodes
4. Delete last 30% of keys loaded (loadOne699 to loadOne999)
5. Do item count, revID, value verification on both clusters.
Regression?
--------------------
Yes. Was not seen on build 724 on which XDCR sanity last ran. However in 724, default internal replication used TAP.
Testrunner command to run test
--------------------------------------------------
./testrunner -i /tmp/XDCR-sanity.ini -t xdcr.uniXDCR.unidirectional.load_with_async_ops_with_warmup,items=1000,rdirection=unidirection,ctopology=chain,doc-ops=delete,warm=source,replication_type=xmem
RevID verification info
-------------------------------
2014-05-27 11:39:19 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:19 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:24 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:25 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:25 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:30 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:30 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:30 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:35 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:35 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:35 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:40 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:40 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:40 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:45 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:45 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:45 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:50 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:50 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:50 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:55 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:55 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:39:55 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:40:00 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:40:00 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
2014-05-27 11:40:00 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
[('/usr/lib/python2.7/threading.py', 524, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.7/threading.py', 551, '__bootstrap_inner', 'self.run()'), ('testrunner', 262, 'run', '**self._Thread__kwargs)'), ('/usr/lib/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib/python2.7/unittest/case.py', 391, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib/python2.7/unittest/case.py', 327, 'run', 'testMethod()'), ('pytests/xdcr/uniXDCR.py', 108, 'load_with_async_ops_with_warmup', 'self.verify_results()'), ('pytests/xdcr/xdcrbasetests.py', 1417, 'verify_results', 'self.verify_xdcr_stats(src_nodes, dest_nodes, verify_src)'), ('pytests/xdcr/xdcrbasetests.py', 1385, 'verify_xdcr_stats', 'self._verify_item_count(dest_nodes, timeout=timeout)'), ('pytests/xdcr/xdcrbasetests.py', 1319, '_verify_item_count', 'task.result(timeout)'), ('lib/tasks/future.py', 162, 'result', 'self.set_exception(TimeoutError())'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]
Tue May 27 11:40:01 2014
2014-05-27 11:40:01 | INFO | MainProcess | test_thread | [xdcrbasetests.verify_xdcr_stats] Verifying RevIds for 172.23.106.45 -> 172.23.106.47 XDCR
2014-05-27 11:40:01 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.45:11210 default
2014-05-27 11:40:01 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.46:11210 default
2014-05-27 11:40:02 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.47:11210 default
2014-05-27 11:40:02 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.48:11210 default
2014-05-27 11:40:03 | INFO | MainProcess | load_gen_task | [task.has_next] RevId Verification : 700 existing items have been verified
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne934 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:1
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:2
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947034330607, Destination cas:4562947034330606, Error Count:3
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947034330606, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne886 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:4
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:5
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947051666520, Destination cas:4562947051666519, Error Count:6
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947051666519, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne899 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:7
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:8
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947037989004, Destination cas:4562947037989003, Error Count:9
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947037989003, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne794 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:10
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:11
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947044557472, Destination cas:4562947044557471, Error Count:12
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947044557471, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne962 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:13
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:14
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947044198546, Destination cas:4562947044198545, Error Count:15
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947044198545, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne924 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:16
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:17
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947048758593, Destination cas:4562947048758592, Error Count:18
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947048758592, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne913 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:19
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:20
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947046369562, Destination cas:4562947046369561, Error Count:21
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947046369561, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne945 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:22
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:23
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947034685125, Destination cas:4562947034685124, Error Count:24
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947034685124, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne889 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:25
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:26
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947051596958, Destination cas:4562947051596957, Error Count:27
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947051596957, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne896 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:28
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:29
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947038394899, Destination cas:4562947038394898, Error Count:30
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947038394898, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne972 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:31
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:32
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947037791994, Destination cas:4562947037791993, Error Count:33
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947037791993, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne805 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:34
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:35
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947037081618, Destination cas:4562947037081617, Error Count:36
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947037081617, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne720 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:37
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:38
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947034962560, Destination cas:4562947034962559, Error Count:39
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947034962559, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne717 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:40
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:41
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947038004570, Destination cas:4562947038004569, Error Count:42
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947038004569, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne980 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:43
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:44
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947047110994, Destination cas:4562947047110993, Error Count:45
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947047110993, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne864 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:46
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:47
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947051233069, Destination cas:4562947051233068, Error Count:48
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947051233068, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne853 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:49
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:50
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947047624566, Destination cas:4562947047624565, Error Count:51
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947047624565, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne741 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:52
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:53
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947041380736, Destination cas:4562947041380735, Error Count:54
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947041380735, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne832 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:55
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:56
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947035292464, Destination cas:4562947035292463, Error Count:57
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947035292463, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne769 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:58
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:59
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947048593869, Destination cas:4562947048593868, Error Count:60
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947048593868, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne708 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:61
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:62
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947044272223, Destination cas:4562947044272222, Error Count:63
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947044272222, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne815 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:64
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:65
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947043234311, Destination cas:4562947043234310, Error Count:66
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947043234310, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne730 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:67
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:68
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947049453109, Destination cas:4562947049453108, Error Count:69
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947049453108, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne843 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:70
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:71
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947041016346, Destination cas:4562947041016345, Error Count:72
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947041016345, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne707 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:73
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:74
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947050857638, Destination cas:4562947050857637, Error Count:75
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947050857637, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne766 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:76
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:77
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947051608206, Destination cas:4562947051608205, Error Count:78
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947051608205, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne990 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:79
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:80
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947040882035, Destination cas:4562947040882034, Error Count:81
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947040882034, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne718 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:82
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:83
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947038383779, Destination cas:4562947038383778, Error Count:84
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947038383778, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne779 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:85
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:86
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947050523555, Destination cas:4562947050523554, Error Count:87
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947050523554, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne915 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:88
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:89
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947046032126, Destination cas:4562947046032125, Error Count:90
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947046032125, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne922 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:91
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:92
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947048650377, Destination cas:4562947048650376, Error Count:93
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947048650376, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne943 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:94
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:95
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947034705694, Destination cas:4562947034705693, Error Count:96
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947034705693, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne890 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:97
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:98
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947038315972, Destination cas:4562947038315971, Error Count:99
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947038315971, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne974 =====
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:100
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:101
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947037692074, Destination cas:4562947037692073, Error Count:102
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:
2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:
{'deleted': 0, 'seqno': 1, 'cas': 4562947037692073, 'flags': 0, 'expiration': 0}2014-05-27 11:40:03 | INFO | MainProcess | load_gen_task | [task.has_next] RevId Verification : 86 deleted items have been verified
FAIL