Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-11228

XDCR : UPR connections not re-established after bucket warmup

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Test Blocker
    • 3.0
    • 3.0
    • couchbase-bucket
    • Security Level: Public
    • None
    • Untriaged
    • Yes
    • June 30 - July 18

    Description

      Build
      --------
      3.0.0-741

      Scenario (internal repl = UPR, XDCR on UPR)
      -------------
      1. Uni-xdcr between 2 clusters
      2. Load 1000 items on source
      3. Warmup non-master source nodes
      4. Delete last 30% of keys loaded (loadOne699 to loadOne999)
      5. Do item count, revID, value verification on both clusters.

      Regression?
      --------------------
      Yes. Was not seen on build 724 on which XDCR sanity last ran. However in 724, default internal replication used TAP.

      Testrunner command to run test
      --------------------------------------------------
      ./testrunner -i /tmp/XDCR-sanity.ini -t xdcr.uniXDCR.unidirectional.load_with_async_ops_with_warmup,items=1000,rdirection=unidirection,ctopology=chain,doc-ops=delete,warm=source,replication_type=xmem

      RevID verification info
      -------------------------------

      2014-05-27 11:39:19 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:19 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:24 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:25 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:25 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:30 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:30 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:30 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:35 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:35 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:35 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:40 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:40 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:40 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:45 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:45 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:45 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:50 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:50 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:50 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:55 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:55 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:39:55 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:40:00 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:40:00 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      2014-05-27 11:40:00 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 810 == 700 expected on '172.23.106.47:8091''172.23.106.48:8091', default bucket
      [('/usr/lib/python2.7/threading.py', 524, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.7/threading.py', 551, '__bootstrap_inner', 'self.run()'), ('testrunner', 262, 'run', '**self._Thread__kwargs)'), ('/usr/lib/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib/python2.7/unittest/case.py', 391, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib/python2.7/unittest/case.py', 327, 'run', 'testMethod()'), ('pytests/xdcr/uniXDCR.py', 108, 'load_with_async_ops_with_warmup', 'self.verify_results()'), ('pytests/xdcr/xdcrbasetests.py', 1417, 'verify_results', 'self.verify_xdcr_stats(src_nodes, dest_nodes, verify_src)'), ('pytests/xdcr/xdcrbasetests.py', 1385, 'verify_xdcr_stats', 'self._verify_item_count(dest_nodes, timeout=timeout)'), ('pytests/xdcr/xdcrbasetests.py', 1319, '_verify_item_count', 'task.result(timeout)'), ('lib/tasks/future.py', 162, 'result', 'self.set_exception(TimeoutError())'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]
      Tue May 27 11:40:01 2014
      2014-05-27 11:40:01 | INFO | MainProcess | test_thread | [xdcrbasetests.verify_xdcr_stats] Verifying RevIds for 172.23.106.45 -> 172.23.106.47 XDCR
      2014-05-27 11:40:01 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.45:11210 default
      2014-05-27 11:40:01 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.46:11210 default
      2014-05-27 11:40:02 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.47:11210 default
      2014-05-27 11:40:02 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.48:11210 default
      2014-05-27 11:40:03 | INFO | MainProcess | load_gen_task | [task.has_next] RevId Verification : 700 existing items have been verified
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne934 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:1
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:2
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947034330607, Destination cas:4562947034330606, Error Count:3
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947034330607, 'flags': 0, 'expiration': 1401215452}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947034330606, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne886 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:4
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:5
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947051666520, Destination cas:4562947051666519, Error Count:6
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947051666520, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947051666519, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne899 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:7
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:8
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947037989004, Destination cas:4562947037989003, Error Count:9
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947037989004, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947037989003, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne794 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:10
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:11
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947044557472, Destination cas:4562947044557471, Error Count:12
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947044557472, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947044557471, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne962 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:13
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:14
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947044198546, Destination cas:4562947044198545, Error Count:15
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947044198546, 'flags': 0, 'expiration': 1401215452}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947044198545, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne924 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:16
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:17
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947048758593, Destination cas:4562947048758592, Error Count:18
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947048758593, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947048758592, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne913 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:19
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:20
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947046369562, Destination cas:4562947046369561, Error Count:21
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947046369562, 'flags': 0, 'expiration': 1401215452}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947046369561, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne945 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:22
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:23
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947034685125, Destination cas:4562947034685124, Error Count:24
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947034685125, 'flags': 0, 'expiration': 1401215452}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947034685124, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne889 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:25
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:26
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947051596958, Destination cas:4562947051596957, Error Count:27
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947051596958, 'flags': 0, 'expiration': 1401215452}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947051596957, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne896 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:28
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:29
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947038394899, Destination cas:4562947038394898, Error Count:30
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947038394899, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947038394898, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne972 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:31
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:32
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947037791994, Destination cas:4562947037791993, Error Count:33
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947037791994, 'flags': 0, 'expiration': 1401215452}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947037791993, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne805 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:34
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:35
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947037081618, Destination cas:4562947037081617, Error Count:36
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947037081618, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947037081617, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne720 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:37
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:38
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947034962560, Destination cas:4562947034962559, Error Count:39
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947034962560, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947034962559, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne717 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:40
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:41
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947038004570, Destination cas:4562947038004569, Error Count:42
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947038004570, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947038004569, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne980 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:43
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:44
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947047110994, Destination cas:4562947047110993, Error Count:45
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947047110994, 'flags': 0, 'expiration': 1401215452}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947047110993, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne864 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:46
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:47
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947051233069, Destination cas:4562947051233068, Error Count:48
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947051233069, 'flags': 0, 'expiration': 1401215452}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947051233068, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne853 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:49
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:50
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947047624566, Destination cas:4562947047624565, Error Count:51
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947047624566, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947047624565, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne741 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:52
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:53
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947041380736, Destination cas:4562947041380735, Error Count:54
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947041380736, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947041380735, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne832 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:55
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:56
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947035292464, Destination cas:4562947035292463, Error Count:57
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947035292464, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947035292463, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne769 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:58
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:59
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947048593869, Destination cas:4562947048593868, Error Count:60
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947048593869, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947048593868, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne708 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:61
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:62
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947044272223, Destination cas:4562947044272222, Error Count:63
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947044272223, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947044272222, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne815 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:64
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:65
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947043234311, Destination cas:4562947043234310, Error Count:66
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947043234311, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947043234310, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne730 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:67
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:68
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947049453109, Destination cas:4562947049453108, Error Count:69
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947049453109, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947049453108, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne843 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:70
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:71
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947041016346, Destination cas:4562947041016345, Error Count:72
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947041016346, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947041016345, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne707 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:73
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:74
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947050857638, Destination cas:4562947050857637, Error Count:75
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947050857638, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947050857637, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne766 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:76
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:77
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947051608206, Destination cas:4562947051608205, Error Count:78
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947051608206, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947051608205, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne990 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:79
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:80
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947040882035, Destination cas:4562947040882034, Error Count:81
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947040882035, 'flags': 0, 'expiration': 1401215452}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947040882034, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne718 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:82
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:83
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947038383779, Destination cas:4562947038383778, Error Count:84
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947038383779, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947038383778, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne779 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:85
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:86
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947050523555, Destination cas:4562947050523554, Error Count:87
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947050523555, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947050523554, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne915 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:88
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:89
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947046032126, Destination cas:4562947046032125, Error Count:90
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947046032126, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947046032125, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne922 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:91
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:92
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947048650377, Destination cas:4562947048650376, Error Count:93
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947048650377, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947048650376, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne943 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:94
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:95
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947034705694, Destination cas:4562947034705693, Error Count:96
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947034705694, 'flags': 0, 'expiration': 1401215452}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947034705693, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne890 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:97
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:98
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947038315972, Destination cas:4562947038315971, Error Count:99
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947038315972, 'flags': 0, 'expiration': 1401215451}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947038315971, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] ===== Verifying rev_ids failed for key: loadOne974 =====
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:100
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:101
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] cas mismatch: Source cas:4562947037692074, Destination cas:4562947037692073, Error Count:102
      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Source meta data:

      {'deleted': 1, 'seqno': 2, 'cas': 4562947037692074, 'flags': 0, 'expiration': 1401215452}

      2014-05-27 11:40:03 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

      {'deleted': 0, 'seqno': 1, 'cas': 4562947037692073, 'flags': 0, 'expiration': 0}

      2014-05-27 11:40:03 | INFO | MainProcess | load_gen_task | [task.has_next] RevId Verification : 86 deleted items have been verified
      FAIL

      Attachments

        1. test_1 (2).zip
          5.84 MB
        2. test_1 (3).zip
          5.81 MB
        For Gerrit Dashboard: MB-11228
        # Subject Branch Project Status CR V
        37695,2 MB-11228: handle end_stream right after stream_request master ns_server Status: MERGED +2 +1
        37697,2 MB-11228: start LastSeenSeqno with StartSeqno master ns_server Status: MERGED +2 +1

        Activity

          People

            apiravi Aruna Piravi (Inactive)
            apiravi Aruna Piravi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty