Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
3.0
-
Security Level: Public
-
None
-
Build 3.0.0-614
-
Untriaged
-
Centos 64-bit
-
-
Unknown
-
June 30 - July 18
Description
[Job]
http://qa.hq.northscale.net/job/centos_x64--31_01--uniXDCR-P1/32/consoleFull
[Test]
./testrunner -i /tmp/ubuntu-64-2.0-uniXDCR.ini GROUP=CHAIN,num_items=50000,get-cbcollect-info=True -t xdcr.uniXDCR.unidirectional.load_with_async_ops,items=100000,rdirection=unidirection,ctopology=chain,expires=60,doc-ops=update-delete,replication_type=xmem,GROUP=CHAIN;P0;xmem
[Test Error] -> There are errors for 7 keys
[2014-04-26 10:35:57,973] - [task:1182] ERROR - ===== Verifying rev_ids failed for key: loadOne29446 =====
[2014-04-26 10:35:57,974] - [task:1183] ERROR - seqno mismatch: Source seqno:3, Destination seqno:4, Error Count:1
[2014-04-26 10:35:57,974] - [task:1183] ERROR - cas mismatch: Source cas:8592894078905619, Destination cas:8592894078905620, Error Count:2
[2014-04-26 10:35:57,975] - [task:1184] ERROR - Source meta data:
[2014-04-26 10:35:57,975] - [task:1185] ERROR - Dest meta data:
{'deleted': 1, 'seqno': 4, 'cas': 8592894078905620, 'flags': 0, 'expiration': 1398533141}[2014-04-26 10:36:09,680] - [task:1182] ERROR - ===== Verifying rev_ids failed for key: loadOne29578 =====
[2014-04-26 10:36:09,680] - [task:1183] ERROR - seqno mismatch: Source seqno:3, Destination seqno:4, Error Count:3
[2014-04-26 10:36:09,681] - [task:1183] ERROR - cas mismatch: Source cas:18463067062267569, Destination cas:18463067062267570, Error Count:4
[2014-04-26 10:36:09,682] - [task:1184] ERROR - Source meta data:
[2014-04-26 10:36:09,683] - [task:1185] ERROR - Dest meta data:
{'deleted': 1, 'seqno': 4, 'cas': 18463067062267570, 'flags': 0, 'expiration': 1398533141}[2014-04-26 10:36:19,864] - [task:1182] ERROR - ===== Verifying rev_ids failed for key: loadOne29451 =====
[2014-04-26 10:36:19,864] - [task:1183] ERROR - seqno mismatch: Source seqno:3, Destination seqno:4, Error Count:5
[2014-04-26 10:36:19,864] - [task:1183] ERROR - cas mismatch: Source cas:8950894458988185, Destination cas:8950894458988186, Error Count:6
[2014-04-26 10:36:19,864] - [task:1184] ERROR - Source meta data:
[2014-04-26 10:36:19,865] - [task:1185] ERROR - Dest meta data:
{'deleted': 1, 'seqno': 4, 'cas': 8950894458988186, 'flags': 0, 'expiration': 1398533140}[2014-04-26 10:36:41,956] - [task:1182] ERROR - ===== Verifying rev_ids failed for key: loadOne412 =====
[2014-04-26 10:36:41,956] - [task:1183] ERROR - seqno mismatch: Source seqno:3, Destination seqno:4, Error Count:7
[2014-04-26 10:36:41,957] - [task:1183] ERROR - cas mismatch: Source cas:18462956912623569, Destination cas:18462956912623570, Error Count:8
[2014-04-26 10:36:41,957] - [task:1184] ERROR - Source meta data:
[2014-04-26 10:36:41,958] - [task:1185] ERROR - Dest meta data:
{'deleted': 1, 'seqno': 4, 'cas': 18462956912623570, 'flags': 0, 'expiration': 1398532988}[2014-04-26 10:37:27,692] - [task:1182] ERROR - ===== Verifying rev_ids failed for key: loadOne29556 =====
[2014-04-26 10:37:27,693] - [task:1183] ERROR - seqno mismatch: Source seqno:3, Destination seqno:4, Error Count:9
[2014-04-26 10:37:27,693] - [task:1183] ERROR - cas mismatch: Source cas:8592894387719619, Destination cas:8592894387719620, Error Count:10
[2014-04-26 10:37:27,694] - [task:1184] ERROR - Source meta data:
[2014-04-26 10:37:27,694] - [task:1185] ERROR - Dest meta data:
{'deleted': 1, 'seqno': 4, 'cas': 8592894387719620, 'flags': 0, 'expiration': 1398533141}[2014-04-26 10:38:18,620] - [task:1182] ERROR - ===== Verifying rev_ids failed for key: loadOne582 =====
[2014-04-26 10:38:18,620] - [task:1183] ERROR - seqno mismatch: Source seqno:3, Destination seqno:4, Error Count:11
[2014-04-26 10:38:18,621] - [task:1183] ERROR - cas mismatch: Source cas:18462957665894569, Destination cas:18462957665894570, Error Count:12
[2014-04-26 10:38:18,621] - [task:1184] ERROR - Source meta data:
[2014-04-26 10:38:18,623] - [task:1185] ERROR - Dest meta data:
{'deleted': 1, 'seqno': 4, 'cas': 18462957665894570, 'flags': 0, 'expiration': 1398532988}[2014-04-26 10:38:22,957] - [task:1182] ERROR - ===== Verifying rev_ids failed for key: loadOne28739 =====
[2014-04-26 10:38:22,957] - [task:1183] ERROR - seqno mismatch: Source seqno:3, Destination seqno:4, Error Count:13
[2014-04-26 10:38:22,958] - [task:1183] ERROR - cas mismatch: Source cas:18463064250570569, Destination cas:18463064250570570, Error Count:14
[2014-04-26 10:38:22,959] - [task:1184] ERROR - Source meta data:
[2014-04-26 10:38:22,960] - [task:1185] ERROR - Dest meta data:
{'deleted': 1, 'seqno': 4, 'cas': 18463064250570570, 'flags': 0, 'expiration': 1398533141}[Test Steps]
1. Setup 4-4 Nodes SRC and DEST clusters with xmem XDCR.
2. Load 1M items on SRC nodes. (1 sasl buckets, 1 default bucket)
3. Perform 30% deletes.
4. Updates 30% items with expiration time to 60 seconds. Wait for expiration time.
5. There are two errors
a) Items count mismtach on destination side Expected 40000, Actual: 39993 (7 keys are missing as mentioned above).
b) seq and cas mismatch for 7 keys between Source and Destination
c) Why sequence number raised to 3 or 4, since updation is done only once? As per my understanding, it starts from 1 and increase by 1 for each update. Effectively it should be 2.
Data files and logs are uploaded.
Attachments
Issue Links
- duplicates
-
MB-10856 Persistence and internal replication(TAP/UPR) are broken , affects XDCR
- Closed