Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Major
Fix Version/s: 4.5.0
Affects Version/s: 4.5.0
Component/s: XDCR
Security Level: Public
Labels:
None

Triage:
Untriaged
Is this a Regression?:
Yes

Description

./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True,fail_on_errors=1,GROUP=ALL -t xdcr.rebalanceXDCR.Rebalance.async_rebalance_in,items=100000,rdirection=unidirection,ctopology=chain,update=C1,delete=C1,expires=60,rebalance=C2,num_rebalance=2,GROUP=P0;xmem

This test case used to pass in Sherlock - now it is failing with following error:

[2016-03-15 11:55:50,172] - [task:1629] INFO - RevID verification: in progress for default ...

[2016-03-15 11:56:38,474] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-1911, bucket:default =====

[2016-03-15 11:56:38,474] - [task:1724] ERROR - cas mismatch: Source cas:1458067671463821314, Destination cas:1458067934317117440, Error Count:1

[2016-03-15 11:56:38,474] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671463821314, 'flags': 0, 'expiration': 1458067671}

[2016-03-15 11:56:38,474] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067934317117440, 'flags': 0, 'expiration': 1458067934}

[2016-03-15 11:56:38,505] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-18076, bucket:default =====

[2016-03-15 11:56:38,505] - [task:1724] ERROR - cas mismatch: Source cas:1458067671478239234, Destination cas:1458067940976820224, Error Count:2

[2016-03-15 11:56:38,506] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671478239234, 'flags': 0, 'expiration': 1458067671}

[2016-03-15 11:56:38,506] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067940976820224, 'flags': 0, 'expiration': 1458067940}

[2016-03-15 11:56:38,533] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-14571, bucket:default =====

[2016-03-15 11:56:38,533] - [task:1724] ERROR - cas mismatch: Source cas:1458067671570841606, Destination cas:1458067939315875841, Error Count:3

[2016-03-15 11:56:38,533] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671570841606, 'flags': 0, 'expiration': 1458067673}

[2016-03-15 11:56:38,533] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067939315875841, 'flags': 0, 'expiration': 1458067939}

[2016-03-15 11:56:38,540] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-25612, bucket:default =====

[2016-03-15 11:56:38,540] - [task:1724] ERROR - cas mismatch: Source cas:1458067671592337408, Destination cas:1458067943152877569, Error Count:4

[2016-03-15 11:56:38,540] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671592337408, 'flags': 0, 'expiration': 1458067671}

[2016-03-15 11:56:38,540] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067943152877569, 'flags': 0, 'expiration': 1458067942}

[2016-03-15 11:56:38,596] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-27077, bucket:default =====

[2016-03-15 11:56:38,596] - [task:1724] ERROR - cas mismatch: Source cas:1458067671442587650, Destination cas:1458067944190902272, Error Count:5

[2016-03-15 11:56:38,597] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671442587650, 'flags': 0, 'expiration': 1458067671}

[2016-03-15 11:56:38,597] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067944190902272, 'flags': 0, 'expiration': 1458067944}

[2016-03-15 11:56:38,616] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-12933, bucket:default =====

[2016-03-15 11:56:38,616] - [task:1724] ERROR - cas mismatch: Source cas:1458067671610425348, Destination cas:1458067938544451584, Error Count:6

[2016-03-15 11:56:38,616] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671610425348, 'flags': 0, 'expiration': 1458067671}

[2016-03-15 11:56:38,616] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067938544451584, 'flags': 0, 'expiration': 1458067938}

[2016-03-15 11:56:38,677] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-28925, bucket:default =====

[2016-03-15 11:56:38,677] - [task:1724] ERROR - cas mismatch: Source cas:1458067671477977093, Destination cas:1458067944512290816, Error Count:7

[2016-03-15 11:56:38,677] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671477977093, 'flags': 0, 'expiration': 1458067671}

[2016-03-15 11:56:38,677] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067944512290816, 'flags': 0, 'expiration': 1458067944}

[2016-03-15 11:56:38,690] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-14401, bucket:default =====

[2016-03-15 11:56:38,690] - [task:1724] ERROR - cas mismatch: Source cas:1458067671444160525, Destination cas:1458067939378003968, Error Count:8

[2016-03-15 11:56:38,690] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671444160525, 'flags': 0, 'expiration': 1458067671}

[2016-03-15 11:56:38,690] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067939378003968, 'flags': 0, 'expiration': 1458067939}

[2016-03-15 11:56:38,697] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-13684, bucket:default =====

[2016-03-15 11:56:38,697] - [task:1724] ERROR - cas mismatch: Source cas:1458067671492001801, Destination cas:1458067938996125696, Error Count:9

[2016-03-15 11:56:38,697] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671492001801, 'flags': 0, 'expiration': 1458067673}

[2016-03-15 11:56:38,698] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067938996125696, 'flags': 0, 'expiration': 1458067939}

[2016-03-15 11:56:38,703] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-25509, bucket:default =====

[2016-03-15 11:56:38,703] - [task:1724] ERROR - cas mismatch: Source cas:1458067671455498241, Destination cas:1458067943594328064, Error Count:10

[2016-03-15 11:56:38,703] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671455498241, 'flags': 0, 'expiration': 1458067671}

[2016-03-15 11:56:38,703] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067943594328064, 'flags': 0, 'expiration': 1458067943}

[2016-03-15 11:56:38,785] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-7040, bucket:default =====

[2016-03-15 11:56:38,785] - [task:1724] ERROR - cas mismatch: Source cas:1458067671511007234, Destination cas:1458067936436289536, Error Count:11

[2016-03-15 11:56:38,786] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671511007234, 'flags': 0, 'expiration': 1458067671}

[2016-03-15 11:56:38,786] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067936436289536, 'flags': 0, 'expiration': 1458067936}

[2016-03-15 11:56:38,797] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-24817, bucket:default =====

[2016-03-15 11:56:38,797] - [task:1724] ERROR - cas mismatch: Source cas:1458067671511465993, Destination cas:1458067942778011650, Error Count:12

[2016-03-15 11:56:38,798] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671511465993, 'flags': 0, 'expiration': 1458067673}

[2016-03-15 11:56:38,798] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067942778011650, 'flags': 0, 'expiration': 1458067942}

[2016-03-15 11:56:38,809] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-29980, bucket:default =====

[2016-03-15 11:56:38,809] - [task:1724] ERROR - cas mismatch: Source cas:1458067671518871563, Destination cas:1458067944854585345, Error Count:13

[2016-03-15 11:56:38,810] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671518871563, 'flags': 0, 'expiration': 1458067673}

[2016-03-15 11:56:38,810] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067944854585345, 'flags': 0, 'expiration': 1458067944}

[2016-03-15 11:56:38,813] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-12242, bucket:default =====

[2016-03-15 11:56:38,813] - [task:1724] ERROR - cas mismatch: Source cas:1458067671554392069, Destination cas:1458067938416984064, Error Count:14

[2016-03-15 11:56:38,813] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671554392069, 'flags': 0, 'expiration': 1458067671}

[2016-03-15 11:56:38,813] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067938416984064, 'flags': 0, 'expiration': 1458067938}

[2016-03-15 11:56:38,868] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-27107, bucket:default =====

[2016-03-15 11:56:38,868] - [task:1724] ERROR - cas mismatch: Source cas:1458067671565860872, Destination cas:1458067944250802176, Error Count:15

[2016-03-15 11:56:38,868] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671565860872, 'flags': 0, 'expiration': 1458067673}

[2016-03-15 11:56:38,868] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067944250802176, 'flags': 0, 'expiration': 1458067944}

[2016-03-15 11:56:38,876] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-11911, bucket:default =====

[2016-03-15 11:56:38,877] - [task:1724] ERROR - cas mismatch: Source cas:1458067671489839108, Destination cas:1458067938190032896, Error Count:16

[2016-03-15 11:56:38,877] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671489839108, 'flags': 0, 'expiration': 1458067673}

[2016-03-15 11:56:38,877] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067938190032896, 'flags': 0, 'expiration': 1458067937}

[2016-03-15 11:56:38,939] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-12332, bucket:default =====

[2016-03-15 11:56:38,939] - [task:1724] ERROR - cas mismatch: Source cas:1458067671478435852, Destination cas:1458067938563784704, Error Count:17

[2016-03-15 11:56:38,940] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671478435852, 'flags': 0, 'expiration': 1458067673}

[2016-03-15 11:56:38,940] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067938563784704, 'flags': 0, 'expiration': 1458067938}

[2016-03-15 11:56:38,959] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-19853, bucket:default =====

[2016-03-15 11:56:38,960] - [task:1724] ERROR - cas mismatch: Source cas:1458067671456940040, Destination cas:1458067941328289792, Error Count:18

[2016-03-15 11:56:38,960] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671456940040, 'flags': 0, 'expiration': 1458067671}

[2016-03-15 11:56:38,960] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067941328289792, 'flags': 0, 'expiration': 1458067941}

[2016-03-15 11:56:38,984] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-29678, bucket:default =====

[2016-03-15 11:56:38,984] - [task:1724] ERROR - cas mismatch: Source cas:1458067671502159876, Destination cas:1458067945018359808, Error Count:19

[2016-03-15 11:56:38,985] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671502159876, 'flags': 0, 'expiration': 1458067671}

[2016-03-15 11:56:38,985] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067945018359808, 'flags': 0, 'expiration': 1458067944}

[2016-03-15 11:56:39,005] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-2095, bucket:default =====

[2016-03-15 11:56:39,006] - [task:1724] ERROR - cas mismatch: Source cas:1458067671520247818, Destination cas:1458067934604754944, Error Count:20

[2016-03-15 11:56:39,006] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671520247818, 'flags': 0, 'expiration': 1458067673}

[2016-03-15 11:56:39,006] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067934604754944, 'flags': 0, 'expiration': 1458067934}

[2016-03-15 11:56:39,035] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-21257, bucket:default =====

[2016-03-15 11:56:39,035] - [task:1724] ERROR - cas mismatch: Source cas:1458067671472603145, Destination cas:1458067941714558976, Error Count:21

[2016-03-15 11:56:39,036] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671472603145, 'flags': 0, 'expiration': 1458067671}

[2016-03-15 11:56:39,036] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067941714558976, 'flags': 0, 'expiration': 1458067941}

[2016-03-15 11:56:39,044] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-20025, bucket:default =====

[2016-03-15 11:56:39,044] - [task:1724] ERROR - cas mismatch: Source cas:1458067671580999680, Destination cas:1458067941627789312, Error Count:22

[2016-03-15 11:56:39,044] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671580999680, 'flags': 0, 'expiration': 1458067673}

[2016-03-15 11:56:39,045] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067941627789312, 'flags': 0, 'expiration': 1458067941}

[2016-03-15 11:56:39,080] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-8313, bucket:default =====

[2016-03-15 11:56:39,080] - [task:1724] ERROR - cas mismatch: Source cas:1458067671579688964, Destination cas:1458067936771178496, Error Count:23

[2016-03-15 11:56:39,080] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671579688964, 'flags': 0, 'expiration': 1458067673}

[2016-03-15 11:56:39,080] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067936771178496, 'flags': 0, 'expiration': 1458067936}

[2016-03-15 11:56:39,084] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-8862, bucket:default =====

[2016-03-15 11:56:39,084] - [task:1724] ERROR - cas mismatch: Source cas:1458067671584604164, Destination cas:1458067936956055552, Error Count:24

[2016-03-15 11:56:39,084] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671584604164, 'flags': 0, 'expiration': 1458067673}

[2016-03-15 11:56:39,084] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067936956055552, 'flags': 0, 'expiration': 1458067936}

[2016-03-15 11:56:39,099] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-16101, bucket:default =====

[2016-03-15 11:56:39,100] - [task:1724] ERROR - cas mismatch: Source cas:1458067671485644806, Destination cas:1458067940123017216, Error Count:25

[2016-03-15 11:56:39,100] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671485644806, 'flags': 0, 'expiration': 1458067671}

[2016-03-15 11:56:39,100] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067940123017216, 'flags': 0, 'expiration': 1458067940}

[2016-03-15 11:56:39,109] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-4379, bucket:default =====

[2016-03-15 11:56:39,109] - [task:1724] ERROR - cas mismatch: Source cas:1458067671523196935, Destination cas:1458067935302909952, Error Count:26

[2016-03-15 11:56:39,109] - [task:1725] ERROR - Source meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067671523196935, 'flags': 0, 'expiration': 1458067673}

[2016-03-15 11:56:39,109] - [task:1726] ERROR - Dest meta data: {'deleted': 1, 'seqno': 3, 'cas': 1458067935302909952, 'flags': 0, 'expiration': 1458067935}

[2016-03-15 11:56:39,118] - [task:1723] ERROR - ===== Verifying rev_ids failed for key: C1-key-29070, bucket:default =====

.....

Similar bug was filed before and it was resolved by increasing the expires time limit but we cannot do so in this testcase as they are designed to run with the specified time limit - attaching logs

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

consoleFull.rtf
22/Apr/16 1:23 AM
299 kB
Arunkumar Senthilnathan
Hide
172.23.121.164-20160315-1212-diag.zip
18/Mar/16 5:14 AM
5.44 MB
Arunkumar Senthilnathan
Extracting archive...
Show
172.23.121.164-20160315-1212-diag.zip
18/Mar/16 5:14 AM
5.44 MB
Arunkumar Senthilnathan
Hide
172.23.121.163-20160315-1211-diag.zip
18/Mar/16 5:14 AM
1.55 MB
Arunkumar Senthilnathan
Extracting archive...
Show
172.23.121.163-20160315-1211-diag.zip
18/Mar/16 5:14 AM
1.55 MB
Arunkumar Senthilnathan
Hide
172.23.121.162-20160315-1210-diag.zip
18/Mar/16 5:14 AM
760 kB
Arunkumar Senthilnathan
Extracting archive...
Show
172.23.121.162-20160315-1210-diag.zip
18/Mar/16 5:14 AM
760 kB
Arunkumar Senthilnathan
Hide
172.23.121.161-20160315-1208-diag.zip
18/Mar/16 5:14 AM
764 kB
Arunkumar Senthilnathan
Extracting archive...
Show
172.23.121.161-20160315-1208-diag.zip
18/Mar/16 5:14 AM
764 kB
Arunkumar Senthilnathan
Hide
172.23.107.161-20160315-1206-diag.zip
18/Mar/16 5:14 AM
4.96 MB
Arunkumar Senthilnathan
Extracting archive...
Show
172.23.107.161-20160315-1206-diag.zip
18/Mar/16 5:14 AM
4.96 MB
Arunkumar Senthilnathan
Hide
172.23.106.251-20160315-1201-diag.zip
18/Mar/16 5:14 AM
4.11 MB
Arunkumar Senthilnathan
Extracting archive...
Show
172.23.106.251-20160315-1201-diag.zip
18/Mar/16 5:14 AM
4.11 MB
Arunkumar Senthilnathan
Hide
172.23.106.115-20160315-1203-diag.zip
18/Mar/16 5:14 AM
7.36 MB
Arunkumar Senthilnathan
Extracting archive...
Show
172.23.106.115-20160315-1203-diag.zip
18/Mar/16 5:14 AM
7.36 MB
Arunkumar Senthilnathan
Hide
172.23.105.172-20160315-1158-diag.zip
18/Mar/16 5:14 AM
5.26 MB
Arunkumar Senthilnathan
Extracting archive...
Show
172.23.105.172-20160315-1158-diag.zip
18/Mar/16 5:14 AM
5.26 MB
Arunkumar Senthilnathan

Issue Links

duplicates

MB-18920 RevID verification fails for rebalance XDCR

Closed

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

People

Assignee:: Arunkumar Senthilnathan (Inactive)

Reporter:: Arunkumar Senthilnathan (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 18/Mar/16 5:14 AM

Updated:: 25/Apr/16 10:12 PM

Resolved:: 25/Apr/16 10:43 AM

Gerrit Reviews

There are no open Gerrit changes

[XDCR] revid verification failure due to cas mismatch

Details

Description

Attachments

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty