Mismatch on Expiry flags on expired-update items during unidirectional replication from source to destination cluster.

Setup
-----------
1. Create 2:2 node clusters, and setup a unidirecitonal replication from source to destination cluster.
2. Load 1000 items on source cluster, expect 1000 items are replicated to destination cluster.
3. Update first 300 items on source as update ( key, expires=10), expect the updates to be replicated to destination cluster.
4. Delete last 300 items on source as delete (key), expect the deletes to be replicated to destination cluster.
5. Verify that we have (1000 minus 300 minus 300) 400 items on source and 400 items on destination.

Output
----------
1. Verified 400 items on source, and 400 items on destination after xdcr replication is complete.
2. Verifying meta information of deleted/expired keys on source and destination - ERROR

Mismatch on the Expiry Flags.

Source Node has expired flags set to value say "134567"
Destination Node has expired flags set to value "0"

The expired items on replication are not replicating the expected expiry-value to the destination.

We would expect the same seqNo/CAS/expiry/other_flag meta information for same key on both source/destination.
Here it looks like the expired item is treated differently on the source v/s during xdcr-replication.

Chiyoung, could you take a look at this? Please let me know if you need additional information.

----------------------------------------
Command for VMs:
----------------------------------------
./testrunner -i myIni.ini -t xdcr.uniXDCR.unidirectional.load_with_ops,items=1000,rdirection=unidirection,ctopology=chain,doc-ops=create-update-delete,expires=10
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Output for Vms: - All expired keys(300) show mismatch on Expiry Flag values at source/ destination.
----------------------------------------------------------------------------------------------------------------------------------------------------------------
[2012-08-03 18:40:31,441] - [task:609] ERROR - Mismatch on Expiry Flags for key loadOne216 Source Expiry Flags:1344044058 Destination Expiry Flags:0 Error Count286
[2012-08-03 18:40:31,447] - [task:609] ERROR - Mismatch on Expiry Flags for key loadOne144 Source Expiry Flags:1344044058 Destination Expiry Flags:0 Error Count287
[2012-08-03 18:40:31,454] - [task:609] ERROR - Mismatch on Expiry Flags for key loadOne268 Source Expiry Flags:1344044058 Destination Expiry Flags:0 Error Count288
[2012-08-03 18:40:31,457] - [task:609] ERROR - Mismatch on Expiry Flags for key loadOne85 Source Expiry Flags:1344044058 Destination Expiry Flags:0 Error Count289
[2012-08-03 18:40:31,460] - [task:609] ERROR - Mismatch on Expiry Flags for key loadOne221 Source Expiry Flags:1344044058 Destination Expiry Flags:0 Error Count290
[2012-08-03 18:40:31,463] - [task:609] ERROR - Mismatch on Expiry Flags for key loadOne240 Source Expiry Flags:1344044058 Destination Expiry Flags:0 Error Count291
[2012-08-03 18:40:31,466] - [task:609] ERROR - Mismatch on Expiry Flags for key loadOne26 Source Expiry Flags:1344044058 Destination Expiry Flags:0 Error Count292
[2012-08-03 18:40:31,469] - [task:609] ERROR - Mismatch on Expiry Flags for key loadOne47 Source Expiry Flags:1344044058 Destination Expiry Flags:0 Error Count293
[2012-08-03 18:40:31,505] - [task:609] ERROR - Mismatch on Expiry Flags for key loadOne199 Source Expiry Flags:1344044058 Destination Expiry Flags:0 Error Count294
[2012-08-03 18:40:31,508] - [task:609] ERROR - Mismatch on Expiry Flags for key loadOne173 Source Expiry Flags:1344044058 Destination Expiry Flags:0 Error Count295
[2012-08-03 18:40:31,511] - [task:609] ERROR - Mismatch on Expiry Flags for key loadOne70 Source Expiry Flags:1344044058 Destination Expiry Flags:0 Error Count296
[2012-08-03 18:40:31,515] - [task:609] ERROR - Mismatch on Expiry Flags for key loadOne277 Source Expiry Flags:1344044058 Destination Expiry Flags:0 Error Count297
[2012-08-03 18:40:31,528] - [task:609] ERROR - Mismatch on Expiry Flags for key loadOne39 Source Expiry Flags:1344044058 Destination Expiry Flags:0 Error Count298
[2012-08-03 18:40:31,531] - [task:609] ERROR - Mismatch on Expiry Flags for key loadOne11 Source Expiry Flags:1344044058 Destination Expiry Flags:0 Error Count299
[2012-08-03 18:40:31,534] - [task:609] ERROR - Mismatch on Expiry Flags for key loadOne58 Source Expiry Flags:1344044058 Destination Expiry Flags:0 Error Count300
[2012-08-03 18:40:31,538] - [xdcrbasetests:97] INFO - ============== XDCRbasetests cleanup was started for test #1 load_with_ops ==============

Also, quick note.
I ran this on cluster_run on a 1:1 node cluster setup, unidirectional and I see CAS value mismatches.

----------------------------------------
Command for cluster_run:
----------------------------------------
./testrunner -i b/resources/dev-2-nodes-xdcr.ini -t xdcr.uniXDCR.unidirectional.load_with_ops,items=1000,rdirection=unidirection,ctopology=chain,doc-ops=create-update-delete,expires=1
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Output for cluster_run: - All expired keys(300) show mismatch on CAS values at source/ destination.
----------------------------------------------------------------------------------------------------------------------------------------------------------------
[2012-08-03 18:06:11,332] - [task:602] ERROR - Mismatch on CAS for key loadOne268 Source CAS:28313554348 Destination CAS:28313554347 Error Count288
[2012-08-03 18:06:11,333] - [task:602] ERROR - Mismatch on CAS for key loadOne85 Source CAS:28283106263 Destination CAS:28283106262 Error Count289
[2012-08-03 18:06:11,334] - [task:602] ERROR - Mismatch on CAS for key loadOne221 Source CAS:28306383045 Destination CAS:28306383044 Error Count290
[2012-08-03 18:06:11,334] - [task:602] ERROR - Mismatch on CAS for key loadOne240 Source CAS:28309302914 Destination CAS:28309302913 Error Count291
[2012-08-03 18:06:11,335] - [task:602] ERROR - Mismatch on CAS for key loadOne26 Source CAS:28273028827 Destination CAS:28273028826 Error Count292
[2012-08-03 18:06:11,336] - [task:602] ERROR - Mismatch on CAS for key loadOne47 Source CAS:28276334675 Destination CAS:28276334674 Error Count293
[2012-08-03 18:06:11,341] - [task:602] ERROR - Mismatch on CAS for key loadOne199 Source CAS:28302802276 Destination CAS:28302802275 Error Count294
[2012-08-03 18:06:11,342] - [task:602] ERROR - Mismatch on CAS for key loadOne173 Source CAS:28298182694 Destination CAS:28298182693 Error Count295
[2012-08-03 18:06:11,343] - [task:602] ERROR - Mismatch on CAS for key loadOne70 Source CAS:28280607764 Destination CAS:28280607763 Error Count296
[2012-08-03 18:06:11,343] - [task:602] ERROR - Mismatch on CAS for key loadOne277 Source CAS:28315209310 Destination CAS:28315209309 Error Count297
[2012-08-03 18:06:11,346] - [task:602] ERROR - Mismatch on CAS for key loadOne39 Source CAS:28274914700 Destination CAS:28274914699 Error Count298
[2012-08-03 18:06:11,347] - [task:602] ERROR - Mismatch on CAS for key loadOne11 Source CAS:28270902550 Destination CAS:28270902549 Error Count299
[2012-08-03 18:06:11,348] - [task:602] ERROR - Mismatch on CAS for key loadOne58 Source CAS:28278455963 Destination CAS:28278455962 Error Count300
[2012-08-03 18:06:11,349] - [xdcrbasetests:97] INFO - ============== XDCRbasetests cleanup was started for test #1 load_with_ops ==============