[RC3-SSL-XDCR-Unidirectional] Deletions were not replicated to destination after warmup

[Jenkins]
http://qa.hq.northscale.net/job/centos_x64--31_02--uniXDCR_SSL-P1/47/consoleFull

[Test]
./testrunner i centos_x6431_01-uniXDCR-P1.ini GROUP=CHAIN,get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True,demand_encryption=1 -t xdcr.uniXDCR.unidirectional.load_with_async_ops_with_warmup,items=100000,rdirection=unidirection,ctopology=chain,sasl_buckets=1,doc-ops=delete,warm=source,replication_type=xmem,GROUP=P0;CHAIN;xmem

[Test Steps]
1. Setup 4-4 Node Uni-directional XMEM replication. SSL=True
2. Buckets: default, sasl_bucket_1
3. Load 1M items on each buckets.
4. Warmup One Source node (10.3.3.210).
5. Perform 30K deletions on Source cluster.
6. Verify items on each cluster.
a) Items on Source cluster is 70K on both the buckets which was expected.
b) Items on default of Destination cluster is 70K as expected.
c) Items mismatch on Destination Cluster "sasl_bucket_1". Expected: 70000 Actual: 70024.

There were Outbound mutations left on Source cluster's sasl_bucket_1:

[2014-09-05 20:04:50,113] - [xdcrbasetests:1318] INFO - Current outbound mutations on cluster node: 10.3.121.65 for bucket sasl_bucket_1 is 77
[2014-09-05 20:04:50,273] - [xdcrbasetests:375] INFO - sleep for 10 secs. ...
[2014-09-05 20:05:00,456] - [xdcrbasetests:1318] INFO - Current outbound mutations on cluster node: 10.3.121.65 for bucket sasl_bucket_1 is 77
[2014-09-05 20:05:00,637] - [xdcrbasetests:375] INFO - sleep for 10 secs. ...
[2014-09-05 20:05:10,830] - [xdcrbasetests:1318] INFO - Current outbound mutations on cluster node: 10.3.121.65 for bucket sasl_bucket_1 is 77
[2014-09-05 20:05:11,012] - [xdcrbasetests:375] INFO - sleep for 10 secs. ...
[2014-09-05 20:05:21,212] - [xdcrbasetests:1318] INFO - Current outbound mutations on cluster node: 10.3.121.65 for bucket sasl_bucket_1 is 77
[2014-09-05 20:05:21,370] - [xdcrbasetests:375] INFO - sleep for 10 secs. ...
[2014-09-05 20:05:31,579] - [xdcrbasetests:1318] INFO - Current outbound mutations on cluster node: 10.3.121.65 for bucket sasl_bucket_1 is 77
[2014-09-05 20:05:31,744] - [xdcrbasetests:375] INFO - sleep for 10 secs. ...
[2014-09-05 20:05:41,923] - [xdcrbasetests:1318] INFO - Current outbound mutations on cluster node: 10.3.121.65 for bucket sasl_bucket_1 is 77

Following Keys has meta data mis-match On below 24 keys: [' loadOne88725 ', ' loadOne86102 ', ' loadOne82917 ', ' loadOne83013 ', ' loadOne87806 ', ' loadOne85323 ', ' loadOne72828 ', ' loadOne80232 ', ' loadOne77939 ', ' loadOne80540 ', ' loadOne91549 ', ' loadOne85451 ', ' loadOne78168 ', ' loadOne94458 ', ' loadOne83761 ', ' loadOne83183 ', ' loadOne89953 ', ' loadOne92768 ', ' loadOne87996 ', ' loadOne88057 ', ' loadOne86092 ', ' loadOne86670 ', ' loadOne97679 ', ' loadOne82887 ']

2014-09-05 20:11:52,181] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne88725 =====
[2014-09-05 20:11:52,182] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9102037377909, 'flags': 0, 'expiration': 1409972374}

[2014-09-05 20:11:52,182] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9102037377908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:11:53,265] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne86102 =====
[2014-09-05 20:11:53,266] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9099987685909, 'flags': 0, 'expiration': 1409972374}

[2014-09-05 20:11:53,266] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9099987685908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:11:57,369] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne82917 =====
[2014-09-05 20:11:57,373] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9096223797909, 'flags': 0, 'expiration': 1409972341}

[2014-09-05 20:11:57,374] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9096223797908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:12:04,955] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne83013 =====
[2014-09-05 20:12:04,962] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9097160866909, 'flags': 0, 'expiration': 1409972341}

[2014-09-05 20:12:04,964] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9097160866908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:12:11,906] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne87806 =====
[2014-09-05 20:12:11,909] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9101036419909, 'flags': 0, 'expiration': 1409972374}

[2014-09-05 20:12:11,909] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9101036419908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:12:16,143] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne85323 =====
[2014-09-05 20:12:16,147] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9099078731909, 'flags': 0, 'expiration': 1409972374}

[2014-09-05 20:12:16,149] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9099078731908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:12:17,078] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne72828 =====
[2014-09-05 20:12:17,082] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9087763065909, 'flags': 0, 'expiration': 1409972309}

[2014-09-05 20:12:17,082] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9087763065908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:12:24,889] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne80232 =====
[2014-09-05 20:12:24,891] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9094688048909, 'flags': 0, 'expiration': 1409972341}

[2014-09-05 20:12:24,892] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9094688048908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:12:29,005] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne77939 =====
[2014-09-05 20:12:29,006] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9092194411909, 'flags': 0, 'expiration': 1409972341}

[2014-09-05 20:12:29,007] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9092194411908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:12:34,407] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne80540 =====
[2014-09-05 20:12:34,410] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9094737801909, 'flags': 0, 'expiration': 1409972341}

[2014-09-05 20:12:34,412] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9094737801908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:12:36,252] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne91549 =====
[2014-09-05 20:12:36,256] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9104651742909, 'flags': 0, 'expiration': 1409972374}

[2014-09-05 20:12:36,257] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9104651742908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:12:39,751] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne85451 =====
[2014-09-05 20:12:39,752] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9099043273909, 'flags': 0, 'expiration': 1409972374}

[2014-09-05 20:12:39,752] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9099043273908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:12:42,400] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne78168 =====
[2014-09-05 20:12:42,401] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9093103149909, 'flags': 0, 'expiration': 1409972341}

[2014-09-05 20:12:42,401] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9093103149908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:12:49,561] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne94458 =====
[2014-09-05 20:12:49,562] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9107323149909, 'flags': 0, 'expiration': 1409972374}

[2014-09-05 20:12:49,562] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9107323149908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:12:50,421] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne83761 =====
[2014-09-05 20:12:50,422] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9097159065909, 'flags': 0, 'expiration': 1409972341}

[2014-09-05 20:12:50,423] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9097159065908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:12:51,071] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne83183 =====
[2014-09-05 20:12:51,074] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9097139357909, 'flags': 0, 'expiration': 1409972341}

[2014-09-05 20:12:51,075] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9097139357908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:12:54,902] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne89953 =====
[2014-09-05 20:12:54,905] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9102751366909, 'flags': 0, 'expiration': 1409972374}

[2014-09-05 20:12:54,907] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9102751366908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:12:55,267] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne92768 =====
[2014-09-05 20:12:55,271] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9105516359909, 'flags': 0, 'expiration': 1409972374}

[2014-09-05 20:12:55,272] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9105516359908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:12:56,682] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne87996 =====
[2014-09-05 20:12:56,687] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9101032935909, 'flags': 0, 'expiration': 1409972374}

[2014-09-05 20:12:56,688] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9101032935908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:12:58,984] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne88057 =====
[2014-09-05 20:12:58,989] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9102011877909, 'flags': 0, 'expiration': 1409972374}

[2014-09-05 20:12:58,990] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9102011877908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:12:59,679] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne86092 =====
[2014-09-05 20:12:59,683] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9099988888909, 'flags': 0, 'expiration': 1409972374}

[2014-09-05 20:12:59,684] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9099988888908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:12:59,746] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne86670 =====
[2014-09-05 20:12:59,750] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9099972218909, 'flags': 0, 'expiration': 1409972374}

[2014-09-05 20:12:59,752] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9099972218908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:13:06,284] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne97679 =====
[2014-09-05 20:13:06,291] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9109372769909, 'flags': 0, 'expiration': 1409972410}

[2014-09-05 20:13:06,293] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9109372769908, 'flags': 0, 'expiration': 0}

[2014-09-05 20:13:09,311] - [task:1228] ERROR - ===== Verifying rev_ids failed for key: loadOne82887 =====
[2014-09-05 20:13:09,315] - [task:1230] ERROR - Source meta data:

{'deleted': 1, 'seqno': 2, 'cas': 9096243219909, 'flags': 0, 'expiration': 1409972341}

[2014-09-05 20:13:09,315] - [task:1231] ERROR - Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 9096243219908, 'flags': 0, 'expiration': 0}

All the keys belongs to the Warmed-up node i.e. 10.3.3.210, vbucket=815. Except loadOne85323.

Test is passed on non-ssl XDCR.