KV+ XDCR System test: Missing items in bi-xdcr only

Build
-------
3.0.0-998

Clusters
-----------
C1 : http://172.23.105.44:8091/
C2 : http://172.23.105.54:8091/
Free for investigation. Not attaching data files.

Steps
--------
1a. Load on both clusters till vb_active_resident_items_ratio < 50.
1b. Setup bi-xdcr on "standardbucket", uni-xdcr on "standardbucket1"
2. Access phase with 50% gets, 50%deletes for 3 hrs
3. Rebalance-out 1 node at cluster1
4. Rebalance-in 1 node at cluster1
5. Failover and remove node at cluster1
6. Failover and add-back node at cluster1
7. Rebalance-out 1 node at cluster2
8. Rebalance-in 1 node at cluster2
9. Failover and remove node at cluster2
10. Failover and add-back node at cluster2
11. Soft restart all nodes in cluster1 one by one
Verify item count

Problem
-------------
standardbucket(C1) <---> standardbucket(C2)
On C1 - 57890744 items
On C2 - 57957032 items
standardbucket1(C1) ----> standardbucket1(C2)
On C1 - 14053020 items
On C2 - 14053020 items

Total number of missing items : 66,288

Bucket priority
-----------------------
Both standardbucket and standardbucket1 have high priority.

Attached
-------------
cbcollect and list of keys that are missing on vb0

Missing keys
-------------------
Atleast 50-60 keys missing in every vbucket. Attaching all missing keys from vb0

vb0
-------

{'C1_node:': u'172.23.105.44', 'vb': 0, 'C2_node': u'172.23.105.54', 'C1_key_count': 78831, 'C2_key_count': 78929, 'missing_keys': 98}

id: 06FA8A8B-11_110 deleted, tombstone exists
id: 06FA8A8B-11_1354 present, report a bug!
id: 06FA8A8B-11_1426 present, report a bug!
id: 06FA8A8B-11_2175 present, report a bug!
id: 06FA8A8B-11_2607 present, report a bug!
id: 06FA8A8B-11_2797 present, report a bug!
id: 06FA8A8B-11_3871 deleted, tombstone exists
id: 06FA8A8B-11_4245 deleted, tombstone exists
id: 06FA8A8B-11_4537 present, report a bug!
id: 06FA8A8B-11_662 deleted, tombstone exists
id: 06FA8A8B-11_6960 present, report a bug!
id: 06FA8A8B-11_7064 present, report a bug!
id: 3600C830-80_1298 present, report a bug!
id: 3600C830-80_1308 present, report a bug!
id: 3600C830-80_2129 present, report a bug!
id: 3600C830-80_4219 deleted, tombstone exists
id: 3600C830-80_4389 deleted, tombstone exists
id: 3600C830-80_7038 present, report a bug!
id: 3FEF1B93-91_2890 present, report a bug!
id: 3FEF1B93-91_2900 present, report a bug!
id: 3FEF1B93-91_3004 present, report a bug!
id: 3FEF1B93-91_3194 present, report a bug!
id: 3FEF1B93-91_3776 deleted, tombstone exists
id: 3FEF1B93-91_753 present, report a bug!
id: 52D6D916-120_1837 present, report a bug!
id: 52D6D916-120_3282 present, report a bug!
id: 52D6D916-120_3312 present, report a bug!
id: 52D6D916-120_3460 present, report a bug!
id: 52D6D916-120_376 deleted, tombstone exists
id: 52D6D916-120_404 deleted, tombstone exists
id: 52D6D916-120_4926 present, report a bug!
id: 52D6D916-120_5022 present, report a bug!
id: 52D6D916-120_5750 present, report a bug!
id: 52D6D916-120_594 deleted, tombstone exists
id: 52D6D916-120_6203 present, report a bug!
id: 5C12B75A-142_2889 present, report a bug!
id: 5C12B75A-142_2919 present, report a bug!
id: 5C12B75A-142_569 deleted, tombstone exists
id: 73C89FDB-102_1013 present, report a bug!
id: 73C89FDB-102_1183 present, report a bug!
id: 73C89FDB-102_1761 present, report a bug!
id: 73C89FDB-102_2232 present, report a bug!
id: 73C89FDB-102_2540 present, report a bug!
id: 73C89FDB-102_4092 deleted, tombstone exists
id: 73C89FDB-102_4102 deleted, tombstone exists
id: 73C89FDB-102_668 deleted, tombstone exists
id: 87B03DB1-62_3369 present, report a bug!
id: 8DA39D2B-131_1949 present, report a bug!
id: 8DA39D2B-131_725 deleted, tombstone exists
id: A2CC835C-00_2926 present, report a bug!
id: A2CC835C-00_3022 present, report a bug!
id: A2CC835C-00_3750 present, report a bug!
id: A2CC835C-00_5282 present, report a bug!
id: A2CC835C-00_5312 present, report a bug!
id: A2CC835C-00_5460 present, report a bug!
id: A2CC835C-00_6133 present, report a bug!
id: A2CC835C-00_6641 present, report a bug!
id: A5C9F867-33_1091 present, report a bug!
id: A5C9F867-33_1101 present, report a bug!
id: A5C9F867-33_1673 present, report a bug!
id: A5C9F867-33_2320 present, report a bug!
id: A5C9F867-33_2452 present, report a bug!
id: A5C9F867-33_4010 deleted, tombstone exists
id: A5C9F867-33_4180 deleted, tombstone exists
id: CD7B0436-153_3638 present, report a bug!
id: CD7B0436-153_828 present, report a bug!
id: D94DA3B2-51_829 present, report a bug!
id: DE161E9D-40_1235 present, report a bug!
id: DE161E9D-40_1547 present, report a bug!
id: DE161E9D-40_2014 present, report a bug!
id: DE161E9D-40_2184 present, report a bug!
id: DE161E9D-40_2766 present, report a bug!
id: DE161E9D-40_3880 deleted, tombstone exists
id: DE161E9D-40_3910 deleted, tombstone exists
id: DE161E9D-40_4324 deleted, tombstone exists
id: DE161E9D-40_4456 deleted, tombstone exists
id: DE161E9D-40_6801 present, report a bug!
id: DE161E9D-40_6991 present, report a bug!
id: DE161E9D-40_7095 present, report a bug!
id: DE161E9D-40_7105 present, report a bug!
id: DE161E9D-40_940 present, report a bug!
id: E9F46ECC-22_173 deleted, tombstone exists
id: E9F46ECC-22_2883 present, report a bug!
id: E9F46ECC-22_2913 present, report a bug!
id: E9F46ECC-22_3017 present, report a bug!
id: E9F46ECC-22_3187 present, report a bug!
id: E9F46ECC-22_3765 deleted, tombstone exists
id: E9F46ECC-22_5327 present, report a bug!
id: E9F46ECC-22_5455 present, report a bug!
id: E9F46ECC-22_601 deleted, tombstone exists
id: E9F46ECC-22_6096 present, report a bug!
id: E9F46ECC-22_6106 present, report a bug!
id: E9F46ECC-22_6674 present, report a bug!
id: E9F46ECC-22_791 present, report a bug!
id: ECD6BE16-113_2961 present, report a bug!
id: ECD6BE16-113_3065 present, report a bug!
id: ECD6BE16-113_3687 present, report a bug!
id: ECD6BE16-113_3717 present, report a bug!

74 undeleted key(s) present on C2(.54) compared to C1(.44)