Details
Description
http://qa.hq.northscale.net/job/centos_x64--01_01--uniXDCR_biXDCR-P0/12/consoleFull
[Test]
./testrunner i centos_x6401_01-uniXDCR_biXDCR-P0.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True -t xdcr.biXDCR.bidirectional.load_with_async_ops_and_joint_sets,items=5000,ctopology=chain,rdirection=bidirection,standard_buckets=2,default_bucket=False,doc-ops=create-update,doc-ops-dest=create-update,upd=30,GROUP=P1
[Test Error]
======================================================================
ERROR: load_with_async_ops_and_joint_sets (xdcr.biXDCR.bidirectional)
----------------------------------------------------------------------
Traceback (most recent call last):
File "pytests/xdcr/biXDCR.py", line 92, in load_with_async_ops_and_joint_sets
self.verify_results(verify_src=True)
File "pytests/xdcr/xdcrbasetests.py", line 1420, in verify_results
self.verify_xdcr_stats(src_nodes, dest_nodes, verify_src)
File "pytests/xdcr/xdcrbasetests.py", line 1385, in verify_xdcr_stats
self._verify_data_all_buckets(self.src_master, max_verify=self.max_verify)
File "pytests/xdcr/xdcrbasetests.py", line 1301, in _verify_data_all_buckets
is_verified = self._poll_for_condition(verify)
File "pytests/xdcr/xdcrbasetests.py", line 938, in _poll_for_condition
return self._poll_for_condition_rec(condition, interval, num_itr)
File "pytests/xdcr/xdcrbasetests.py", line 944, in _poll_for_condition_rec
if condition():
File "pytests/xdcr/xdcrbasetests.py", line 1294, in verify
task.result(timeout)
File "lib/tasks/future.py", line 160, in result
return self.__get_result()
File "lib/tasks/future.py", line 112, in __get_result
raise self._exception
Exception: Key: loadOne749 Bad hash result: 6668 != 22556
----------------------------------------------------------------------
Ran 1 test in 351.090s
[Test Steps]
STEP 1. Source Nodes: 10.1.3.96, 10.1.3.97, 10.1.3.99, 10.1.2.12
Dest Nodes: 10.1.3.93, 10.1.3.94, 10.1.3.95
STEP 2. Create bucket: default
STEP 3. Setup CAPI or XMem based bidirectional xdcr between source and destination nodes.
STEP 4. Load 5000 unique items on each buckets at each cluster.
STEP 5. Perform 30% i.e. 1500 updates on Source cluster.
STEP 6. Perform 30% i.e. 1500 updates on Destination cluster for same items on Source.
STEP 7. Verify value of each key and value on either side. ---> Failed here.
[Test Execution Summary]
[ Step-4: Set on Source]
CREATE: Server: 10.1.3.96, bucket: default, key: loadOne749, value: loadOne749aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaWaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[Step-5: Update on Source]
UPDATE Server: 10.1.3.96, bucket: default, key: loadOne749, value: loadOne749aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaIaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaWaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[ Step-6: Update on Destination, Approx 3-4 seconds after update on Source]
UPDATE Server: 10.1.3.93, bucket: default, key: loadOne749, value: loadOne749aaaaaaaaaaaaaaaaaaaBaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaWaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[ Step-7: Read from Source Cluster] --> Failed here, test was expecting value of key should be the value updated on destination cluster, as it was done later i.e. on Step-6, while value is same as updated on Step-5 on Source.
READ Server: 10.1.3.96, bucket: default, key: loadOne749, value: loadOne749aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaIaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaWaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[Note]
Important thing is Value for Key loadOne49 is same on Source and Destination cluster eventually.
loadOne749aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaIaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaWaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Seems XDCR ignored the update on Destination and override the with changes on Source during replication.