Details
-
Bug
-
Resolution: Fixed
-
Critical
-
4.6.0, 5.0.0
-
Untriaged
-
Unknown
Description
Rebalance tests failing with the following errors, but quite intermittently.
07:15:25
|
07:15:25 ======================================================================
|
07:15:25 FAIL: rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests)
|
07:15:25 ----------------------------------------------------------------------
|
07:15:25 Traceback (most recent call last):
|
07:15:25 File "pytests/rebalance/rebalancein.py", line 171, in rebalance_in_with_ops
|
07:15:25 self.verify_cluster_stats(self.servers[:self.nodes_in + self.nodes_init])
|
07:15:25 File "pytests/basetestcase.py", line 891, in verify_cluster_stats
|
07:15:25 self.assertTrue(verified, "Lost items!!! Replication was completed but "
|
07:15:25 AssertionError: Lost items!!! Replication was completed but sum(curr_items) don't match the curr_items_total
|
07:15:25
|
07:15:25 ----------------------------------------------------------------------
|
Example test run: http://server.jenkins.couchbase.com/view/build-sanity/job/build_sanity_matrix/DISTRO=centos7,TYPE=4node/2109/
This is happening once in a while for both 4.6.0 and spock.
It is quite possible that this is an env issue. In at least 2 of the last 3 occasions, the main node (where all services are running) was 172.23.122.121. I have taken that node out of rotation and we can see how things go. But since the error message sounds like it is missing some items, it may be worthwhile to look at it, rather than dismissing it as an env issue. Or perhaps see if there is any process using excess resources that eventually causes the env issue.