Stats inconsistent in Suse 11

Build

4.0.0-1841

Testcase
--------
./testrunner -i /tmp/four-opensuse-nodes.ini -t xdcr.biXDCR.bidirectional.load_with_async_ops,replicas=1,items=10000,ctopology=chain,rdirection=bidirection

Steps

1. C1[.84,.85] <--> C2 [.86,.87]
2. Load 10000 keys on C1 and C2
3. Verify keys

This cannot be reproduced consistently.

After waiting for 6 (3 for active and 3 for replica) mins, active, replica key count did not match expected value.

test log - https://friendpaste.com/68dIv86jV3Cp9EMUWDLbI0
in http://qa.sc.couchbase.com/job/suse11-p0-sanit-vset09-00/64/consoleFull

2015-04-13 07:50:02 | INFO | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Saw: vb_active_curr_items 20000 == 20000 expected on C1, default bucket
2015-04-13 07:50:02 | INFO | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Saw: vb_replica_curr_items 20000 == 20000 expected on C1, default bucket
2015-04-13 07:50:02 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:50:07 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:50:12 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:50:17 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:50:22 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:50:27 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:50:32 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:50:38 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:50:43 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:50:48 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:50:53 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:50:58 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:51:03 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:51:08 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:51:13 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:51:18 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:51:23 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:51:28 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:51:33 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:51:38 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:51:43 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:51:49 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:51:54 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:51:59 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:52:04 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:52:09 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:52:14 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:52:19 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:52:24 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:52:29 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:52:34 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:52:39 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:52:44 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:52:49 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:52:54 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:52:59 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_active_curr_items 3055 == 20000 expected on C2, default bucket
2015-04-13 07:53:04 | ERROR | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] ERROR: Timed-out waiting for active item count to match
2015-04-13 07:53:05 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:53:08 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:53:11 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:53:14 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:53:17 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:53:20 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:53:23 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:53:26 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:53:29 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:53:32 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:53:35 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:53:38 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:53:41 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:53:44 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:53:47 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:53:50 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:53:54 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:53:57 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:00 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:03 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:06 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:09 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:12 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:15 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:18 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:21 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:24 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:27 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:30 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:33 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:36 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:40 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:43 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:46 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:49 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:52 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:55 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:54:58 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:01 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:04 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:07 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:10 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:13 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:16 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:19 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:22 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:26 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:29 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:32 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:35 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:38 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:41 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:44 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:47 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:50 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:53 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:56 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:55:59 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket
2015-04-13 07:56:02 | WARNING | MainProcess | test_thread | [xdcrnewbasetests.verify_items_count] Not Ready: vb_replica_curr_items 3056 == 20000 expected on C2, default bucket

Attaching logs. Don't see anything suspicious in logs though. Both .84 and .85 seem to have replicated 5000 docs each.