Details
-
Bug
-
Resolution: Unresolved
-
Major
-
Cheshire-Cat
-
Untriaged
-
Centos 64-bit
-
1
-
Unknown
Description
It appears that under certain loads, ns server stats endpoints are returning aggregate numbers from each node but with some of the node numbers missing from the aggregate. This only happens for 100% updates, durability none, uniform distribution workload. The test uses 4 nodes with 2 replicas. We have other similar tests with higher durability levels that do not show this stats behaviour. Here is a test where you can see from cbmonitor graphs that the stats appear as 4 distinct bands.
http://showfast.sc.couchbase.com/#/timeline/Linux/kv/durability/Throughput
Avg Throughput (ops/sec), SDK 3, Durability None, 4 nodes, Uniform Distribution, Workload K
http://perf.jenkins.couchbase.com/job/hercules/5687/
http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hercules_700-3583_access_0548
Running manual check:
curl -u Administrator:password -v -X GET http://172.23.100.121:8091/pools/default/buckets/bucket-1/stats | jq '.op.samples.vb_replica_curr_items'
[
30000567,
30000567,
19996965,
19996965,
19996965,
30000567,
30000567,
20005160,
20005160,
30000567,
30000567,
30000567,
20000991,
20000991,
20000991,
20000991,
20000991,
20000991,
20000991,
20000991,
20000991,
20000991,
20000991,
20000991,
20000991,
20000991,
20000991
]
You can see that vbucket replica curr items jumped by 10 million in a single time step.
Will be adding a run with logs shortly...