Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-42502

stats endpoint returning volatile values

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 1
    • Unknown

    Description

      It appears that under certain loads, ns server stats endpoints are returning aggregate numbers from each node but with some of the node numbers missing from the aggregate. This only happens for 100% updates, durability none, uniform distribution workload. The test uses 4 nodes with 2 replicas. We have other similar tests with higher durability levels that do not show this stats behaviour. Here is a test where you can see from cbmonitor graphs that the stats appear as 4 distinct bands.

      http://showfast.sc.couchbase.com/#/timeline/Linux/kv/durability/Throughput

      Avg Throughput (ops/sec), SDK 3, Durability None, 4 nodes, Uniform Distribution, Workload K

      http://perf.jenkins.couchbase.com/job/hercules/5687/

      http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hercules_700-3583_access_0548

       

      Running manual check: 

      curl -u Administrator:password -v -X GET http://172.23.100.121:8091/pools/default/buckets/bucket-1/stats | jq '.op.samples.vb_replica_curr_items'

      [
      30000567,
      30000567,
      19996965,
      19996965,
      19996965,
      30000567,
      30000567,
      20005160,
      20005160,
      30000567,
      30000567,
      30000567,
      20000991,
      20000991,
      20000991,
      20000991,
      20000991,
      20000991,
      20000991,
      20000991,
      20000991,
      20000991,
      20000991,
      20000991,
      20000991,
      20000991,
      20000991
      ]

       

      You can see that vbucket replica curr items jumped by 10 million in a single time step. 

      Will be adding a run with logs shortly...

      Attachments

        Activity

          People

            shivani.gupta Shivani Gupta
            korrigan.clark Korrigan Clark (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              PagerDuty