Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-21033

CLONE - much higher and unbalanced CPU utilization in a two node cluster

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • bug-backlog
    • 4.1.0, 4.1.1
    • couchbase-bucket
    • Triaged
    • Centos 64-bit
    • Unknown

    Description

      When testing the CB 4.1.0 with latest stable YCSB. I found that in a two-node cluster, one node has normal CPU utilization but another has much higher CPU utilization.

      Below are the testing procedure and screenshot.

      1. execute below ycsb testing command from both Client nodes with the record count =20000000
      On Client Node 1:
      ./bin/ycsb load couchbase -s -P ./workloads/workloada -p couchbase.url=http://22.188.5.105:8091/pools -threads 192 -p couchbase.bucket=testdb1

      On Client Node 2:
      ./bin/ycsb load couchbase -s -P ./workloads/workloada -p couchbase.url=http://22.188.5.105:8091/pools -threads 192 -p couchbase.bucket=testdb1 -p insertstart=20000001

      2. After both client ycsb are launched and stabilized, below are the CB Server load of each server node.

      +3. Below is the two-node cluster total load +

      4. Below are each CB server nodes' load status



      *+
      5. Below are each CB server nodes' TOP status+*


      Two CB servers has completely same hardware and CentOS 6.4 configuration(cores/numa-off/THP off/disk partition layout/SElinux disable/Iptables off...). And we also testing the network latency between each CB Sever node and the network is pretty well. (via qperf utility)

      You can see the cpu utilization of the node IP *.105 are much higher than *.104 in the test.

      Is this a bug/issue or a normal characteristic of CB ? And what actions/activities or jobs significantly affect Server node *.105 's CPU , any other telemetry way can monitoring the unusual node when this happen?

      Thanks,

      Attachments

        1. 2.cpu差值.JPG
          2.cpu差值.JPG
          42 kB
        2. 3.整体负载.JPG
          3.整体负载.JPG
          100 kB
        3. 4.104负载.JPG
          4.104负载.JPG
          101 kB
        4. 5.105负载.JPG
          5.105负载.JPG
          85 kB
        5. 6.104_top信息.JPG
          6.104_top信息.JPG
          177 kB
        6. 7.105_top信息.JPG
          7.105_top信息.JPG
          196 kB

        Activity

          People

            vickie Vickie Zeng (Inactive)
            vickie Vickie Zeng (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty