Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19660

much higher and unbalanced CPU utilization in a two node cluster

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • bug-backlog
    • 4.1.0, 4.1.1
    • couchbase-bucket
    • Triaged
    • Centos 64-bit
    • Unknown

    Description

      When testing the CB 4.1.0 with latest stable YCSB. I found that in a two-node cluster, one node has normal CPU utilization but another has much higher CPU utilization.

      Below are the testing procedure and screenshot.

      1. execute below ycsb testing command from both Client nodes with the record count =20000000
      On Client Node 1:
      ./bin/ycsb load couchbase -s -P ./workloads/workloada -p couchbase.url=http://22.188.5.105:8091/pools -threads 192 -p couchbase.bucket=testdb1

      On Client Node 2:
      ./bin/ycsb load couchbase -s -P ./workloads/workloada -p couchbase.url=http://22.188.5.105:8091/pools -threads 192 -p couchbase.bucket=testdb1 -p insertstart=20000001

      2. After both client ycsb are launched and stabilized, below are the CB Server load of each server node.

      +3. Below is the two-node cluster total load +

      4. Below are each CB server nodes' load status



      *+
      5. Below are each CB server nodes' TOP status+*


      Two CB servers has completely same hardware and CentOS 6.4 configuration(cores/numa-off/THP off/disk partition layout/SElinux disable/Iptables off...). And we also testing the network latency between each CB Sever node and the network is pretty well. (via qperf utility)

      You can see the cpu utilization of the node IP *.105 are much higher than *.104 in the test.

      Is this a bug/issue or a normal characteristic of CB ? And what actions/activities or jobs significantly affect Server node *.105 's CPU , any other telemetry way can monitoring the unusual node when this happen?

      Thanks,

      Attachments

        1. 2.cpu差值.JPG
          2.cpu差值.JPG
          42 kB
        2. 3.整体负载.JPG
          3.整体负载.JPG
          100 kB
        3. 4.104负载.JPG
          4.104负载.JPG
          101 kB
        4. 5.105负载.JPG
          5.105负载.JPG
          85 kB
        5. 6.104_top信息.JPG
          6.104_top信息.JPG
          177 kB
        6. 7.105_top信息.JPG
          7.105_top信息.JPG
          196 kB
        7. CPU Usage across nodes.png
          CPU Usage across nodes.png
          425 kB
        8. Terminal CPU usage couchbase.png
          Terminal CPU usage couchbase.png
          482 kB
        9. cpu usage from top commands.png
          cpu usage from top commands.png
          995 kB
        10. CPU usage from console.png
          CPU usage from console.png
          495 kB
        11. cpu usage from top commands.png
          cpu usage from top commands.png
          1.77 MB
        12. cpu usage from console.png
          cpu usage from console.png
          362 kB
        13. cpu usage from top commands.png
          cpu usage from top commands.png
          501 kB
        14. cpu usage from console.png
          cpu usage from console.png
          111 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ingenthr Matt Ingenthron
            nntp nntp
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty