Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-32645

[high-bucket] High CPU utilisation during kv rebalance

    XMLWordPrintable

Details

    Description

      Build 6.0.0-1693

      As discussed in high bucket density sync-up meeting, logging this issue for investigation.
      Observed CPU utilisation spikes upto 80% on 24 core orchestrator machine during KV rebalance going on with 30 buckets present in cluster.

      CPU utilisation graph during rebalance-

      cbmonitor link- http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=arke_basic_600-1693_run_kv_rebalance_dd46

      Logs: 

      KV node- https://s3.amazonaws.com/bugdb/jira/index_reb_multibucket/collectinfo-2019-01-08T151840-ns_1%40172.23.97.12.zip
      KV node- https://s3.amazonaws.com/bugdb/jira/index_reb_multibucket/collectinfo-2019-01-08T151840-ns_1%40172.23.97.13.zip
      KV node- https://s3.amazonaws.com/bugdb/jira/index_reb_multibucket/collectinfo-2019-01-08T151840-ns_1%40172.23.97.14.zip

      Attachments

        1. 1_bucket_ns_server.png
          1_bucket_ns_server.png
          496 kB
        2. 1_bucket.png
          1_bucket.png
          667 kB
        3. 10_bucket_ns_server.png
          10_bucket_ns_server.png
          472 kB
        4. 10_bucket.png
          10_bucket.png
          411 kB
        5. 30_buckets_kv_cpu_util.png
          30_buckets_kv_cpu_util.png
          51 kB
        6. 30_buckets_ns_server.png
          30_buckets_ns_server.png
          559 kB
        7. 30_buckets.png
          30_buckets.png
          592 kB
        8. 5_buckets_ns_server.png
          5_buckets_ns_server.png
          610 kB
        9. 5_buckets.png
          5_buckets.png
          509 kB
        10. 6.6.2_CPU.png
          6.6.2_CPU.png
          497 kB
        11. 7.0.0_CPU.png
          7.0.0_CPU.png
          282 kB
        12. 8cores_24cores.png
          8cores_24cores.png
          411 kB
        13. eventing.png
          eventing.png
          532 kB
        14. fts.png
          fts.png
          372 kB
        15. image-2019-01-15-12-11-15-521.png
          image-2019-01-15-12-11-15-521.png
          640 kB
        16. index_query_15.png
          index_query_15.png
          465 kB
        17. index_query_19.png
          index_query_19.png
          655 kB
        18. index_query_20.png
          index_query_20.png
          554 kB
        19. kv_12.png
          kv_12.png
          588 kB
        20. kv_13.png
          kv_13.png
          621 kB
        21. kv_14.png
          kv_14.png
          577 kB
        22. new_30_bucket_ns_server.png
          new_30_bucket_ns_server.png
          610 kB
        23. new_30_bucket.png
          new_30_bucket.png
          640 kB
        24. oc_and_cbas.png
          oc_and_cbas.png
          463 kB
        25. orchestrator_vs_fts_only_node.png
          orchestrator_vs_fts_only_node.png
          611 kB
        26. Screen Shot 2019-01-15 at 09.23.26.png
          Screen Shot 2019-01-15 at 09.23.26.png
          90 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          mahesh.mandhare Mahesh Mandhare (Inactive) created issue -
          drigby Dave Rigby made changes -
          Field Original Value New Value
          Attachment Screen Shot 2019-01-15 at 09.23.26.png [ 63313 ]
          drigby Dave Rigby made changes -
          Assignee Dave Rigby [ drigby ] Mahesh Mandhare [ mahesh.mandhare ]
          drigby Dave Rigby made changes -
          Component/s ns_server [ 10019 ]
          Component/s couchbase-bucket [ 10173 ]
          drigby Dave Rigby made changes -
          Component/s couchbase-bucket [ 10173 ]
          Component/s ns_server [ 10019 ]
          mahesh.mandhare Mahesh Mandhare (Inactive) made changes -
          Component/s ns_server [ 10019 ]
          Component/s couchbase-bucket [ 10173 ]
          mahesh.mandhare Mahesh Mandhare (Inactive) made changes -
          Assignee Mahesh Mandhare [ mahesh.mandhare ] Poonam Dhavale [ poonam ]
          lynn.straus Lynn Straus made changes -
          Fix Version/s Mad-Hatter [ 15037 ]
          poonam Poonam Dhavale made changes -
          Assignee Poonam Dhavale [ poonam ] Mahesh Mandhare [ mahesh.mandhare ]
          mahesh.mandhare Mahesh Mandhare (Inactive) made changes -
          Assignee Mahesh Mandhare [ mahesh.mandhare ] Poonam Dhavale [ poonam ]
          poonam Poonam Dhavale made changes -
          Assignee Poonam Dhavale [ poonam ] Mahesh Mandhare [ mahesh.mandhare ]
          mahesh.mandhare Mahesh Mandhare (Inactive) made changes -
          Assignee Mahesh Mandhare [ mahesh.mandhare ] Poonam Dhavale [ poonam ]
          poonam Poonam Dhavale made changes -
          Assignee Poonam Dhavale [ poonam ] Mahesh Mandhare [ mahesh.mandhare ]
          mahesh.mandhare Mahesh Mandhare (Inactive) made changes -
          Assignee Mahesh Mandhare [ mahesh.mandhare ] Poonam Dhavale [ poonam ]
          poonam Poonam Dhavale made changes -
          Assignee Poonam Dhavale [ poonam ] Mahesh Mandhare [ mahesh.mandhare ]
          poonam Poonam Dhavale made changes -
          Attachment orchestrator_vs_fts_only_node.png [ 67371 ]
          Attachment fts.png [ 67372 ]
          Attachment index_query_20.png [ 67373 ]
          Attachment index_query_19.png [ 67374 ]
          Attachment eventing.png [ 67375 ]
          Attachment index_query_15.png [ 67376 ]
          Attachment kv_14.png [ 67377 ]
          Attachment kv_13.png [ 67378 ]
          Attachment kv_12.png [ 67379 ]
          Attachment oc_and_cbas.png [ 67380 ]
          mahesh.mandhare Mahesh Mandhare (Inactive) made changes -
          Assignee Mahesh Mandhare [ mahesh.mandhare ] Poonam Dhavale [ poonam ]
          poonam Poonam Dhavale made changes -
          Attachment 30_buckets_ns_server.png [ 68545 ]
          Attachment 30_buckets.png [ 68546 ]
          Attachment 10_bucket_ns_server.png [ 68547 ]
          Attachment 10_bucket.png [ 68548 ]
          Attachment 5_buckets_ns_server.png [ 68549 ]
          Attachment 5_buckets.png [ 68550 ]
          Attachment 1_bucket_ns_server.png [ 68551 ]
          Attachment 1_bucket.png [ 68552 ]
          poonam Poonam Dhavale made changes -
          Attachment new_30_bucket.png [ 68556 ]
          Attachment new_30_bucket_ns_server.png [ 68557 ]
          poonam Poonam Dhavale made changes -
          Assignee Poonam Dhavale [ poonam ] Ajit Yagaty [ ajit.yagaty ]
          ajit.yagaty Ajit Yagaty [X] (Inactive) made changes -
          Assignee Ajit Yagaty [ ajit.yagaty ] Mahesh Mandhare [ mahesh.mandhare ]
          lynn.straus Lynn Straus made changes -
          Fix Version/s Cheshire-Cat [ 15915 ]
          Fix Version/s Mad-Hatter [ 15037 ]
          Labels high-bucket-density deferred-from-Mad-Hatter high-bucket-density
          raju Raju Suravarjjala made changes -
          Assignee Mahesh Mandhare [ mahesh.mandhare ] Wayne Siu [ wayne ]
          wayne Wayne Siu made changes -
          Labels deferred-from-Mad-Hatter high-bucket-density deferred-from-Mad-Hatter high-bucket-density performance
          wayne Wayne Siu made changes -
          Issue Type Bug [ 1 ] Task [ 3 ]
          meni.hillel Meni Hillel (Inactive) made changes -
          Component/s performance [ 10222 ]
          wayne Wayne Siu made changes -
          Summary High CPU utilisation during kv rebalance [high-bucket] High CPU utilisation during kv rebalance
          bo-chun.wang Bo-Chun Wang made changes -
          Attachment 30_buckets_kv_cpu_util.png [ 132051 ]
          meni.hillel Meni Hillel (Inactive) made changes -
          Resolution Done [ 6 ]
          Status Open [ 1 ] Closed [ 6 ]
          meni.hillel Meni Hillel (Inactive) made changes -
          Assignee Wayne Siu [ wayne ] Meni Hillel [ JIRAUSER25407 ]
          Resolution Done [ 6 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          dfinlay Dave Finlay made changes -
          Assignee Meni Hillel [ JIRAUSER25407 ] Bo-Chun Wang [ bo-chun.wang ]
          bo-chun.wang Bo-Chun Wang made changes -
          Attachment 7.0.0_CPU.png [ 132322 ]
          bo-chun.wang Bo-Chun Wang made changes -
          Attachment 6.6.2_CPU.png [ 132323 ]
          dfinlay Dave Finlay made changes -
          Resolution Fixed [ 1 ]
          Status Reopened [ 4 ] Resolved [ 5 ]
          bo-chun.wang Bo-Chun Wang made changes -
          Attachment 8cores_24cores.png [ 135753 ]
          lynn.straus Lynn Straus made changes -
          Fix Version/s 7.0.0 [ 17233 ]
          lynn.straus Lynn Straus made changes -
          Fix Version/s Cheshire-Cat [ 15915 ]

          People

            bo-chun.wang Bo-Chun Wang
            mahesh.mandhare Mahesh Mandhare (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty