Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36917

Analytics nodes peg CPU for unknown reason

    XMLWordPrintable

Details

    • Untriaged
    • Unknown
    • CX Sprint 176

    Description

      I have a production server running Couchbase 6.0.0. It has 2 analytics nodes, and 2 data/index/query nodes.

      I've been experimenting with analytics queries, sometimes a query is too slow and I cancel it from the UI. I'm not currently running any queries, but the CPU on both analytics nodes is pegged at 100%. The Java process is dominant, as shown by top:

        PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                    
      		 
      10586 couchba+  20   0 56.147g 0.029t  18328 S 691.8 50.6   1700:39 java                                                                                       
      		 
      31491 couchba+  20   0 3160508 1.160g   3556 S  99.7  2.0 108891:48 beam.smp                                                                                   
      		 
      31383 couchba+  20   0 1298944  29664   3216 S   1.6  0.0 207:50.99 beam.smp                                                                                   
      		 
       8639 eben      20   0  157840   2340   1540 R   1.0  0.0   0:00.24 top                                                                                        
      		 
      31954 couchba+  20   0   12012   2724   1008 S   1.0  0.0   7:51.30 goport                                                                                     
      		 
          1 root      20   0  193708   6384   3488 S   0.3  0.0  85:08.20 systemd                                                                                    

       

      I ran cbcollectinfo, the logs are here:

      Analytics Nodes:
      		 
       
      		 
      http://research.corp.couchbase.com/logs/collectinfo-2019-11-14T173629-ns_1@172.23.104.84.zip
      		 
      http://research.corp.couchbase.com/logs/collectinfo-2019-11-14T173629-ns_1@172.23.104.85.zip
      		 
       
      		 
      Other Nodes:
      		 
       
      		 
      http://research.corp.couchbase.com/logs/collectinfo-2019-11-14T173629-ns_1@172.23.120.21.zip
      		 
      http://research.corp.couchbase.com/logs/collectinfo-2019-11-14T173629-ns_1@172.23.120.22.zip

      I did a POST to http://localhost:8095/analytics/cluster/restart and the CPU returned to normal.

       

       

      Attachments

        Issue Links

          is triggering

          Bug - A problem which impairs or prevents the functions of the product. MB-37015 [CX] Intermittent failure in ClusterExecutionIT lifecycle: restart-node-api

          • Major - Major loss of function.
          • Closed
          links to

          Web Link AsterixdDB commit

          For Gerrit Dashboard: MB-36917
          # Subject Branch Project Status CR V
          118324,2 MB-36917: Use Poll Query in Test alice cbas-core Status: ABANDONED -2 +1
          118473,1 MB-36917: Coordinated Test Change alice cbas-core Status: ABANDONED 0 +1
          118508,3 MB-36917: Adapt Tests to Fail Fast Behavior alice cbas-core Status: MERGED +2 +1

          Activity

            People

              murtadha.hubail Murtadha Hubail
              eben Eben Haber
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty