Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36917

Analytics nodes peg CPU for unknown reason

    XMLWordPrintable

Details

    • Untriaged
    • Unknown
    • CX Sprint 176

    Description

      I have a production server running Couchbase 6.0.0. It has 2 analytics nodes, and 2 data/index/query nodes.

      I've been experimenting with analytics queries, sometimes a query is too slow and I cancel it from the UI. I'm not currently running any queries, but the CPU on both analytics nodes is pegged at 100%. The Java process is dominant, as shown by top:

        PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                    
      10586 couchba+  20   0 56.147g 0.029t  18328 S 691.8 50.6   1700:39 java                                                                                       
      31491 couchba+  20   0 3160508 1.160g   3556 S  99.7  2.0 108891:48 beam.smp                                                                                   
      31383 couchba+  20   0 1298944  29664   3216 S   1.6  0.0 207:50.99 beam.smp                                                                                   
       8639 eben      20   0  157840   2340   1540 R   1.0  0.0   0:00.24 top                                                                                        
      31954 couchba+  20   0   12012   2724   1008 S   1.0  0.0   7:51.30 goport                                                                                     
          1 root      20   0  193708   6384   3488 S   0.3  0.0  85:08.20 systemd                                                                                    
       

       

      I ran cbcollectinfo, the logs are here:

      Analytics Nodes:
       
      http://research.corp.couchbase.com/logs/collectinfo-2019-11-14T173629-ns_1@172.23.104.84.zip
      http://research.corp.couchbase.com/logs/collectinfo-2019-11-14T173629-ns_1@172.23.104.85.zip
       
      Other Nodes:
       
      http://research.corp.couchbase.com/logs/collectinfo-2019-11-14T173629-ns_1@172.23.120.21.zip
      http://research.corp.couchbase.com/logs/collectinfo-2019-11-14T173629-ns_1@172.23.120.22.zip
       

      I did a POST to http://localhost:8095/analytics/cluster/restart and the CPU returned to normal.

       

       

      Attachments

        Issue Links

          Activity

            People

              murtadha.hubail Murtadha Hubail
              eben Eben Haber
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty