Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35961

CPU/memory usage on otherwise idle system

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Unresolved
    • Major
    • backlog
    • 6.5.0
    • ns_server

    Description

      As I add more and more empty buckets to an idle Couchbase system built from just one (1) node with an 8 CPU VM and 24MB of RAM doing nothing the CPU continues to climb for both memcached and beam.smp at 28 buckets (26 empty) we use almost 1/2 of the cores in the 8 CPU VM

      I start with no buckets and the top output is relatively low (top -n 4 user linuxbrew)

      {{PID     USER PR NI  VIRT     RES SHR   S %CPU %MEM   TIME+ COMMAND
      2401 linuxbr+ 20 0 4182012 180416  7128 S  8.0  0.7 0:16.65 beam.smp
      2741 linuxbr+ 20 0 8917820 484636 21860 S  2.0  2.0 0:18.64 java
      2541 linuxbr+ 20 0  743128  32984 19756 S  0.8  0.1 0:01.46 indexer
      2594 linuxbr+ 20 0  916168  54980 39020 S  0.8  0.2 0:00.93 cbq-engine}}

      I add 4 buckets 2 with data (32K items and 5K items) (top -n 4 user linuxbrew)

      {{PID     USER PR NI   VIRT    RES   SHR S %CPU %MEM   TIME+ COMMAND
      2401 linuxbr+ 20 0 4704780 599744  7192 S  61.3 2.4 6:34.93 beam.smp
      2669 linuxbr+ 20 0 2465196 194636 24328 S  51.0 0.8 2:56.27 memcached
      2541 linuxbr+ 20 0 1389776 335672 21896 S  16.4 1.4 2:31.17 indexer
      2553 linuxbr+ 20 0  851196 307380 15856 S   2.4 1.2 0:34.77 projector}}

      (more details in attachment)

      I add 24 more empty buckets (28 total) (top -n 4 user linuxbrew)

      {{PID     USER PR NI   VIRT    RES   SHR S %CPU %MEM    TIME+ COMMAND
      2669 linuxbr+ 20 0 3341740 906616 24328 S 191.8 3.7 30:40.84 memcached
      2401 linuxbr+ 20 0 7157152 1.784g  7192 S 144.7 7.6 36:47.43 beam.smp
      2541 linuxbr+ 20 0 1389968 325808 21896 S  12.8 1.3  6:08.28 indexer
      2459 linuxbr+ 20 0 4222184 315872 16748 S   2.3 1.3  0:48.81 beam.smp}}

      Is this really the expected CPU usage behavior for a one (1) node system doing 
      nothing  e.g. the final CPU for the Couchbase user "linuxbrew" CPU sums to 351.6% 
      this is about 3.5 cores or almost 1/2 of my available compute.

      I have attached an lsof output of running 28 bucket system and in addition some strace outputs.

       

      Attachments

        1. 07_snip_pre_pkill_STOP_beam.smp.JPG
          90 kB
          Jon Strabala
        2. 08_snip_waited_120sec_post_pkill_STOP_beam.smp.JPG
          90 kB
          Jon Strabala
        3. cpu_creep_on_mostly_empty_buckets.txt
          23 kB
          Jon Strabala
        4. R00_snap_top_all_couchbase_just_1_bucket_28_collections_31k_5k_empty_7.0.0-5063.JPG
          164 kB
          Jon Strabala
        5. R01_snap_top_all_users_REDO_7.0.0-5063.JPG
          150 kB
          Jon Strabala
        6. R02_snap_top_all_users_REDO_after_pkill_beam_7.0.0-5063.JPG
          152 kB
          Jon Strabala
        7. R03_snap_top_all_users_stop_start_all_REDO_7.0.0-5063.JPG
          158 kB
          Jon Strabala
        8. R04_snap_top_all_users_stopped_prometheus_REDO_7.0.0-5063.JPG
          160 kB
          Jon Strabala
        9. R05_snap_top_all_users_REDO_no_prometheus_after_pkill_beam_7.0.0-5063.JPG
          150 kB
          Jon Strabala
        10. REDO_7.0.0-5063_lsof_-i_-u_couchbase.txt
          182 kB
          Jon Strabala
        11. REDO_7.0.0-5063_strace_measures.txt
          51 kB
          Jon Strabala
        12. running_lsof_-i _u_linuxbrew.txt
          178 kB
          Jon Strabala
        13. snap_top_all_users.JPG
          106 kB
          Jon Strabala
        14. some_info_on_log_activity_and tails_on_all_active_logs.txt
          58 kB
          Jon Strabala
        15. strace_measures.txt
          2 kB
          Jon Strabala
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            dfinlay Dave Finlay
            jon.strabala Jon Strabala
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty