Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51308

1000s of threads blowing out CPU w/ SDK 3

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Yes
    • Analytics Sprint 27

    Description

      Observed high CPU usage on Morpheus, on investigation noticed 1500+ threads that SDK 3 has started:

      $ grep -A999999999999 '2022-03-04T07:52:24.880-05:00 DEBU CBAS.bootstrap.PeriodicThreadDump' ../ns_server/logs/n_0/analytics_debug.log | sed '/lock dump/q' | grep Id= | grep '"cb-' | wc
          1530    9180   82561
      

      $ grep -A999999999999 '2022-03-04T07:52:24.880-05:00 DEBU CBAS.bootstrap.PeriodicThreadDump' ../ns_server/logs/n_0/analytics_debug.log | sed '/lock dump/q' | grep Id= | grep '"cb-' | head -10
      , "cb-timer-3-1" daemon prio=5 Id=124 TIMED_WAITING
      , "cb-events" daemon prio=10 Id=155 TIMED_WAITING
      , "cb-tracing-1" daemon prio=10 Id=156 TIMED_WAITING
      , "cb-metrics-1" daemon prio=10 Id=157 TIMED_WAITING
      , "cb-orphan-1" daemon prio=10 Id=158 TIMED_WAITING
      , "cb-timer-7-1" daemon prio=5 Id=162 TIMED_WAITING
      , "cb-events" daemon prio=10 Id=163 TIMED_WAITING
      , "cb-tracing-2" daemon prio=10 Id=164 TIMED_WAITING
      , "cb-metrics-2" daemon prio=10 Id=165 TIMED_WAITING
      , "cb-orphan-2" daemon prio=10 Id=166 TIMED_WAITING
      

      Also noticed the following in the error log:

      2022-03-04T07:57:11.529-05:00 ERRO CBAS.util.HashedWheelTimer [nioEventLoopGroup-2-1] You are creating too many HashedWheelTimer instances. HashedWheelTimer is a shared resource that must be reused across the JVM, so that only a few instances are created.
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              michael.blow Michael Blow
              michael.blow Michael Blow
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty