Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.6.0
-
Untriaged
-
1
-
Yes
-
Analytics Sprint 27
Description
Observed high CPU usage on Morpheus, on investigation noticed 1500+ threads that SDK 3 has started:
$ grep -A999999999999 '2022-03-04T07:52:24.880-05:00 DEBU CBAS.bootstrap.PeriodicThreadDump' ../ns_server/logs/n_0/analytics_debug.log | sed '/lock dump/q' | grep Id= | grep '"cb-' | wc
|
1530 9180 82561
|
$ grep -A999999999999 '2022-03-04T07:52:24.880-05:00 DEBU CBAS.bootstrap.PeriodicThreadDump' ../ns_server/logs/n_0/analytics_debug.log | sed '/lock dump/q' | grep Id= | grep '"cb-' | head -10
|
, "cb-timer-3-1" daemon prio=5 Id=124 TIMED_WAITING
|
, "cb-events" daemon prio=10 Id=155 TIMED_WAITING
|
, "cb-tracing-1" daemon prio=10 Id=156 TIMED_WAITING
|
, "cb-metrics-1" daemon prio=10 Id=157 TIMED_WAITING
|
, "cb-orphan-1" daemon prio=10 Id=158 TIMED_WAITING
|
, "cb-timer-7-1" daemon prio=5 Id=162 TIMED_WAITING
|
, "cb-events" daemon prio=10 Id=163 TIMED_WAITING
|
, "cb-tracing-2" daemon prio=10 Id=164 TIMED_WAITING
|
, "cb-metrics-2" daemon prio=10 Id=165 TIMED_WAITING
|
, "cb-orphan-2" daemon prio=10 Id=166 TIMED_WAITING
|
Also noticed the following in the error log:
2022-03-04T07:57:11.529-05:00 ERRO CBAS.util.HashedWheelTimer [nioEventLoopGroup-2-1] You are creating too many HashedWheelTimer instances. HashedWheelTimer is a shared resource that must be reused across the JVM, so that only a few instances are created.
|
Attachments
Issue Links
- relates to
-
MB-58888 Investigate CbRemoteLinks660ITD failure
- Resolved