Details
-
Improvement
-
Resolution: Unresolved
-
Major
-
7.2.1
-
titan cluster
-
0
Description
As seen during investigation of MB-58013, there is significant mutex contention on ActiveStream::streamMutex during ActiveStream::next (on frontend worker threads), and ActiveStream::backfillReceived on AuxIO threads:
- Profile: https://share.firefox.dev/3Dz1RF7
Note that ~12% of worker thread time and ~13% of AuxIO thread time is spent locking and unlocking these mutexes.
I don't believe the issue here is the lock duration / scope, it's more there is contention given around 10% of time in each thread is spend in the kernel calling futex - i.e. userspace finds a locked mutex and hence must call into the kernel to acquire it once unlocked by the other thread.
We therefore probably want to look into reducing the contention - perhaps avoiding taking the "big lock" streamMutex in this path and communicate via some form of concurrent queue...