Details
-
Bug
-
Resolution: Fixed
-
Major
-
6.5.0
-
Triaged
-
Unknown
Description
From recent work looking into cache contention, I've seen that cache lines containing TopKeys::Shards are generally contentious. They're generally not that contentious, on the read heavy test we have a few cache lines with a 1-2% hit modified rate and the majority are sub 1%, but we have a lot of TopKeys::Shards.
First two percentages are hit modified rates, second two are store rates. From the below perf c2c output we can see that we only write to the mutex but get some hit modifieds on the other members of TopKeys::Shards. In general, it seems that the mutex is pretty heavily contended, but that's a separate issue. We should separate pad the mutexes at least so that there is no false sharing with other data. Note also, TopKey::Shards are not cache aligned themselves, we see the mutex and other members at different relative addresses in different cache lines.
-------------------------------------------------------------
|
2 24 14 45 3 0x7f080f60d1c0
|
-------------------------------------------------------------
|
37.50% 21.43% 0.00% 0.00% 0x8 0x460a5d 250 101 100 17 18 [.] TopKeys::Shard::updateKey memcached stl_iterator.h:780 0{18 66.7% n/a} 1{18 33.3% n/a}
|
|
|
---TopKeys::Shard::updateKey
|
TopKeys::updateKey
|
|
|
|--11450--GetCommandContext::sendResponse
|
| GetCommandContext::step
|
| SteppableCommandContext::drive
|
| get_executor
|
| execute_client_request_packet
|
| Cookie::execute
|
| StateMachine::conn_execute
|
| Connection::runStateMachinery
|
| Connection::runEventLoop
|
| run_event_loop
|
| event_handler
|
| event_process_active_single_queue.isra.26
|
| event_base_loop
|
| worker_libevent
|
| platform_thread_wrap
|
| start_thread
|
| __clone
|
|
|
--4558--MutationCommandContext::sendResponse
|
MutationCommandContext::step
|
SteppableCommandContext::drive
|
set_executor
|
execute_client_request_packet
|
Cookie::execute
|
StateMachine::conn_execute
|
Connection::runStateMachinery
|
Connection::runEventLoop
|
run_event_loop
|
event_handler
|
event_process_active_single_queue.isra.26
|
event_base_loop
|
worker_libevent
|
platform_thread_wrap
|
start_thread
|
__clone
|
|
|
0.00% 7.14% 0.00% 0.00% 0x8 0x460b10 0 103 171 13 12 [.] TopKeys::Shard::updateKey memcached topkeys.cc:199 0{12 0.0% n/a} 1{12 100.0% n/a}
|
|
|
---TopKeys::Shard::updateKey
|
TopKeys::updateKey
|
|
|
|--8161--GetCommandContext::sendResponse
|
| GetCommandContext::step
|
| SteppableCommandContext::drive
|
| get_executor
|
| execute_client_request_packet
|
| Cookie::execute
|
| StateMachine::conn_execute
|
| Connection::runStateMachinery
|
| Connection::runEventLoop
|
| run_event_loop
|
| event_handler
|
| event_process_active_single_queue.isra.26
|
| event_base_loop
|
| worker_libevent
|
| platform_thread_wrap
|
| start_thread
|
| __clone
|
|
|
--2577--MutationCommandContext::sendResponse
|
MutationCommandContext::step
|
SteppableCommandContext::drive
|
set_executor
|
execute_client_request_packet
|
Cookie::execute
|
StateMachine::conn_execute
|
Connection::runStateMachinery
|
Connection::runEventLoop
|
run_event_loop
|
event_handler
|
event_process_active_single_queue.isra.26
|
event_base_loop
|
worker_libevent
|
platform_thread_wrap
|
start_thread
|
__clone
|
|
|
8.33% 7.14% 0.00% 0.00% 0x10 0x460a61 247 98 157 28 20 [.] TopKeys::Shard::updateKey memcached stl_iterator.h:780 0{20 33.3% n/a} 1{20 66.7% n/a}
|
|
|
---TopKeys::Shard::updateKey
|
TopKeys::updateKey
|
|
|
|--18377--GetCommandContext::sendResponse
|
| GetCommandContext::step
|
| SteppableCommandContext::drive
|
| get_executor
|
| execute_client_request_packet
|
| Cookie::execute
|
| StateMachine::conn_execute
|
| Connection::runStateMachinery
|
| Connection::runEventLoop
|
| run_event_loop
|
| event_handler
|
| event_process_active_single_queue.isra.26
|
| event_base_loop
|
| worker_libevent
|
| platform_thread_wrap
|
| start_thread
|
| __clone
|
|
|
--5085--MutationCommandContext::sendResponse
|
MutationCommandContext::step
|
SteppableCommandContext::drive
|
set_executor
|
execute_client_request_packet
|
Cookie::execute
|
StateMachine::conn_execute
|
Connection::runStateMachinery
|
Connection::runEventLoop
|
run_event_loop
|
event_handler
|
event_process_active_single_queue.isra.26
|
event_base_loop
|
worker_libevent
|
platform_thread_wrap
|
start_thread
|
__clone
|
|
|
8.33% 0.00% 0.00% 0.00% 0x20 0x460b25 332 0 134 12 11 [.] TopKeys::Shard::updateKey memcached topkeys.cc:161 0{11 50.0% n/a} 1{11 50.0% n/a}
|
|
|
---TopKeys::Shard::updateKey
|
TopKeys::updateKey
|
|
|
|--7176--GetCommandContext::sendResponse
|
| GetCommandContext::step
|
| SteppableCommandContext::drive
|
| get_executor
|
| execute_client_request_packet
|
| Cookie::execute
|
| StateMachine::conn_execute
|
| Connection::runStateMachinery
|
| Connection::runEventLoop
|
| run_event_loop
|
| event_handler
|
| event_process_active_single_queue.isra.26
|
| event_base_loop
|
| worker_libevent
|
| platform_thread_wrap
|
| start_thread
|
| __clone
|
|
|
--2170--MutationCommandContext::sendResponse
|
MutationCommandContext::step
|
SteppableCommandContext::drive
|
set_executor
|
execute_client_request_packet
|
Cookie::execute
|
StateMachine::conn_execute
|
Connection::runStateMachinery
|
Connection::runEventLoop
|
run_event_loop
|
event_handler
|
event_process_active_single_queue.isra.26
|
event_base_loop
|
worker_libevent
|
platform_thread_wrap
|
start_thread
|
__clone
|
|
|
0.00% 0.00% 48.89% 33.33% 0x28 0x7f082a50dc8d 0 0 358 48 29 [.] pthread_mutex_lock libpthread-2.17.so .:0 0{29 n/a 69.6%} 1{29 n/a 30.4%}
|
|
|
---pthread_mutex_lock
|
TopKeys::Shard::updateKey
|
TopKeys::updateKey
|
|
|
|--12506404--GetCommandContext::sendResponse
|
| GetCommandContext::step
|
| SteppableCommandContext::drive
|
| get_executor
|
| execute_client_request_packet
|
| Cookie::execute
|
| StateMachine::conn_execute
|
| Connection::runStateMachinery
|
| Connection::runEventLoop
|
| run_event_loop
|
| event_handler
|
| event_process_active_single_queue.isra.26
|
| event_base_loop
|
| worker_libevent
|
| platform_thread_wrap
|
| start_thread
|
| __clone
|
|
|
--472772--MutationCommandContext::sendResponse
|
MutationCommandContext::step
|
SteppableCommandContext::drive
|
set_executor
|
execute_client_request_packet
|
Cookie::execute
|
StateMachine::conn_execute
|
Connection::runStateMachinery
|
Connection::runEventLoop
|
run_event_loop
|
event_handler
|
event_process_active_single_queue.isra.26
|
event_base_loop
|
worker_libevent
|
platform_thread_wrap
|
start_thread
|
__clone
|
|
|
0.00% 0.00% 33.33% 0.00% 0x28 0x7f082a50ee1b 0 0 519 57 30 [.] pthread_mutex_unlock libpthread-2.17.so .:0 0{30 n/a 73.3%} 1{30 n/a 26.7%}
|
|
|
---pthread_mutex_unlock
|
TopKeys::Shard::updateKey
|
TopKeys::updateKey
|
|
|
|--6793559--GetCommandContext::sendResponse
|
| GetCommandContext::step
|
| SteppableCommandContext::drive
|
| get_executor
|
| execute_client_request_packet
|
| Cookie::execute
|
| StateMachine::conn_execute
|
| Connection::runStateMachinery
|
| Connection::runEventLoop
|
| run_event_loop
|
| event_handler
|
| event_process_active_single_queue.isra.26
|
| event_base_loop
|
| worker_libevent
|
| platform_thread_wrap
|
| start_thread
|
| __clone
|
|
|
--1027047--MutationCommandContext::sendResponse
|
MutationCommandContext::step
|
SteppableCommandContext::drive
|
set_executor
|
execute_client_request_packet
|
Cookie::execute
|
StateMachine::conn_execute
|
Connection::runStateMachinery
|
Connection::runEventLoop
|
run_event_loop
|
event_handler
|
event_process_active_single_queue.isra.26
|
event_base_loop
|
worker_libevent
|
platform_thread_wrap
|
start_thread
|
__clone
|
|
|
0.00% 0.00% 17.78% 66.67% 0x30 0x7f082a50ee0a 0 0 0 10 8 [.] pthread_mutex_unlock libpthread-2.17.so .:0 0{ 8 n/a 90.0%} 1{ 8 n/a 10.0%}
|
|
|
---pthread_mutex_unlock
|
TopKeys::Shard::updateKey
|
TopKeys::updateKey
|
|
|
|--3400839--GetCommandContext::sendResponse
|
| GetCommandContext::step
|
| SteppableCommandContext::drive
|
| get_executor
|
| execute_client_request_packet
|
| Cookie::execute
|
| StateMachine::conn_execute
|
| Connection::runStateMachinery
|
| Connection::runEventLoop
|
| run_event_loop
|
| event_handler
|
| event_process_active_single_queue.isra.26
|
| event_base_loop
|
| worker_libevent
|
| platform_thread_wrap
|
| start_thread
|
| __clone
|
|
|
--1289988--MutationCommandContext::sendResponse
|
MutationCommandContext::step
|
SteppableCommandContext::drive
|
set_executor
|
execute_client_request_packet
|
Cookie::execute
|
StateMachine::conn_execute
|
Connection::runStateMachinery
|
Connection::runEventLoop
|
run_event_loop
|
event_handler
|
event_process_active_single_queue.isra.26
|
event_base_loop
|
worker_libevent
|
platform_thread_wrap
|
start_thread
|
__clone
|
|
|
45.83% 57.14% 0.00% 0.00% 0x38 0x7f082a50dc30 265 102 136 24 17 [.] pthread_mutex_lock libpthread-2.17.so .:0 0{17 84.2% n/a} 1{17 15.8% n/a}
|
|
|
---pthread_mutex_lock
|
TopKeys::Shard::updateKey
|
TopKeys::updateKey
|
|
|
|--17517--GetCommandContext::sendResponse
|
| GetCommandContext::step
|
| SteppableCommandContext::drive
|
| get_executor
|
| execute_client_request_packet
|
| Cookie::execute
|
| StateMachine::conn_execute
|
| Connection::runStateMachinery
|
| Connection::runEventLoop
|
| run_event_loop
|
| event_handler
|
| event_process_active_single_queue.isra.26
|
| event_base_loop
|
| worker_libevent
|
| platform_thread_wrap
|
| start_thread
|
| __clone
|
|
|
--3402--MutationCommandContext::sendResponse
|
MutationCommandContext::step
|
SteppableCommandContext::drive
|
set_executor
|
execute_client_request_packet
|
Cookie::execute
|
StateMachine::conn_execute
|
Connection::runStateMachinery
|
Connection::runEventLoop
|
run_event_loop
|
event_handler
|
event_process_active_single_queue.isra.26
|
event_base_loop
|
worker_libevent
|
platform_thread_wrap
|
start_thread
|
__clone
|
|
|
0.00% 7.14% 0.00% 0.00% 0x38 0x7f082a50edf0 0 129 0 1 1 [.] pthread_mutex_unlock libpthread-2.17.so .:0 1{ 1 100.0% n/a}
|
|
|
---pthread_mutex_unlock
|
TopKeys::Shard::updateKey
|
TopKeys::updateKey
|
GetCommandContext::sendResponse
|
GetCommandContext::step
|
SteppableCommandContext::drive
|
get_executor
|
execute_client_request_packet
|
Cookie::execute
|
StateMachine::conn_execute
|
Connection::runStateMachinery
|
Connection::runEventLoop
|
run_event_loop
|
event_handler
|
event_process_active_single_queue.isra.26
|
event_base_loop
|
worker_libevent
|
platform_thread_wrap
|
start_thread
|
__clone
|
-------------------------------------------------------------
|
3 23 18 53 2 0x7f080f60d100
|
-------------------------------------------------------------
|
26.09% 33.33% 0.00% 0.00% 0x8 0x460a5d 251 99 113 20 16 [.] TopKeys::Shard::updateKey memcached stl_iterator.h:780 0{16 75.0% n/a} 1{16 25
|
.0% n/a}
|
8.70% 0.00% 0.00% 0.00% 0x10 0x460a61 238 0 168 32 25 [.] TopKeys::Shard::updateKey memcached stl_iterator.h:780 0{25 100.0% n/a} 1{25 0
|
.0% n/a}
|
0.00% 0.00% 43.40% 50.00% 0x28 0x7f082a50dc8d 0 0 293 51 33 [.] pthread_mutex_lock libpthread-2.17.so .:0 0{33 n/a 70.8%} 1{33
|
n/a 29.2%}
|
0.00% 0.00% 30.19% 0.00% 0x28 0x7f082a50ee1b 0 0 437 41 25 [.] pthread_mutex_unlock libpthread-2.17.so .:0 0{25 n/a 81.2%} 1{25
|
n/a 18.8%}
|
0.00% 0.00% 1.89% 0.00% 0x28 0x7f082a51242f 0 0 253 7 7 [.] __lll_lock_wait libpthread-2.17.so .:0 0{ 7 n/a 0.0%} 1{ 7
|
n/a 100.0%}
|
0.00% 0.00% 20.75% 50.00% 0x30 0x7f082a50ee0a 0 0 0 12 11 [.] pthread_mutex_unlock libpthread-2.17.so .:0 0{11 n/a 66.7%} 1{11
|
n/a 33.3%}
|
0.00% 0.00% 3.77% 0.00% 0x34 0x7f082a50ee11 0 0 144 3 3 [.] pthread_mutex_unlock libpthread-2.17.so .:0 0{ 3 n/a 100.0%} 1{ 3
|
n/a 0.0%}
|
65.22% 66.67% 0.00% 0.00% 0x38 0x7f082a50dc30 255 130 121 34 23 [.] pthread_mutex_lock libpthread-2.17.so .:0 0{23 66.7% n/a} 1{23 33
|
.3% n/a}
|
|
|
-------------------------------------------------------------
|
4 18 21 70 1 0x7f080f60d1c0
|
-------------------------------------------------------------
|
44.44% 38.10% 0.00% 0.00% 0x8 0x460a5d 253 100 133 24 21 [.] TopKeys::Shard::updateKey memcached stl_iterator.h:780 0{21 75.0% n/a} 1{21 25
|
.0% n/a}
|
11.11% 9.52% 0.00% 0.00% 0x10 0x460a61 266 118 134 25 18 [.] TopKeys::Shard::updateKey memcached stl_iterator.h:780 0{18 75.0% n/a} 1{18 25
|
.0% n/a}
|
0.00% 0.00% 52.86% 100.00% 0x28 0x7f082a50dc8d 0 0 261 67 30 [.] pthread_mutex_lock libpthread-2.17.so .:0 0{30 n/a 65.8%} 1{30
|
n/a 34.2%}
|
0.00% 0.00% 35.71% 0.00% 0x28 0x7f082a50ee1b 0 0 433 57 31 [.] pthread_mutex_unlock libpthread-2.17.so .:0 0{31 n/a 68.0%} 1{31
|
n/a 32.0%}
|
0.00% 0.00% 1.43% 0.00% 0x28 0x7f082a51242f 0 0 178 5 5 [.] __lll_lock_wait libpthread-2.17.so .:0 0{ 5 n/a 100.0%} 1{ 5
|
n/a 0.0%}
|
0.00% 0.00% 5.71% 0.00% 0x30 0x7f082a50ee0a 0 0 0 4 4 [.] pthread_mutex_unlock libpthread-2.17.so .:0 0{ 4 n/a 100.0%}
|
|
|
0.00% 0.00% 4.29% 0.00% 0x34 0x7f082a50dca4 0 0 141 17 13 [.] pthread_mutex_lock libpthread-2.17.so .:0 0{13 n/a 33.3%} 1{13
|
n/a 66.7%}
|
44.44% 52.38% 0.00% 0.00% 0x38 0x7f082a50dc30 258 160 135 28 24 [.] pthread_mutex_lock libpthread-2.17.so .:0 0{24 47.4% n/a} 1{24 52
|
.6% n/a}
|
|
|
-------------------------------------------------------------
|
5 21 15 51 0 0x7f080f60d000
|
-------------------------------------------------------------
|
4.76% 0.00% 0.00% 0.00% 0x0 0x460b25 330 0 138 9 8 [.] TopKeys::Shard::updateKey memcached topkeys.cc:161 0{ 8 100.0%
|
n/a} 1{ 8 0.0% n/a}
|
4.76% 0.00% 1.96% 0.00% 0x8 0x7f082a51242f 370 0 102 3 3 [.] __lll_lock_wait libpthread-2.17.so .:0 0{ 3 100.0% 0.0%} 1{ 3 0.0% 100.0%}
|
0.00% 0.00% 39.22% 0.00% 0x8 0x7f082a50dc8d 0 0 242 48 28 [.] pthread_mutex_lock libpthread-2.17.so .:0 0{28 n/a 40.0%} 1{28 n/a 60.0%}
|
0.00% 0.00% 37.25% 0.00% 0x8 0x7f082a50ee1b 0 0 438 44 26 [.] pthread_mutex_unlock libpthread-2.17.so .:0 0{26 n/a 63.2%} 1{26 n/a 36.8%}
|
0.00% 0.00% 11.76% 0.00% 0x10 0x7f082a50ee0a 0 0 0 6 5 [.] pthread_mutex_unlock libpthread-2.17.so .:0 0{ 5 n/a 50.0%} 1{ 5 n/a 50.0%}
|
0.00% 6.67% 0.00% 0.00% 0x14 0x7f082a50dca4 0 93 172 30 20 [.] pthread_mutex_lock libpthread-2.17.so .:0 0{20 0.0% n/a} 1{20 100.0% n/a}
|
90.48% 80.00% 0.00% 0.00% 0x18 0x7f082a50dc30 252 135 123 33 26 [.] pthread_mutex_lock libpthread-2.17.so .:0 0{26 61.3% n/a} 1{26 38.7% n/a}
|
0.00% 6.67% 0.00% 0.00% 0x18 0x7f082a50edf0 0 103 90 2 3 [.] pthread_mutex_unlock libpthread-2.17.so .:0 0{ 3 100.0% n/a}
|
0.00% 0.00% 9.80% 0.00% 0x30 0x7f082acd90b7 0 0 0 5 4 [.] std::__detail::_List_node_base::_M_transfer libstdc++.so list.cc:102 0{ 4 n/a 40.0%} 1{ 4 n/a 60.0%}
|
0.00% 6.67% 0.00% 0.00% 0x38 0x460d2e 0 117 96 2 2 [.] TopKeys::Shard::updateKey memcached stl_list.h:177 0{ 2 100.0% n/a} 1{ 2 0.0% n/a}
|
Attachments
Issue Links
- duplicates
-
MB-25894
Lock contention in topkeys
-
- Closed
-