Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.1.0
-
Triaged
-
1
-
Yes
-
KV 2021-Nov
Description
Time taken to perform mcstat collections-details for nodes with lots of collections has regressed since CC. This is not a server side regression (confirmed through tracing, and using new mcstat against a CC cluster).
Perf for CC mcstat against master KV, 1000 collections:
27.16 s 100.0% 0 s mcstat (81063)
|
27.16 s 100.0% 0 s Main Thread 0x223a700
|
27.15 s 99.9% 0 s start
|
27.15 s 99.9% 0 s main
|
27.14 s 99.9% 0 s request_stat(MemcachedConnection&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
|
27.10 s 99.7% 35.00 ms MemcachedConnection::stats(std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::function<std::__1::vector<std::__1::unique_ptr<FrameInfo, std::__1::default_delete<FrameInfo> >, std::__1::allocator<std::__1::unique_ptr<FrameInfo, std::__1::default_delete<FrameInfo> > > > ()>)
|
15.83 s 58.2% 57.00 ms MemcachedConnection::recvResponse(BinprotResponse&)
|
15.51 s 57.1% 165.00 ms MemcachedConnection::recvFrame(Frame&)
|
7.65 s 28.1% 5.00 ms MemcachedConnection::read(Frame&, unsigned long)
|
7.62 s 28.0% 129.00 ms MemcachedConnection::readPlain(Frame&, unsigned long)
|
|
master mcstat, same node:
1.61 min 100.0% 0 s mcstat (80646)
|
1.61 min 100.0% 0 s Main Thread 0x2235033
|
1.61 min 99.9% 0 s start
|
1.61 min 99.9% 0 s main
|
1.61 min 99.9% 0 s request_stat(MemcachedConnection&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
|
1.61 min 99.9% 48.00 ms MemcachedConnection::stats(std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::function<std::__1::vector<std::__1::unique_ptr<FrameInfo, std::__1::default_delete<FrameInfo> >, std::__1::allocator<std::__1::unique_ptr<FrameInfo, std::__1::default_delete<FrameInfo> > > > ()>)
|
1.36 min 84.7% 51.00 ms MemcachedConnection::recvResponse(BinprotResponse&, cb::mcbp::ClientOpcode, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >)
|
1.36 min 84.2% 54.00 ms MemcachedConnection::recvFrame(Frame&, cb::mcbp::ClientOpcode, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >)
|
1.31 min 81.1% 55.00 ms AsyncReadCallback::drain(unsigned long)
|
1.31 min 81.1% 1.31 min _platform_memmove$VARIANT$Haswell
|
|
|
AsyncReadCallback::drain |
void drain(size_t nb) {
|
if ((available - nb) > 0) {
|
std::memmove(backing.data(), backing.data() + nb, available - nb);
|
}
|
available -= nb;
|
}
|
A lot of time is spent shuffling data as bytes are consumed from the backing vector.
Attachments
Issue Links
- relates to
-
-
- Closed
-