Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49388

mcstat collections-details performance regression

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • Neo
    • Neo
    • couchbase-bucket
    • Triaged
    • 1
    • Yes
    • KV 2021-Nov

    Description

      Time taken to perform mcstat collections-details for nodes with lots of collections has regressed since CC. This is not a server side regression (confirmed through tracing, and using new mcstat against a CC cluster).

      Perf for CC mcstat against master KV, 1000 collections:

      27.16 s  100.0%	0 s	 	mcstat (81063)
      27.16 s  100.0%	0 s	 	 Main Thread  0x223a700
      27.15 s   99.9%	0 s	 	  start
      27.15 s   99.9%	0 s	 	   main
      27.14 s   99.9%	0 s	 	    request_stat(MemcachedConnection&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
      27.10 s   99.7%	35.00 ms	 	     MemcachedConnection::stats(std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::function<std::__1::vector<std::__1::unique_ptr<FrameInfo, std::__1::default_delete<FrameInfo> >, std::__1::allocator<std::__1::unique_ptr<FrameInfo, std::__1::default_delete<FrameInfo> > > > ()>)
      15.83 s   58.2%	57.00 ms	 	      MemcachedConnection::recvResponse(BinprotResponse&)
      15.51 s   57.1%	165.00 ms	 	       MemcachedConnection::recvFrame(Frame&)
      7.65 s   28.1%	5.00 ms	 	        MemcachedConnection::read(Frame&, unsigned long)
      7.62 s   28.0%	129.00 ms	 	         MemcachedConnection::readPlain(Frame&, unsigned long)
      
      

      master mcstat, same node:

      1.61 min  100.0%	0 s	 	mcstat (80646)
      1.61 min  100.0%	0 s	 	 Main Thread  0x2235033
      1.61 min   99.9%	0 s	 	  start
      1.61 min   99.9%	0 s	 	   main
      1.61 min   99.9%	0 s	 	    request_stat(MemcachedConnection&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
      1.61 min   99.9%	48.00 ms	 	     MemcachedConnection::stats(std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::function<std::__1::vector<std::__1::unique_ptr<FrameInfo, std::__1::default_delete<FrameInfo> >, std::__1::allocator<std::__1::unique_ptr<FrameInfo, std::__1::default_delete<FrameInfo> > > > ()>)
      1.36 min   84.7%	51.00 ms	 	      MemcachedConnection::recvResponse(BinprotResponse&, cb::mcbp::ClientOpcode, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >)
      1.36 min   84.2%	54.00 ms	 	       MemcachedConnection::recvFrame(Frame&, cb::mcbp::ClientOpcode, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >)
      1.31 min   81.1%	55.00 ms	 	        AsyncReadCallback::drain(unsigned long)
      1.31 min   81.1%	1.31 min	 	         _platform_memmove$VARIANT$Haswell
      
      

      AsyncReadCallback::drain

      void drain(size_t nb) {
              if ((available - nb) > 0) {
                  std::memmove(backing.data(), backing.data() + nb, available - nb);
              }
              available -= nb;
          }
      

      A lot of time is spent shuffling data as bytes are consumed from the backing vector.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Client connection moved to folly::IOBuf, time taken for collection-details back to a value comparable to CC.

            james.harrison James Harrison added a comment - Client connection moved to folly::IOBuf , time taken for collection-details back to a value comparable to CC.

            Build couchbase-server-7.1.0-1660 contains kv_engine commit 5438820 with commit message:
            MB-49388: Avoid repeated memmove in AsyncReadCallback

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1660 contains kv_engine commit 5438820 with commit message: MB-49388 : Avoid repeated memmove in AsyncReadCallback

            People

              james.harrison James Harrison
              james.harrison James Harrison
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty