Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-32430

75% throughput drop on Q1

    XMLWordPrintable

Details

    • Untriaged
    • Impediment
    • Yes

    Description

      In between build 1901 and 1909 N1QL throughput has gone down the drain.
      On iris, Q1 throughput has gone down from 125k to 36k.
      I believe the change is down to

      http://review.couchbase.org/c/102597/4/cbauthimpl/impl.go#287

      Please note that this issue is making n1ql slower than it was in 4.5 and is preventing me from diagnosing MB-32158.

      FYI

      http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=iris_650-1901_access_be02&snapshot=iris_650-1920_access_afb7

      (sorry, I don't have the time to do a 1907 run, but from pipeline-daily, I know things went down in 1909 or prior:

      http://172.23.123.43:8000/getchangelog?product=couchbase-server&fromb=6.5.0-1901&tob=6.5.0-1909
      )

      Any help would be appreciated.

      Attachments

        For Gerrit Dashboard: MB-32430
        # Subject Branch Project Status CR V

        Activity

          marco.greco Marco Greco added a comment -

          Timofey Barmin I'm sure you have a good reason to do a deep copy, although I don't get it, but since you are doing a copy only if things haven't changed, can't you do the copy using a shared lock, so that multiple requests can copy at the same time?

          marco.greco Marco Greco added a comment - Timofey Barmin I'm sure you have a good reason to do a deep copy, although I don't get it, but since you are doing a copy only if things haven't changed, can't you do the copy using a shared lock, so that multiple requests can copy at the same time?
          dfinlay Dave Finlay added a comment -

          Marco Greco: thanks for the investigation this.

          Timofey Barmin: I see what you're trying to do in this patch, but I think we need to rethink it. If no one is modifying the creds DB, maybe we don't need to copy it - maybe conventionally we can assume it to be read only. Otherwise, if it is being modified perhaps we use copy-on-write. Or perhaps if it's only the TLS config piece that gets modified, we split it out. There's probably a bunch of things to be thought about here. If it's going to take some time, it would be good if we can revert the change for now while we figure it out.

          dfinlay Dave Finlay added a comment - Marco Greco : thanks for the investigation this. Timofey Barmin : I see what you're trying to do in this patch, but I think we need to rethink it. If no one is modifying the creds DB, maybe we don't need to copy it - maybe conventionally we can assume it to be read only. Otherwise, if it is being modified perhaps we use copy-on-write. Or perhaps if it's only the TLS config piece that gets modified, we split it out. There's probably a bunch of things to be thought about here. If it's going to take some time, it would be good if we can revert the change for now while we figure it out.
          dfinlay Dave Finlay added a comment -

          Just reverted. Apologies for interrupting you.

          dfinlay Dave Finlay added a comment - Just reverted. Apologies for interrupting you.

          Build couchbase-server-6.5.0-1934 contains cbauth commit 4005ef6 with commit message:
          MB-32430: Revert "Deep copy db instead of shallow copy in fetchDB function"

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-1934 contains cbauth commit 4005ef6 with commit message: MB-32430 : Revert "Deep copy db instead of shallow copy in fetchDB function"
          marco.greco Marco Greco added a comment - Dave Finlay Timofey Barmin much obliged
          wayne Wayne Siu added a comment -

          The performance has been in the 125K range in latest builds (6.5.0-4676).

          Closing the ticket.

          wayne Wayne Siu added a comment - The performance has been in the 125K range in latest builds (6.5.0-4676). Closing the ticket.

          People

            wayne Wayne Siu
            marco.greco Marco Greco
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty