Uploaded image for project: 'Couchbase Gateway'
  1. Couchbase Gateway
  2. CBG-233

Improve efficiency of pending sequences processing

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.2
    • Fix Version/s: 2.5.0, 2.1.3
    • Component/s: SyncGateway
    • Security Level: Public
    • Labels:
      None

      Description

      • Picture a scenario where there's a very large list of sequences pending, possibly through tweaking max_*_pending.
      • Eventually, the magic doc we've all been waiting for comes along, and everything files neatly out of the pending list and into the cache.
      • That process seems to get slower and slower, and we're spending most of our time in a call to base.Set.Union(), and within that, Set.Copy().
      • Trouble is, as we iterate over everything in the pending list (which we expect to be reasonably sized) we union the channels with everything else we've seen on this iteration of _addPendingLogs(). If we're "unpending" millions of docs with millions of channels, those union calls are going to get really costly...

      Now, obviously, as mentioned, we don't really expect the list to be that big on each iteration of this, but I suppose it could be similarly possible if you had a more modest number of docs with hundreds/thousands of unique channels each (in my case, it's 3 channels per doc, 1 unique).

      It looks like (credit to Adam Fraser for this bit!) the behaviour of Set.Union() here is much more than what we actually need - for this use case we don't need to be doing the Copy() within it.

      Rather than refactor Set itself, for this particular use in _addPendingLogs() we might be able to simply use a map and then wrap that in a Set to return (or potentially some Golang magic that's beyond me!).

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          adamf Adam Fraser added a comment -

          There are a lot of places in cache processing where processEntry can trigger a Union.  Instead of trying to handle each of those, added an Update method on Set that can be used in scenarios where we don't need to return a new set.

          Show
          adamf Adam Fraser added a comment - There are a lot of places in cache processing where processEntry can trigger a Union.  Instead of trying to handle each of those, added an Update method on Set that can be used in scenarios where we don't need to return a new set.
          Hide
          build-team Couchbase Build Team added a comment -

          Build sync_gateway-2.5.0-222 contains sync_gateway commit a98bafe with commit message:
          CBG-233 Use Update when Set copy is not required (#3967)

          Show
          build-team Couchbase Build Team added a comment - Build sync_gateway-2.5.0-222 contains sync_gateway commit a98bafe with commit message: CBG-233 Use Update when Set copy is not required (#3967)
          Hide
          build-team Couchbase Build Team added a comment -

          Build sync_gateway-2.1.2.1-3 contains sync_gateway commit c7fd980 with commit message:
          Backport CBG-233 to 2.1.2.1 (#3979)

          Show
          build-team Couchbase Build Team added a comment - Build sync_gateway-2.1.2.1-3 contains sync_gateway commit c7fd980 with commit message: Backport CBG-233 to 2.1.2.1 (#3979)
          Hide
          build-team Couchbase Build Team added a comment -

          Build sync_gateway-2.1.2.1-3 contains sync_gateway commit c7fd980 with commit message:
          Backport CBG-233 to 2.1.2.1 (#3979)

          Show
          build-team Couchbase Build Team added a comment - Build sync_gateway-2.1.2.1-3 contains sync_gateway commit c7fd980 with commit message: Backport CBG-233 to 2.1.2.1 (#3979)
          Hide
          daniel.petersen Daniel Petersen added a comment -

          had to reopen to remove releasenote label

          Show
          daniel.petersen Daniel Petersen added a comment - had to reopen to remove releasenote label

            People

            • Assignee:
              The One The One
              Reporter:
              James Flather James Flather
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty

                  Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.