Uploaded image for project: 'Couchbase Gateway'
  1. Couchbase Gateway
  2. CBG-1113

CBL1.x/Websocket based changes feeds leak goroutine on disconnect

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 2.0.0
    • 3.0
    • SyncGateway
    • Security Level: Public
    • CBG Sprint 59, CBG Sprint 60
    • 5

    Description

      Issuing a websocket-based changes feed and disconnecting causes the "SimpleMultiChangesFeed" goroutine to leak, even after the websocket request has timed out due to the 5 minute heartbeat.

      With enough clients over a long enough period, this can result in a leak of a significant amount of memory and can cause OOMs.

      This seems to have been the case since at least CBL 1.5.3, and is mostly caused by CBL 1.x clients, which use the websocket changes API. CBL 2.x uses a different API, and so does not cause this issue.

       

      Next steps before fix:

      • Evaluate whether a "clean" disconnect still causes this.
        • Repro'd with websocket command line tool and ^C to close the request, which is an "unclean" disconnect, which relies on the 5 minute heartbeat to actually timeout the request.
          • $ echo "{}" | wsd -url='ws://127.0.0.1:4985/db1/_changes?since=0&feed=websocket&continuous=true&filter=sync_gateway/bychannel&channels=abc,def'
            connecting to ws://127.0.0.1:4985/db1/_changes?since=0&feed=websocket&continuous=true&filter=sync_gateway/bychannel&channels=abc,def from http://localhost/...
            successfully connected to ws://127.0.0.1:4985/db1/_changes?since=0&feed=websocket&continuous=true&filter=sync_gateway/bychannel&channels=abc,def
            < []
            > ^C

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            ben.brooks Ben Brooks created issue -
            ben.brooks Ben Brooks made changes -
            Field Original Value New Value
            Labels backport-candidate
            adamf Adam Fraser made changes -
            Fix Version/s Lithium [ 16180 ]
            adamf Adam Fraser made changes -
            Sprint CBG Sprint 59 [ 1263 ]
            adamf Adam Fraser made changes -
            Rank Ranked lower
            adamf Adam Fraser made changes -
            Story Points 5
            ben.brooks Ben Brooks made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            ben.brooks Ben Brooks made changes -
            Rank Ranked higher
            Automated transition triggered when Ben Brooks created pull request #4818 in GitHub -
            Status In Progress [ 3 ] In Review [ 10107 ]
            adamf Adam Fraser made changes -
            Sprint CBG Sprint 59 [ 1263 ] CBG Sprint 59, CBG Sprint 60 [ 1263, 1281 ]

            Build sync_gateway-3.0.0-29 contains sync_gateway commit a70d9fa with commit message:
            CBG-1113 - Notify termination for waiting ChangeWaiter triggered by Admins/Guest (#4818)

            build-team Couchbase Build Team added a comment - Build sync_gateway-3.0.0-29 contains sync_gateway commit a70d9fa with commit message: CBG-1113 - Notify termination for waiting ChangeWaiter triggered by Admins/Guest (#4818)
            Automated transition triggered when Adam Fraser merged pull request #4818 in GitHub -
            Resolution Fixed [ 1 ]
            Status In Review [ 10107 ] Resolved [ 5 ]
            ben.brooks Ben Brooks made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            ben.brooks Ben Brooks made changes -
            Link This issue is cloned by CBG-1723 [ CBG-1723 ]
            ben.brooks Ben Brooks made changes -
            Link This issue backports to CBG-1723 [ CBG-1723 ]

            Build sync_gateway-2.8.3-7 contains sync_gateway commit 65b9d2b with commit message:
            CBG-1723 Backport CBG-1113 - Notify termination for waiting ChangeWaiter triggered by Admins/Guest (#4818) (#5282)

            build-team Couchbase Build Team added a comment - Build sync_gateway-2.8.3-7 contains sync_gateway commit 65b9d2b with commit message: CBG-1723 Backport CBG-1113 - Notify termination for waiting ChangeWaiter triggered by Admins/Guest (#4818) (#5282)

            People

              ben.brooks Ben Brooks
              ben.brooks Ben Brooks
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty