Uploaded image for project: 'Couchbase Gateway'
  1. Couchbase Gateway
  2. CBG-278

Sync Gateway panic when stopping 2.0 pull replication with attachments

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.5.0, 2.0.0, 2.1.0, 2.1.1, 2.1.2, 2.1.3
    • Fix Version/s: 2.5.0
    • Component/s: SyncGateway
    • Security Level: Public
    • Labels:
      None
    • Story Points:
      2

      Description

      Description

      When stopping a 2.0 pull replication from the client side with attachments, Sync Gateway recovers from a blip handler panic, but then tries to double-close the blip sync context which causes another unrecovered panic, taking down the Sync Gateway process.

      This has been present since 2.0, but only occurs when aborting a replication whilst pulling a document containing attachment digests. This is also less of a concern when Sync Gateway runs under a service, as a crashed service will auto-restart pretty quickly.

      Fix versions

      • Iridium: Fixed in Iridium/2.5.0 (as of build 260)
      • 2.1.3: Not in
      • 2.1.x: Should be backported in post-2.1.3 release (CBG-289)

      Suggested fix

      Make blipSyncContext.close() safe to call multiple times (via sync.Once)

      Repro

      Reproduced 3 out of 3 times on master with these steps:

      • Insert 100,000 docs into SG with attachments
      • Start a pull replication from a real device
      • Once revs are being sent to the client, stop replication from the client
      • Observe following crash:

      2019-03-11T12:06:07.481Z [WRN] [7b06ccd5] PANIC handling 'changes' response: Can't get response before message has been sent
      goroutine 214 [running]:
      runtime/debug.Stack(0xc0145f76b8, 0x480d4a0, 0x4a97250)
          /usr/local/Cellar/go/1.12/libexec/src/runtime/debug/stack.go:24 +0x9d
      github.com/couchbase/sync_gateway/rest.(*blipHandler).handleChangesResponse.func1(0xc002d89080)
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:452 +0x6e
      panic(0x480d4a0, 0x4a97250)
          /usr/local/Cellar/go/1.12/libexec/src/runtime/panic.go:522 +0x1b5
      github.com/couchbase/go-blip.(*Message).Response(0xc0049c1440, 0x0)
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/go-blip/message.go:214 +0x281
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties(0xc002d89080, 0xc008479ce0, 0xc005222540, 0x0, 0x0, 0x0, 0x26c, 0x0, 0x0, 0x0, ...)
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:773 +0x55c
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevision(0xc002d89080, 0xc008479ce0, 0xc005222540, 0x0, 0x0, 0x0, 0x26c, 0x0, 0x0, 0x0, ...)
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:724 +0x53c
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevOrNorev(0xc002d89080, 0xc005222540, 0x0, 0x0, 0x0, 0x26c, 0x0, 0x0, 0x0, 0x0, ...)
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:683 +0x205
      github.com/couchbase/sync_gateway/rest.(*blipHandler).handleChangesResponse(0xc002d89080, 0xc005222540, 0xc0035c0480, 0xc0143e8000, 0xc8, 0xc8, 0xbf19b19dea7f1eb0, 0x13cdd8a338, 0x5263ea0)
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:524 +0xa09
      created by github.com/couchbase/sync_gateway/rest.(*blipHandler).sendBatchOfChanges
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:435 +0x613
       -- rest.(*blipHandler).handleChangesResponse.func1() at blip_sync.go:452
      2019-03-11T12:06:07.480Z [WRN] [7b06ccd5] PANIC handling 'sendRevision' response: Can't get response before message has been sent
      goroutine 668 [running]:
      runtime/debug.Stack(0xc014433e78, 0x480d4a0, 0x4a97250)
          /usr/local/Cellar/go/1.12/libexec/src/runtime/debug/stack.go:24 +0x9d
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties.func1.1(0xc002d89080)
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:761 +0x6e
      panic(0x480d4a0, 0x4a97250)
          /usr/local/Cellar/go/1.12/libexec/src/runtime/panic.go:522 +0x1b5
      github.com/couchbase/go-blip.(*Message).Response(0xc0012b0d80, 0x0)
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/go-blip/message.go:214 +0x281
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties.func1(0xc002d89080, 0xc009602b20, 0x1, 0x1, 0xc000010cf8)
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:766 +0x9f
      created by github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:758 +0x7af
       -- rest.(*blipHandler).sendRevisionWithProperties.func1.1() at blip_sync.go:761
      2019-03-11T12:06:07.481Z [WRN] [7b06ccd5] PANIC handling 'sendRevision' response: Can't get response before message has been sent
      goroutine 710 [running]:
      runtime/debug.Stack(0xc0157c2e78, 0x480d4a0, 0x4a97250)
          /usr/local/Cellar/go/1.12/libexec/src/runtime/debug/stack.go:24 +0x9d
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties.func1.1(0xc002d89080)
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:761 +0x6e
      panic(0x480d4a0, 0x4a97250)
          /usr/local/Cellar/go/1.12/libexec/src/runtime/panic.go:522 +0x1b5
      github.com/couchbase/go-blip.(*Message).Response(0xc002d91170, 0x0)
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/go-blip/message.go:214 +0x281
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties.func1(0xc002d89080, 0xc00aa50e50, 0x1, 0x1, 0xc001b708a8)
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:766 +0x9f
      created by github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:758 +0x7af
       -- rest.(*blipHandler).sendRevisionWithProperties.func1.1() at blip_sync.go:761
      panic: Can't get response before message has been sent [recovered]
          panic: close of closed channel
       
      goroutine 668 [running]:
      github.com/couchbase/sync_gateway/rest.(*blipSyncContext).close(0xc0050c32c0)
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:204 +0x39
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties.func1.1(0xc002d89080)
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:762 +0x188
      panic(0x480d4a0, 0x4a97250)
          /usr/local/Cellar/go/1.12/libexec/src/runtime/panic.go:522 +0x1b5
      github.com/couchbase/go-blip.(*Message).Response(0xc0012b0d80, 0x0)
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/go-blip/message.go:214 +0x281
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties.func1(0xc002d89080, 0xc009602b20, 0x1, 0x1, 0xc000010cf8)
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:766 +0x9f
      created by github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:758 +0x7af
      

      When the docs don't have attachments, we get a panic, but we recover and don't crash SG (because the attachment handling goroutine hasn't been started):

      2019-03-11T12:29:20.344Z [WRN] [768e6896] PANIC sending changes: Can't get response before message has been sent
      goroutine 57 [running]:
      runtime/debug.Stack(0xc0006355e8, 0x480d4a0, 0x4a97250)
      	/usr/local/Cellar/go/1.12/libexec/src/runtime/debug/stack.go:24 +0x9d
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendChanges.func1(0xc0099caba0)
      	/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:362 +0x6e
      panic(0x480d4a0, 0x4a97250)
      	/usr/local/Cellar/go/1.12/libexec/src/runtime/panic.go:522 +0x1b5
      github.com/couchbase/go-blip.(*Message).Response(0xc0016bc510, 0x0)
      	/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/go-blip/message.go:214 +0x281
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendBatchOfChanges(0xc0099caba0, 0xc00a1c7b00, 0xc01430b300, 0x14, 0xc8)
      	/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:435 +0x589
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendChanges.func2(0x1)
      	/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:390 +0x73
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendChanges.func3(0x0, 0x0, 0x0, 0x0, 0x0)
      	/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:411 +0x538
      github.com/couchbase/sync_gateway/rest.generateBlipSyncChanges(0xc0098ab380, 0xc0063ffc50, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
      	/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/changes_api.go:454 +0x14d
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendChanges(0xc0099caba0, 0xc00a1c7b00, 0xc0058df5f0)
      	/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:395 +0x4a3
      github.com/couchbase/sync_gateway/rest.(*blipHandler).handleSubChanges.func1(0xc0099caba0, 0xc005982f30, 0xc0058df5f0)
      	/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:351 +0x181
      created by github.com/couchbase/sync_gateway/rest.(*blipHandler).handleSubChanges
      	/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:336 +0x4ae
       -- rest.(*blipHandler).sendChanges.func1() at blip_sync.go:362
      2019-03-11T12:29:20.425Z [DBG] SyncMsg+: c:[768e6896] #2: Type:subChanges   --> Time:1m1.37943449s User:admin
      

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            daniel.petersen Daniel Petersen added a comment -

            Release is out. Closing.

            Show
            daniel.petersen Daniel Petersen added a comment - Release is out. Closing.
            Hide
            daniel.petersen Daniel Petersen added a comment -

            had to reopen to remove releasenote label

            Show
            daniel.petersen Daniel Petersen added a comment - had to reopen to remove releasenote label

              People

              • Assignee:
                ben.brooks Ben Brooks
                Reporter:
                ben.brooks Ben Brooks
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty

                    Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.