Uploaded image for project: 'Couchbase Gateway'
  1. Couchbase Gateway
  2. CBG-278

Sync Gateway panic when stopping 2.0 pull replication with attachments

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 2.5.0
    • 2.5.0, 2.0.0, 2.1.0, 2.1.1, 2.1.2, 2.1.3
    • SyncGateway
    • Security Level: Public
    • None
    • 2

    Description

      Description

      When stopping a 2.0 pull replication from the client side with attachments, Sync Gateway recovers from a blip handler panic, but then tries to double-close the blip sync context which causes another unrecovered panic, taking down the Sync Gateway process.

      This has been present since 2.0, but only occurs when aborting a replication whilst pulling a document containing attachment digests. This is also less of a concern when Sync Gateway runs under a service, as a crashed service will auto-restart pretty quickly.

      Fix versions

      • Iridium: Fixed in Iridium/2.5.0 (as of build 260)
      • 2.1.3: Not in
      • 2.1.x: Should be backported in post-2.1.3 release (CBG-289)

      Suggested fix

      Make blipSyncContext.close() safe to call multiple times (via sync.Once)

      Repro

      Reproduced 3 out of 3 times on master with these steps:

      • Insert 100,000 docs into SG with attachments
      • Start a pull replication from a real device
      • Once revs are being sent to the client, stop replication from the client
      • Observe following crash:

      2019-03-11T12:06:07.481Z [WRN] [7b06ccd5] PANIC handling 'changes' response: Can't get response before message has been sent
      		 
      goroutine 214 [running]:
      		 
      runtime/debug.Stack(0xc0145f76b8, 0x480d4a0, 0x4a97250)
      		 
          /usr/local/Cellar/go/1.12/libexec/src/runtime/debug/stack.go:24 +0x9d
      		 
      github.com/couchbase/sync_gateway/rest.(*blipHandler).handleChangesResponse.func1(0xc002d89080)
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:452 +0x6e
      		 
      panic(0x480d4a0, 0x4a97250)
      		 
          /usr/local/Cellar/go/1.12/libexec/src/runtime/panic.go:522 +0x1b5
      		 
      github.com/couchbase/go-blip.(*Message).Response(0xc0049c1440, 0x0)
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/go-blip/message.go:214 +0x281
      		 
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties(0xc002d89080, 0xc008479ce0, 0xc005222540, 0x0, 0x0, 0x0, 0x26c, 0x0, 0x0, 0x0, ...)
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:773 +0x55c
      		 
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevision(0xc002d89080, 0xc008479ce0, 0xc005222540, 0x0, 0x0, 0x0, 0x26c, 0x0, 0x0, 0x0, ...)
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:724 +0x53c
      		 
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevOrNorev(0xc002d89080, 0xc005222540, 0x0, 0x0, 0x0, 0x26c, 0x0, 0x0, 0x0, 0x0, ...)
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:683 +0x205
      		 
      github.com/couchbase/sync_gateway/rest.(*blipHandler).handleChangesResponse(0xc002d89080, 0xc005222540, 0xc0035c0480, 0xc0143e8000, 0xc8, 0xc8, 0xbf19b19dea7f1eb0, 0x13cdd8a338, 0x5263ea0)
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:524 +0xa09
      		 
      created by github.com/couchbase/sync_gateway/rest.(*blipHandler).sendBatchOfChanges
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:435 +0x613
      		 
       -- rest.(*blipHandler).handleChangesResponse.func1() at blip_sync.go:452
      		 
      2019-03-11T12:06:07.480Z [WRN] [7b06ccd5] PANIC handling 'sendRevision' response: Can't get response before message has been sent
      		 
      goroutine 668 [running]:
      		 
      runtime/debug.Stack(0xc014433e78, 0x480d4a0, 0x4a97250)
      		 
          /usr/local/Cellar/go/1.12/libexec/src/runtime/debug/stack.go:24 +0x9d
      		 
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties.func1.1(0xc002d89080)
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:761 +0x6e
      		 
      panic(0x480d4a0, 0x4a97250)
      		 
          /usr/local/Cellar/go/1.12/libexec/src/runtime/panic.go:522 +0x1b5
      		 
      github.com/couchbase/go-blip.(*Message).Response(0xc0012b0d80, 0x0)
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/go-blip/message.go:214 +0x281
      		 
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties.func1(0xc002d89080, 0xc009602b20, 0x1, 0x1, 0xc000010cf8)
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:766 +0x9f
      		 
      created by github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:758 +0x7af
      		 
       -- rest.(*blipHandler).sendRevisionWithProperties.func1.1() at blip_sync.go:761
      		 
      2019-03-11T12:06:07.481Z [WRN] [7b06ccd5] PANIC handling 'sendRevision' response: Can't get response before message has been sent
      		 
      goroutine 710 [running]:
      		 
      runtime/debug.Stack(0xc0157c2e78, 0x480d4a0, 0x4a97250)
      		 
          /usr/local/Cellar/go/1.12/libexec/src/runtime/debug/stack.go:24 +0x9d
      		 
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties.func1.1(0xc002d89080)
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:761 +0x6e
      		 
      panic(0x480d4a0, 0x4a97250)
      		 
          /usr/local/Cellar/go/1.12/libexec/src/runtime/panic.go:522 +0x1b5
      		 
      github.com/couchbase/go-blip.(*Message).Response(0xc002d91170, 0x0)
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/go-blip/message.go:214 +0x281
      		 
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties.func1(0xc002d89080, 0xc00aa50e50, 0x1, 0x1, 0xc001b708a8)
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:766 +0x9f
      		 
      created by github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:758 +0x7af
      		 
       -- rest.(*blipHandler).sendRevisionWithProperties.func1.1() at blip_sync.go:761
      		 
      panic: Can't get response before message has been sent [recovered]
      		 
          panic: close of closed channel
      		 
       
      		 
      goroutine 668 [running]:
      		 
      github.com/couchbase/sync_gateway/rest.(*blipSyncContext).close(0xc0050c32c0)
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:204 +0x39
      		 
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties.func1.1(0xc002d89080)
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:762 +0x188
      		 
      panic(0x480d4a0, 0x4a97250)
      		 
          /usr/local/Cellar/go/1.12/libexec/src/runtime/panic.go:522 +0x1b5
      		 
      github.com/couchbase/go-blip.(*Message).Response(0xc0012b0d80, 0x0)
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/go-blip/message.go:214 +0x281
      		 
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties.func1(0xc002d89080, 0xc009602b20, 0x1, 0x1, 0xc000010cf8)
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:766 +0x9f
      		 
      created by github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties
      		 
          /Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:758 +0x7af

      When the docs don't have attachments, we get a panic, but we recover and don't crash SG (because the attachment handling goroutine hasn't been started):

      2019-03-11T12:29:20.344Z [WRN] [768e6896] PANIC sending changes: Can't get response before message has been sent
      		 
      goroutine 57 [running]:
      		 
      runtime/debug.Stack(0xc0006355e8, 0x480d4a0, 0x4a97250)
      		 
      	/usr/local/Cellar/go/1.12/libexec/src/runtime/debug/stack.go:24 +0x9d
      		 
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendChanges.func1(0xc0099caba0)
      		 
      	/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:362 +0x6e
      		 
      panic(0x480d4a0, 0x4a97250)
      		 
      	/usr/local/Cellar/go/1.12/libexec/src/runtime/panic.go:522 +0x1b5
      		 
      github.com/couchbase/go-blip.(*Message).Response(0xc0016bc510, 0x0)
      		 
      	/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/go-blip/message.go:214 +0x281
      		 
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendBatchOfChanges(0xc0099caba0, 0xc00a1c7b00, 0xc01430b300, 0x14, 0xc8)
      		 
      	/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:435 +0x589
      		 
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendChanges.func2(0x1)
      		 
      	/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:390 +0x73
      		 
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendChanges.func3(0x0, 0x0, 0x0, 0x0, 0x0)
      		 
      	/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:411 +0x538
      		 
      github.com/couchbase/sync_gateway/rest.generateBlipSyncChanges(0xc0098ab380, 0xc0063ffc50, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
      		 
      	/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/changes_api.go:454 +0x14d
      		 
      github.com/couchbase/sync_gateway/rest.(*blipHandler).sendChanges(0xc0099caba0, 0xc00a1c7b00, 0xc0058df5f0)
      		 
      	/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:395 +0x4a3
      		 
      github.com/couchbase/sync_gateway/rest.(*blipHandler).handleSubChanges.func1(0xc0099caba0, 0xc005982f30, 0xc0058df5f0)
      		 
      	/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:351 +0x181
      		 
      created by github.com/couchbase/sync_gateway/rest.(*blipHandler).handleSubChanges
      		 
      	/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:336 +0x4ae
      		 
       -- rest.(*blipHandler).sendChanges.func1() at blip_sync.go:362
      		 
      2019-03-11T12:29:20.425Z [DBG] SyncMsg+: c:[768e6896] #2: Type:subChanges   --> Time:1m1.37943449s User:admin

      Attachments

        Issue Links

          backports to

          Task - A task that needs to be done. CBG-289 Backport CBG-278 to 2.1.4

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ben.brooks Ben Brooks
              ben.brooks Ben Brooks
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty