Details
-
Bug
-
Resolution: Fixed
-
Major
-
2.5.0, 2.0.0, 2.1.0, 2.1.1, 2.1.2, 2.1.3
-
Security Level: Public
-
None
-
2
Description
Description
When stopping a 2.0 pull replication from the client side with attachments, Sync Gateway recovers from a blip handler panic, but then tries to double-close the blip sync context which causes another unrecovered panic, taking down the Sync Gateway process.
This has been present since 2.0, but only occurs when aborting a replication whilst pulling a document containing attachment digests. This is also less of a concern when Sync Gateway runs under a service, as a crashed service will auto-restart pretty quickly.
Fix versions
- Iridium: Fixed in Iridium/2.5.0 (as of build 260)
- 2.1.3: Not in
- 2.1.x: Should be backported in post-2.1.3 release (
CBG-289)
Suggested fix
Make blipSyncContext.close() safe to call multiple times (via sync.Once)
Repro
Reproduced 3 out of 3 times on master with these steps:
- Insert 100,000 docs into SG with attachments
- Start a pull replication from a real device
- Once revs are being sent to the client, stop replication from the client
- Observe following crash:
2019-03-11T12:06:07.481Z [WRN] [7b06ccd5] PANIC handling 'changes' response: Can't get response before message has been sent
|
goroutine 214 [running]:
|
runtime/debug.Stack(0xc0145f76b8, 0x480d4a0, 0x4a97250)
|
/usr/local/Cellar/go/1.12/libexec/src/runtime/debug/stack.go:24 +0x9d
|
github.com/couchbase/sync_gateway/rest.(*blipHandler).handleChangesResponse.func1(0xc002d89080)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:452 +0x6e
|
panic(0x480d4a0, 0x4a97250)
|
/usr/local/Cellar/go/1.12/libexec/src/runtime/panic.go:522 +0x1b5
|
github.com/couchbase/go-blip.(*Message).Response(0xc0049c1440, 0x0)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/go-blip/message.go:214 +0x281
|
github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties(0xc002d89080, 0xc008479ce0, 0xc005222540, 0x0, 0x0, 0x0, 0x26c, 0x0, 0x0, 0x0, ...)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:773 +0x55c
|
github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevision(0xc002d89080, 0xc008479ce0, 0xc005222540, 0x0, 0x0, 0x0, 0x26c, 0x0, 0x0, 0x0, ...)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:724 +0x53c
|
github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevOrNorev(0xc002d89080, 0xc005222540, 0x0, 0x0, 0x0, 0x26c, 0x0, 0x0, 0x0, 0x0, ...)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:683 +0x205
|
github.com/couchbase/sync_gateway/rest.(*blipHandler).handleChangesResponse(0xc002d89080, 0xc005222540, 0xc0035c0480, 0xc0143e8000, 0xc8, 0xc8, 0xbf19b19dea7f1eb0, 0x13cdd8a338, 0x5263ea0)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:524 +0xa09
|
created by github.com/couchbase/sync_gateway/rest.(*blipHandler).sendBatchOfChanges
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:435 +0x613
|
-- rest.(*blipHandler).handleChangesResponse.func1() at blip_sync.go:452
|
2019-03-11T12:06:07.480Z [WRN] [7b06ccd5] PANIC handling 'sendRevision' response: Can't get response before message has been sent
|
goroutine 668 [running]:
|
runtime/debug.Stack(0xc014433e78, 0x480d4a0, 0x4a97250)
|
/usr/local/Cellar/go/1.12/libexec/src/runtime/debug/stack.go:24 +0x9d
|
github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties.func1.1(0xc002d89080)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:761 +0x6e
|
panic(0x480d4a0, 0x4a97250)
|
/usr/local/Cellar/go/1.12/libexec/src/runtime/panic.go:522 +0x1b5
|
github.com/couchbase/go-blip.(*Message).Response(0xc0012b0d80, 0x0)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/go-blip/message.go:214 +0x281
|
github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties.func1(0xc002d89080, 0xc009602b20, 0x1, 0x1, 0xc000010cf8)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:766 +0x9f
|
created by github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:758 +0x7af
|
-- rest.(*blipHandler).sendRevisionWithProperties.func1.1() at blip_sync.go:761
|
2019-03-11T12:06:07.481Z [WRN] [7b06ccd5] PANIC handling 'sendRevision' response: Can't get response before message has been sent
|
goroutine 710 [running]:
|
runtime/debug.Stack(0xc0157c2e78, 0x480d4a0, 0x4a97250)
|
/usr/local/Cellar/go/1.12/libexec/src/runtime/debug/stack.go:24 +0x9d
|
github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties.func1.1(0xc002d89080)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:761 +0x6e
|
panic(0x480d4a0, 0x4a97250)
|
/usr/local/Cellar/go/1.12/libexec/src/runtime/panic.go:522 +0x1b5
|
github.com/couchbase/go-blip.(*Message).Response(0xc002d91170, 0x0)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/go-blip/message.go:214 +0x281
|
github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties.func1(0xc002d89080, 0xc00aa50e50, 0x1, 0x1, 0xc001b708a8)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:766 +0x9f
|
created by github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:758 +0x7af
|
-- rest.(*blipHandler).sendRevisionWithProperties.func1.1() at blip_sync.go:761
|
panic: Can't get response before message has been sent [recovered]
|
panic: close of closed channel
|
|
goroutine 668 [running]:
|
github.com/couchbase/sync_gateway/rest.(*blipSyncContext).close(0xc0050c32c0)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:204 +0x39
|
github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties.func1.1(0xc002d89080)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:762 +0x188
|
panic(0x480d4a0, 0x4a97250)
|
/usr/local/Cellar/go/1.12/libexec/src/runtime/panic.go:522 +0x1b5
|
github.com/couchbase/go-blip.(*Message).Response(0xc0012b0d80, 0x0)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/go-blip/message.go:214 +0x281
|
github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties.func1(0xc002d89080, 0xc009602b20, 0x1, 0x1, 0xc000010cf8)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:766 +0x9f
|
created by github.com/couchbase/sync_gateway/rest.(*blipHandler).sendRevisionWithProperties
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:758 +0x7af
|
When the docs don't have attachments, we get a panic, but we recover and don't crash SG (because the attachment handling goroutine hasn't been started):
2019-03-11T12:29:20.344Z [WRN] [768e6896] PANIC sending changes: Can't get response before message has been sent
|
goroutine 57 [running]:
|
runtime/debug.Stack(0xc0006355e8, 0x480d4a0, 0x4a97250)
|
/usr/local/Cellar/go/1.12/libexec/src/runtime/debug/stack.go:24 +0x9d
|
github.com/couchbase/sync_gateway/rest.(*blipHandler).sendChanges.func1(0xc0099caba0)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:362 +0x6e
|
panic(0x480d4a0, 0x4a97250)
|
/usr/local/Cellar/go/1.12/libexec/src/runtime/panic.go:522 +0x1b5
|
github.com/couchbase/go-blip.(*Message).Response(0xc0016bc510, 0x0)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/go-blip/message.go:214 +0x281
|
github.com/couchbase/sync_gateway/rest.(*blipHandler).sendBatchOfChanges(0xc0099caba0, 0xc00a1c7b00, 0xc01430b300, 0x14, 0xc8)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:435 +0x589
|
github.com/couchbase/sync_gateway/rest.(*blipHandler).sendChanges.func2(0x1)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:390 +0x73
|
github.com/couchbase/sync_gateway/rest.(*blipHandler).sendChanges.func3(0x0, 0x0, 0x0, 0x0, 0x0)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:411 +0x538
|
github.com/couchbase/sync_gateway/rest.generateBlipSyncChanges(0xc0098ab380, 0xc0063ffc50, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/changes_api.go:454 +0x14d
|
github.com/couchbase/sync_gateway/rest.(*blipHandler).sendChanges(0xc0099caba0, 0xc00a1c7b00, 0xc0058df5f0)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:395 +0x4a3
|
github.com/couchbase/sync_gateway/rest.(*blipHandler).handleSubChanges.func1(0xc0099caba0, 0xc005982f30, 0xc0058df5f0)
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:351 +0x181
|
created by github.com/couchbase/sync_gateway/rest.(*blipHandler).handleSubChanges
|
/Users/benbrooks/dev/sync_gateway/master/godeps/src/github.com/couchbase/sync_gateway/rest/blip_sync.go:336 +0x4ae
|
-- rest.(*blipHandler).sendChanges.func1() at blip_sync.go:362
|
2019-03-11T12:29:20.425Z [DBG] SyncMsg+: c:[768e6896] #2: Type:subChanges --> Time:1m1.37943449s User:admin
|
Attachments
Issue Links
- backports to
-
CBG-289 Backport CBG-278 to 2.1.4
- Closed