Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Cannot Reproduce
Priority: Major
Fix Version/s: postHelium
Affects Version/s: 3.0.3
Component/s: SyncGateway
Security Level: Public
Labels:
None

CVSS/Severity:

High

Description

Something really weird suddenly begun to occur to a two-weeks running SG 3.0.3:

suddenly only some admin API get stuck (no response in minutes), like /db/_blipsync and /db/_user (apparently having in common some access to "_sync:user" documents, maybe close to CBG-2057's domain ?)
a costant sequence of messages (apparently noticeably more frequent than under release 3.0.0 ) keeps on being logged, like

Sync+: c:[775561af] PANIC handling BLIP request MSG#5!

      [DBG] Changes+: c:[1893d225] MultiChangesFeed waiting... <ud>  (to sguser@6f2ded60-ca99-11eb-a05c-310679b4afe0)</ud>

in particular,

for the websocket session [KKKK] for user "sguser@ABCD" (replicating channels ["Y"]), suddenly a sequence of

"MultiChangesFeed waiting..."

messages start to continuously appear in SG logs;

then I post a change of document X ("channels":["Y"]) by public REST API;
new content of document X appears on Couchbase cluster, but mobile client is still holding its old revision;
then I modify document X on Couchbase cluster's console, but mobile client keeps on holding its old revision, whereas SG seem to log DCP-import of a new revision of X;
then I force the mobile client opening of a new websocket, then SG receives HTTPS request with user "sguser@ABCD" and the connection is upgraded to websocket [HHHH];
SG immediately logs it has "Multichannel changes"-sent the new revision of X to the [HHHH] session;

with reference to the same scenario under item 3 above, in other cases

websocket [KKKK] is being held by regular websocket PING and PONG after initial exchanges between mobile client and SG;
new content of document X is DCP-imported by SG;
rather than "Multichannel changes"-sending it to session [KKKK], SG closes the websocket;
then (after a few seconds) the mobile client opens session [HHHH] and SG sends the updated document X

a few seconds after SG reboot (e.g., at 2022-07-04T11:54:25Z) two subsequent automatic reboot are triggered, and messages are logged (in sg_debug) like the following ones:

2022-07-04T11:57:27.913Z [DBG] WSFrame+: c:#044 Received frame: MSG#9!~ (flags=   11000, length=115)

2022-07-04T11:57:27.913Z [DBG] WS+: c:#044 Incoming BLIP Request: MSG#9!~

2022-07-04T11:57:27.913Z [INF] SyncMsg: c:[884b54c] #9: Type:proposeChanges #Changes: 1

2022-07-04T11:57:27.913Z [INF] Sync: c:[884b54c] Database bucket closed underneath request MSG#9!~ - asking client to reconnect

2022-07-04T11:57:27.913Z [DBG] Sync+: c:[884b54c] PANIC handling BLIP request MSG#9!~: runtime error: invalid memory address or nil pointer dereference

goroutine 24329 [running]:

runtime/debug.Stack(0x131dab8, 0xc022965440, 0x1500000003)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/go1.16.6/src/runtime/debug/stack.go:24 +0x9f

github.com/couchbase/sync_gateway/db.(*BlipSyncContext).register.func1.1(0xc02297eea0, 0xc02495e360)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/blip_sync_context.go:163 +0xf9

panic(0x1018460, 0x1a63170)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/go1.16.6/src/runtime/panic.go:965 +0x1b9

github.com/couchbase/sync_gateway/db.(*DatabaseContext).GetDocWithXattr(0xc0002e2c00, 0xc020f24150, 0x2c, 0x1157800, 0xc05edba9c0, 0xc00c169a00, 0x4fcc6b, 0x4684a0)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/crud.go:112 +0x5c

github.com/couchbase/sync_gateway/db.(*DatabaseContext).GetDocumentWithRaw(0xc0002e2c00, 0xc020f24150, 0x2c, 0x0, 0x0, 0x0, 0xc00c169ac0, 0x470ffb)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/crud.go:59 +0x8d

github.com/couchbase/sync_gateway/db.(*DatabaseContext).GetDocument(...)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/crud.go:48

github.com/couchbase/sync_gateway/db.(*Database).CheckProposedRev(0xc022374630, 0xc020f24150, 0x2c, 0xc020f24180, 0x2a, 0x0, 0x0, 0x3, 0x44, 0xc059a3abd8)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/crud.go:2416 +0x65

github.com/couchbase/sync_gateway/db.(*blipHandler).handleProposeChanges(0xc01e5fe0c0, 0xc02495e360, 0x0, 0x0)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/blip_handler.go:663 +0x55b

github.com/couchbase/sync_gateway/db.(*BlipSyncContext).register.func1(0xc02495e360)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/blip_sync_context.go:188 +0x16c

github.com/couchbase/go-blip.(*Context).dispatchRequest(0xc02294e300, 0xc02495e360, 0xc0229434a0)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/go-blip/context.go:228 +0x136

github.com/couchbase/go-blip.(*receiver).getPendingRequest.func1(0x0, 0x0)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/go-blip/receiver.go:266 +0x45

github.com/couchbase/go-blip.(*Message).asyncRead.func1(0xc09109a008, 0xc02495e360, 0xc01e5fe0a8)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/go-blip/message.go:371 +0xcf

created by github.com/couchbase/go-blip.(*Message).asyncRead

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/go-blip/message.go:357 +0x185

2022-07-04T11:57:27.913Z [DBG] WSFrame+: c:#044 Push ERR#9!

2022-07-04T11:57:27.913Z [DBG] WSFrame+: c:#044 Sending frame: ERR#9! (flags=   10010, size=   94)

2022-07-04T11:57:27.938Z [DBG] WSFrame+: c:#044 receiveLoop stopped

2022-07-04T11:57:27.938Z [DBG] WSFrame+: c:#044 parseLoop stopped

2022-07-04T11:57:27.938Z [DBG] WSFrame+: c:#044 Sender stopped

2022-07-04T11:57:27.989Z [INF] HTTP: c:[884b54c] #044:    --> BLIP+WebSocket connection closed

2022-07-04T11:57:27.989Z [DBG] Changes+: Notifying to check for _changes feed termination

2022-07-04T11:57:27.989Z [DBG] SyncMsg+: c:[884b54c] #2: Type:subChanges   --> Time:48.392277702s

2022-07-04T11:57:28.060Z [INF] Query: N1QL Query("channels") took 46.779366897s

2022-07-04T11:57:28.060Z [INF] Cache:     Got no rows from query for channel:"<ud>Y</ud>"

2022-07-04T11:57:28.060Z [DBG] Cache+:  changesValidFrom (1469807871) < c.validFrom < changesValidTo (1469869076), setting c.validFrom from 1469869076 -> 1469807871 for "<ud>Y</ud>"

2022-07-04T11:57:28.060Z [INF] Cache: c:[4451bf91] GetChangesInChannel("<ud>Y</ud>") --> 0 rows

2022-07-04T11:57:28.060Z [DBG] Changes+: c:[4451bf91] [changesFeed] Found 0 changes for channel "<ud>Y</ud>"

2022-07-04T11:57:28.060Z [DBG] Changes+: c:[f05dea4] [changesFeed] Found 0 changes for channel "<ud>Y</ud>"

2022-07-04T11:57:28.060Z [INF] Query: N1QL Query("channels") took 40.365185397s

2022-07-04T11:57:28.060Z [ERR] Unexpected panic: runtime error: invalid memory address or nil pointer dereference - stopping process

goroutine 19021 [running]:

runtime/debug.Stack(0xc027a0ca08, 0x1018460, 0x1a63170)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/go1.16.6/src/runtime/debug/stack.go:24 +0x9f

github.com/couchbase/sync_gateway/base.FatalPanicHandler()

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/base/util.go:1477 +0x5b

panic(0x1018460, 0x1a63170)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/go1.16.6/src/runtime/panic.go:965 +0x1b9

github.com/couchbase/gocb/v2.(*meterWrapper).ValueRecorder(0x0, 0x118a460, 0x5, 0x118a460, 0x5, 0x0, 0x0, 0x0, 0x0)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/gocb/v2/metrics.go:121 +0x29

github.com/couchbase/gocb/v2.(*meterWrapper).ValueRecord(0x0, 0x118a460, 0x5, 0x118a460, 0x5, 0xc0a8d34a0397181c, 0xe27938fe9, 0x1a7dc20)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/gocb/v2/metrics.go:147 +0x65

panic(0x1018460, 0x1a63170)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/go1.16.6/src/runtime/panic.go:965 +0x1b9

github.com/couchbase/gocb/v2.createSpan(0x0, 0x0, 0x0, 0x0, 0x118a460, 0x5, 0x118a460, 0x5, 0x0, 0x0)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/gocb/v2/tracing.go:117 +0x38

github.com/couchbase/gocb/v2.(*Cluster).Query(0xc0003242c0, 0xc063341b00, 0x407, 0xc027a0d518, 0x0, 0x0, 0x0)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/gocb/v2/cluster_query.go:298 +0x171

github.com/couchbase/sync_gateway/base.(*Collection).runQuery(0xc0006ee000, 0xc063341b00, 0x407, 0xc0633c5518, 0x1e, 0xc0633c5458, 0x2)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/base/collection_n1ql.go:111 +0xa5

github.com/couchbase/sync_gateway/base.(*Collection).Query(0xc0006ee000, 0xc063354000, 0x3df, 0xc0236fd7d0, 0x2, 0xc021f19700, 0xc0236fd7d0, 0xfb7f40, 0xc02ccad170, 0xfb74c0)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/base/collection_n1ql.go:43 +0x385

github.com/couchbase/sync_gateway/db.(*DatabaseContext).N1QLQueryWithStats(0xc0002e2c00, 0x118c258, 0x8, 0xc063354000, 0x3df, 0xc0236fd7d0, 0x2, 0xc063354000, 0x0, 0x0, ...)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/query.go:287 +0x1a4

github.com/couchbase/sync_gateway/db.(*DatabaseContext).QueryChannels(0xc0002e2c00, 0xc01e804760, 0x9, 0x579bce96, 0x579c6c4a, 0x1388, 0x0, 0x4, 0x40fb32, 0xc000047000, ...)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/query.go:396 +0x5f7

github.com/couchbase/sync_gateway/db.(*DatabaseContext).getChangesInChannelFromQuery(0xc0002e2c00, 0xc01e804760, 0x9, 0x579bce96, 0x579c6c4a, 0x1388, 0xc01ee1ee00, 0x0, 0x0, 0x0, ...)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/changes_view.go:106 +0x6a5

github.com/couchbase/sync_gateway/db.(*singleChannelCacheImpl).GetChanges(0xc026e08a90, 0x0, 0x0, 0x579bce95, 0x1388, 0x1000000, 0xc01ee1eea0, 0x0, 0x0, 0x0, ...)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/channel_cache_single.go:418 +0x489

github.com/couchbase/sync_gateway/db.(*Database).changesFeed.func1(0xc06330fe00, 0x0, 0x1388, 0xc06330fe60, 0xc0222d9aa0, 0x1327760, 0xc026e08a90, 0xc06330fda0, 0xc06268bb80, 0x46)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/changes.go:405 +0x838

created by github.com/couchbase/sync_gateway/db.(*Database).changesFeed

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/changes.go:385 +0x1cf

 -- base.FatalPanicHandler() at util.go:1477

2022-07-04T11:57:28.273Z ==== Couchbase Sync Gateway/3.0.3(1;8187d9a) EE ====

2022-07-04T11:57:28.273Z ==== Couchbase Sync Gateway/3.0.3(1;8187d9a) EE ====

2022-07-04T11:57:28.274Z [INF] Loading content from [/home/sync_gateway/sync_gateway.json] ...

2022-07-04T11:57:28.276Z [INF] Config: Starting in persistent mode using config group "default"

2022-07-04T11:57:28.276Z [INF] Logging: Console disabled

2022-07-04T11:57:28.276Z [INF] Logging: Files to /sg-logs/log/

2022-07-04T11:57:28.276Z [INF] Logging: Console level: none

2022-07-04T11:57:28.276Z [INF] Logging: Console keys: [HTTP]

2022-07-04T11:57:28.276Z [INF] Logging: Redaction level: partial

2022-07-04T11:57:28.276Z [INF] requestedSoftFDLimit >= currentHardFdLimit (400000 >= 131070) capping at 131070

2022-07-04T11:57:28.276Z [INF] Configured process to allow 131070 open file descriptors

2022-07-04T11:57:28.276Z [INF] Logging stats with frequency: &{1m0s}

2022-07-04T11:57:28.294Z [DBG] gocb+: SDK Version: gocbcore/v10.0.2

2022-07-04T11:57:28.294Z [DBG] gocb+: Creating new agent: &{BucketName: UserAgent: SeedConfig:{HTTPAddrs:[172.31.0.99:8091 172.31.11.216:8091 172.31.19.255:8091 172.31.20.133:8091 172.31.27.226:8091 1

72.31.32.6:8091 172.31.32.7:8091 172.31.32.8:8091] MemdAddrs:[172.31.0.99:11210 172.31.11.216:11210 172.31.19.255:11210 172.31.20.133:11210 172.31.27.226:11210 172.31.32.6:11210 172.31.32.7:11210 172.

31.32.8:11210]} SecurityConfig:{UseTLS:false TLSRootCAProvider:0xcd30c0 InitialBootstrapNonTLS:false Auth:0xc0003cbc20 AuthMechanisms:[]} CompressionConfig:{Enabled:false DisableDecompression:false Mi

nSize:0 MinRatio:0} ConfigPollerConfig:{HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s} IoConfig:{NetworkType: UseMutationTokens:false UseDurations:false UseOutO

fOrderResponses:false DisableXErrorHello:false DisableJSONHello:false DisableSyncReplicationHello:false EnablePITRHello:false UseCollections:false} KVConfig:{ConnectTimeout:0s PoolSize:0 MaxQueueSize:

0} HTTPConfig:{MaxIdleConns:0 MaxIdleConnsPerHost:0 IdleConnectionTimeout:0s} DefaultRetryStrategy:<nil> CircuitBreakerConfig:{Enabled:false VolumeThreshold:0 ErrorThresholdPercentage:0 SleepWindow:0s

 RollingWindow:0s CompletionCallback:<nil> CanaryTimeout:0s} OrphanReporterConfig:{Enabled:false ReportInterval:0s SampleSize:0} TracerConfig:{Tracer:<nil> NoRootTraceSpans:false} MeterConfig:{Meter:<

nil>}}

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/go-blip/receiver.go:266 +0x45

github.com/couchbase/go-blip.(*Message).asyncRead.func1(0xc0e56ba060, 0xc03c081170, 0xc0585aa138)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/go-blip/message.go:371 +0xcf

created by github.com/couchbase/go-blip.(*Message).asyncRead

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/go-blip/message.go:357 +0x185

2022-07-04T11:56:26.765Z [DBG] WSFrame+: c:#322 Push ERR#4!

2022-07-04T11:56:26.765Z [DBG] WSFrame+: c:#322 Sending frame: ERR#4! (flags=   10010, size=   94)

2022-07-04T11:56:26.786Z [DBG] WSFrame+: c:#322 receiveLoop stopped

2022-07-04T11:56:26.786Z [DBG] WSFrame+: c:#322 parseLoop stopped

2022-07-04T11:56:26.786Z [DBG] WSFrame+: c:#322 Sender stopped

2022-07-04T11:56:26.788Z [INF] HTTP:  #1270: GET /

2022-07-04T11:56:26.788Z [INF] HTTP+: #1270:     --> 200   (0.0 ms)

2022-07-04T11:56:26.837Z [INF] HTTP: c:[57357dfd] #322:    --> BLIP+WebSocket connection closed

2022-07-04T11:56:26.837Z [DBG] Changes+: Notifying to check for _changes feed termination

2022-07-04T11:56:26.837Z [DBG] SyncMsg+: c:[57357dfd] #3: Type:subChanges   --> Time:50.85647133s

2022-07-04T11:56:26.837Z [DBG] Changes+: No new changes to send to change listener.  Waiting for "replica-bucket"'s count to pass 0

2022-07-04T11:56:26.837Z [DBG] Changes+: c:[19688c22] MultiChangesFeed: channels expand to "<ud>Y1:587472808,Y2:587472808,Y:587472808</ud>" ... <ud>  (to sguser@648f24d0-54b1-11ec-9a7f-37100ff96e82)</ud>

2022-07-04T11:56:26.837Z [DBG] Cache+: c:[19688c22] GetCachedChanges("<ud>Y2</ud>", 1469867005) --> nothing cached

2022-07-04T11:56:26.837Z [DBG] Changes+: c:[19688c22] [changesFeed] Found 0 changes for channel "<ud>Y2</ud>"

2022-07-04T11:56:26.837Z [DBG] Cache+: c:[19688c22] GetCachedChanges("<ud>Y1</ud>", 1469867005) --> nothing cached

2022-07-04T11:56:26.837Z [DBG] Changes+: c:[19688c22] [changesFeed] Found 0 changes for channel "<ud>Y1</ud>"

2022-07-04T11:56:26.837Z [DBG] Cache+: c:[19688c22] GetCachedChanges("<ud>Y</ud>", 1469867005) --> nothing cached

2022-07-04T11:56:26.837Z [DBG] Changes+: c:[19688c22] [changesFeed] Found 0 changes for channel "<ud>Y</ud>"

2022-07-04T11:56:26.837Z [DBG] Changes+: c:[19688c22] MultiChangesFeed waiting... <ud>  (to sguser@648f24d0-54b1-11ec-9a7f-37100ff96e82)</ud>

2022-07-04T11:56:26.837Z [DBG] Changes+: No new changes to send to change listener.  Waiting for "replica-bucket"'s count to pass 25

2022-07-04T11:56:26.837Z [DBG] Sync+: c:[19688c22]     Sending 0 changes

2022-07-04T11:56:26.899Z [INF] Query: N1QL Query("channels") took 58.648787733s

2022-07-04T11:56:26.899Z [INF] Cache:     Got no rows from query for channel:"<ud>Y3</ud>"

2022-07-04T11:56:26.899Z [DBG] Cache+:  changesValidFrom (1469863360) < c.validFrom < changesValidTo (1469866684), setting c.validFrom from 1469866684 -> 1469863360 for "<ud>Y3</ud>"

2022-07-04T11:56:26.899Z [INF] Cache: c:[1551afa7] GetChangesInChannel("<ud>Y3</ud>") --> 0 rows

2022-07-04T11:56:26.899Z [DBG] Changes+: c:[1551afa7] [changesFeed] Found 0 changes for channel "<ud>Y3</ud>"

2022-07-04T11:56:26.899Z [INF] Query: N1QL Query("channels") took 50.420249502s

2022-07-04T11:56:26.899Z [ERR] Unexpected panic: runtime error: invalid memory address or nil pointer dereference - stopping process

goroutine 35027 [running]:

runtime/debug.Stack(0xc012c84a08, 0x1018460, 0x1a63170)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/go1.16.6/src/runtime/debug/stack.go:24 +0x9f

github.com/couchbase/sync_gateway/base.FatalPanicHandler()

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/base/util.go:1477 +0x5b

panic(0x1018460, 0x1a63170)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/go1.16.6/src/runtime/panic.go:965 +0x1b9

github.com/couchbase/gocb/v2.(*meterWrapper).ValueRecorder(0x0, 0x118a460, 0x5, 0x118a460, 0x5, 0xc0411e8de0, 0x44e44e, 0xc0411e8ed0, 0x3)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/gocb/v2/metrics.go:121 +0x29

github.com/couchbase/gocb/v2.(*meterWrapper).ValueRecord(0x0, 0x118a460, 0x5, 0x118a460, 0x5, 0xc0a8d33ab59ffe44, 0x1c43bceec4, 0x1a7dc20)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/gocb/v2/metrics.go:147 +0x65

panic(0x1018460, 0x1a63170)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/go1.16.6/src/runtime/panic.go:965 +0x1b9

github.com/couchbase/gocb/v2.createSpan(0x0, 0x0, 0x0, 0x0, 0x118a460, 0x5, 0x118a460, 0x5, 0x0, 0x0)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/gocb/v2/tracing.go:117 +0x38

github.com/couchbase/gocb/v2.(*Cluster).Query(0xc048044000, 0xc050fe7200, 0x407, 0xc012c85518, 0x0, 0x0, 0x0)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/gocb/v2/cluster_query.go:298 +0x171

github.com/couchbase/sync_gateway/base.(*Collection).runQuery(0xc03bcd4460, 0xc050fe7200, 0x407, 0xc0411e9518, 0x1e, 0xc0411e9458, 0x2)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/base/collection_n1ql.go:111 +0xa5

github.com/couchbase/sync_gateway/base.(*Collection).Query(0xc03bcd4460, 0xc057111800, 0x3df, 0xc058404420, 0x2, 0xc057475e00, 0xc058404420, 0xfb7f40, 0xc0395c31c0, 0xfb74c0)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/base/collection_n1ql.go:43 +0x385

github.com/couchbase/sync_gateway/db.(*DatabaseContext).N1QLQueryWithStats(0xc033794000, 0x118c258, 0x8, 0xc057111800, 0x3df, 0xc058404420, 0x2, 0xc057111800, 0x0, 0x0, ...)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/query.go:287 +0x1a4

github.com/couchbase/sync_gateway/db.(*DatabaseContext).QueryChannels(0xc033794000, 0xc053bdfb80, 0x9, 0x579b5ff4, 0x579c6629, 0x1388, 0x0, 0x4, 0x40f8fb, 0x5, ...)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/query.go:396 +0x5f7

github.com/couchbase/sync_gateway/db.(*DatabaseContext).getChangesInChannelFromQuery(0xc033794000, 0xc053bdfb80, 0x9, 0x579b5ff4, 0x579c6629, 0x1388, 0xc046f25600, 0x0, 0x0, 0x0, ...)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/changes_view.go:106 +0x6a5

github.com/couchbase/sync_gateway/db.(*singleChannelCacheImpl).GetChanges(0xc056d3aea0, 0x0, 0x0, 0x579b5ff3, 0x1388, 0x1000000, 0xc046f25680, 0x0, 0x0, 0x0, ...)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/channel_cache_single.go:418 +0x489

github.com/couchbase/sync_gateway/db.(*Database).changesFeed.func1(0xc056ffad20, 0x0, 0x1388, 0xc056ffad80, 0xc04eb8d2c0, 0x1327760, 0xc056d3aea0, 0xc056ffacc0, 0xc0544285f0, 0x46)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/changes.go:405 +0x838

created by github.com/couchbase/sync_gateway/db.(*Database).changesFeed

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/changes.go:385 +0x1cf

 -- base.FatalPanicHandler() at util.go:1477

2022-07-04T11:56:27.270Z ==== Couchbase Sync Gateway/3.0.3(1;8187d9a) EE ====

and just 60 seconds seconds later:

22022-07-04T11:57:27.913Z [DBG] WSFrame+: c:#044 Received frame: MSG#9!~ (flags=   11000, length=115)

2022-07-04T11:57:27.913Z [DBG] WS+: c:#044 Incoming BLIP Request: MSG#9!~

2022-07-04T11:57:27.913Z [INF] SyncMsg: c:[884b54c] #9: Type:proposeChanges #Changes: 1

2022-07-04T11:57:27.913Z [INF] Sync: c:[884b54c] Database bucket closed underneath request MSG#9!~ - asking client to reconnect

2022-07-04T11:57:27.913Z [DBG] Sync+: c:[884b54c] PANIC handling BLIP request MSG#9!~: runtime error: invalid memory address or nil pointer dereference

goroutine 24329 [running]:

runtime/debug.Stack(0x131dab8, 0xc022965440, 0x1500000003)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/go1.16.6/src/runtime/debug/stack.go:24 +0x9f

github.com/couchbase/sync_gateway/db.(*BlipSyncContext).register.func1.1(0xc02297eea0, 0xc02495e360)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/blip_sync_context.go:163 +0xf9

panic(0x1018460, 0x1a63170)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/go1.16.6/src/runtime/panic.go:965 +0x1b9

github.com/couchbase/sync_gateway/db.(*DatabaseContext).GetDocWithXattr(0xc0002e2c00, 0xc020f24150, 0x2c, 0x1157800, 0xc05edba9c0, 0xc00c169a00, 0x4fcc6b, 0x4684a0)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/crud.go:112 +0x5c

github.com/couchbase/sync_gateway/db.(*DatabaseContext).GetDocumentWithRaw(0xc0002e2c00, 0xc020f24150, 0x2c, 0x0, 0x0, 0x0, 0xc00c169ac0, 0x470ffb)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/crud.go:59 +0x8d

github.com/couchbase/sync_gateway/db.(*DatabaseContext).GetDocument(...)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/crud.go:48

github.com/couchbase/sync_gateway/db.(*Database).CheckProposedRev(0xc022374630, 0xc020f24150, 0x2c, 0xc020f24180, 0x2a, 0x0, 0x0, 0x3, 0x44, 0xc059a3abd8)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/crud.go:2416 +0x65

github.com/couchbase/sync_gateway/db.(*blipHandler).handleProposeChanges(0xc01e5fe0c0, 0xc02495e360, 0x0, 0x0)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/blip_handler.go:663 +0x55b

github.com/couchbase/sync_gateway/db.(*BlipSyncContext).register.func1(0xc02495e360)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/blip_sync_context.go:188 +0x16c

github.com/couchbase/go-blip.(*Context).dispatchRequest(0xc02294e300, 0xc02495e360, 0xc0229434a0)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/go-blip/context.go:228 +0x136

github.com/couchbase/go-blip.(*receiver).getPendingRequest.func1(0x0, 0x0)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/go-blip/receiver.go:266 +0x45

github.com/couchbase/go-blip.(*Message).asyncRead.func1(0xc09109a008, 0xc02495e360, 0xc01e5fe0a8)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/go-blip/message.go:371 +0xcf

created by github.com/couchbase/go-blip.(*Message).asyncRead

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/go-blip/message.go:357 +0x185

2022-07-04T11:57:27.913Z [DBG] WSFrame+: c:#044 Push ERR#9!

2022-07-04T11:57:27.913Z [DBG] WSFrame+: c:#044 Sending frame: ERR#9! (flags=   10010, size=   94)

2022-07-04T11:57:27.938Z [DBG] WSFrame+: c:#044 receiveLoop stopped

2022-07-04T11:57:27.938Z [DBG] WSFrame+: c:#044 parseLoop stopped

2022-07-04T11:57:27.938Z [DBG] WSFrame+: c:#044 Sender stopped

2022-07-04T11:57:27.989Z [INF] HTTP: c:[884b54c] #044:    --> BLIP+WebSocket connection closed

2022-07-04T11:57:27.989Z [DBG] Changes+: Notifying to check for _changes feed termination

2022-07-04T11:57:27.989Z [DBG] SyncMsg+: c:[884b54c] #2: Type:subChanges   --> Time:48.392277702s

2022-07-04T11:57:28.060Z [INF] Query: N1QL Query("channels") took 46.779366897s

2022-07-04T11:57:28.060Z [INF] Cache:     Got no rows from query for channel:"<ud>Y</ud>"

2022-07-04T11:57:28.060Z [DBG] Cache+:  changesValidFrom (1469807871) < c.validFrom < changesValidTo (1469869076), setting c.validFrom from 1469869076 -> 1469807871 for "<ud>Y</ud>"

2022-07-04T11:57:28.060Z [INF] Cache: c:[4451bf91] GetChangesInChannel("<ud>Y</ud>") --> 0 rows

2022-07-04T11:57:28.060Z [DBG] Changes+: c:[4451bf91] [changesFeed] Found 0 changes for channel "<ud>Y</ud>"

2022-07-04T11:57:28.060Z [DBG] Changes+: c:[f05dea4] [changesFeed] Found 0 changes for channel "<ud>Y</ud>"

2022-07-04T11:57:28.060Z [INF] Query: N1QL Query("channels") took 40.365185397s

2022-07-04T11:57:28.060Z [ERR] Unexpected panic: runtime error: invalid memory address or nil pointer dereference - stopping process

goroutine 19021 [running]:

runtime/debug.Stack(0xc027a0ca08, 0x1018460, 0x1a63170)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/go1.16.6/src/runtime/debug/stack.go:24 +0x9f

github.com/couchbase/sync_gateway/base.FatalPanicHandler()

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/base/util.go:1477 +0x5b

panic(0x1018460, 0x1a63170)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/go1.16.6/src/runtime/panic.go:965 +0x1b9

github.com/couchbase/gocb/v2.(*meterWrapper).ValueRecorder(0x0, 0x118a460, 0x5, 0x118a460, 0x5, 0x0, 0x0, 0x0, 0x0)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/gocb/v2/metrics.go:121 +0x29

github.com/couchbase/gocb/v2.(*meterWrapper).ValueRecord(0x0, 0x118a460, 0x5, 0x118a460, 0x5, 0xc0a8d34a0397181c, 0xe27938fe9, 0x1a7dc20)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/gocb/v2/metrics.go:147 +0x65

panic(0x1018460, 0x1a63170)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/go1.16.6/src/runtime/panic.go:965 +0x1b9

github.com/couchbase/gocb/v2.createSpan(0x0, 0x0, 0x0, 0x0, 0x118a460, 0x5, 0x118a460, 0x5, 0x0, 0x0)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/gocb/v2/tracing.go:117 +0x38

github.com/couchbase/gocb/v2.(*Cluster).Query(0xc0003242c0, 0xc063341b00, 0x407, 0xc027a0d518, 0x0, 0x0, 0x0)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/gocb/v2/cluster_query.go:298 +0x171

github.com/couchbase/sync_gateway/base.(*Collection).runQuery(0xc0006ee000, 0xc063341b00, 0x407, 0xc0633c5518, 0x1e, 0xc0633c5458, 0x2)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/base/collection_n1ql.go:111 +0xa5

github.com/couchbase/sync_gateway/base.(*Collection).Query(0xc0006ee000, 0xc063354000, 0x3df, 0xc0236fd7d0, 0x2, 0xc021f19700, 0xc0236fd7d0, 0xfb7f40, 0xc02ccad170, 0xfb74c0)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/base/collection_n1ql.go:43 +0x385

github.com/couchbase/sync_gateway/db.(*DatabaseContext).N1QLQueryWithStats(0xc0002e2c00, 0x118c258, 0x8, 0xc063354000, 0x3df, 0xc0236fd7d0, 0x2, 0xc063354000, 0x0, 0x0, ...)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/query.go:287 +0x1a4

github.com/couchbase/sync_gateway/db.(*DatabaseContext).QueryChannels(0xc0002e2c00, 0xc01e804760, 0x9, 0x579bce96, 0x579c6c4a, 0x1388, 0x0, 0x4, 0x40fb32, 0xc000047000, ...)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/query.go:396 +0x5f7

github.com/couchbase/sync_gateway/db.(*DatabaseContext).getChangesInChannelFromQuery(0xc0002e2c00, 0xc01e804760, 0x9, 0x579bce96, 0x579c6c4a, 0x1388, 0xc01ee1ee00, 0x0, 0x0, 0x0, ...)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/changes_view.go:106 +0x6a5

github.com/couchbase/sync_gateway/db.(*singleChannelCacheImpl).GetChanges(0xc026e08a90, 0x0, 0x0, 0x579bce95, 0x1388, 0x1000000, 0xc01ee1eea0, 0x0, 0x0, 0x0, ...)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/channel_cache_single.go:418 +0x489

github.com/couchbase/sync_gateway/db.(*Database).changesFeed.func1(0xc06330fe00, 0x0, 0x1388, 0xc06330fe60, 0xc0222d9aa0, 0x1327760, 0xc026e08a90, 0xc06330fda0, 0xc06268bb80, 0x46)

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/changes.go:405 +0x838

created by github.com/couchbase/sync_gateway/db.(*Database).changesFeed

        /home/couchbase/jenkins/workspace/sgw-unix-build@2/3.0.3/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/changes.go:385 +0x1cf

 -- base.FatalPanicHandler() at util.go:1477

2022-07-04T11:57:28.273Z ==== Couchbase Sync Gateway/3.0.3(1;8187d9a) EE ====

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

CBG-2180-num_repl.png
453 kB
07/Jul/22 7:23 AM

Issue Links

relates to

CBG-2181 In-flight BLIP connections don't get closed if the database disappears

Closed

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

People

Assignee:: The One

Reporter:: Giacomo Mazzini

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 04/Jul/22 8:47 AM

Updated:: 12/May/23 12:03 PM

Resolved:: 12/May/23 12:03 PM

Gerrit Reviews

There are no open Gerrit changes

SG 3.0.3: suddenly hung websocket and "_user" API, panic, automatic reboot

Details

Description

Attachments

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty