Uploaded image for project: 'Couchbase Gateway'
  1. Couchbase Gateway
  2. CBG-3023

[3.1.1 backport] cbgt cluster connection fails with HTTP polling enabled and TLS disabled

    XMLWordPrintable

Details

    • Task
    • Resolution: Fixed
    • Major
    • 3.1.1
    • 3.1.0
    • SyncGateway
    • Security Level: Public
    • None
    • CBG Sprint 126
    • 1

    Description

      When the cluster does not have available HTTP polling but TLS is disabled, creating a cbgt connect will cause a panic. The underlying problem is that the gocb agent that is created by cbgt has (note difference in NoTLSSeedNode):

      SecurityConfig:{UseTLS:false TLSRootCAProvider:<nil> NoTLSSeedNode:true Auth:0xc037cb1380 AuthMechanisms:[]}

      which causes a panic, our other gocb connections set:

      SecurityConfig:{UseTLS:false TLSRootCAProvider:0xd76a60 NoTLSSeedNode:false Auth:0xc0001dac60 AuthMechanisms:[]}

      Setting UseTLS:false and NoTLSSeedNode:true are incompatible and will cause gocb to panic. The connection would not succeed even if there was no panic.

      To reproduce this problem locally, start a server without TLS and add ?bootstrap_on=cccp to the connection string.

      2023-05-17T09:06:17.690Z [DBG] gocb+: SDK Version: gocbcore/v10.2.2
      		 
      2023-05-17T09:06:17.690Z [DBG] gocb+: Creating new agent: &{BucketName:chat-sync-2 UserAgent:stats SeedConfig:{HTTPAddrs:[] MemdAddrs:[10-123-2-85.cb-dior-dev-srv.dior-cb.svc.cluster.local:11210 10-123-9-148.cb-dior-dev-srv.dior-cb.svc.cluster.local:11210] SRVRecord:0xc037cf9a10} SecurityConfig:{UseTLS:false TLSRootCAProvider:<nil> NoTLSSeedNode:true Auth:0xc037cb1380 AuthMechanisms:[]} CompressionConfig:{Enabled:false DisableDecompression:false MinSize:0 MinRatio:0} ConfigPollerConfig:{HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s} IoConfig:{NetworkType:default UseMutationTokens:false UseDurations:false UseOutOfOrderResponses:false DisableXErrorHello:false DisableJSONHello:false DisableSyncReplicationHello:false EnablePITRHello:false UseCollections:true} KVConfig:{ConnectTimeout:7s ServerWaitBackoff:0s PoolSize:1 MaxQueueSize:0 ConnectionBufferSize:16384} HTTPConfig:{MaxIdleConns:64000 MaxIdleConnsPerHost:256 ConnectTimeout:1m0s IdleConnectionTimeout:1m30s} DefaultRetryStrategy:<nil> CircuitBreakerConfig:{Enabled:false VolumeThreshold:0 ErrorThresholdPercentage:0 SleepWindow:0s RollingWindow:0s CompletionCallback:<nil> CanaryTimeout:0s} OrphanReporterConfig:{Enabled:false ReportInterval:0s SampleSize:0} TracerConfig:{Tracer:<nil> NoRootTraceSpans:false} MeterConfig:{Meter:<nil>} InternalConfig:{EnableResourceUnitsTrackingHello:false}}
      		 
      2023-05-17T09:06:17.691Z [INF] finished initializing server connections
      		 
      2023-05-17T09:06:17.691Z [DBG] SGCluster+: c:chat-sync-2-cfgSG cfg_sg: Get, key: indexDefs, cas: 0
      		 
      2023-05-17T09:06:17.692Z [DBG] SGCluster+: c:chat-sync-2-cfgSG cfg_sg: Get, key: nodeDefs-wanted, cas: 0
      		 
      2023-05-17T09:06:17.692Z [ERR] Unexpected panic: runtime error: index out of range [0] with length 0 - stopping process
      		 
      goroutine 1 [running]:
      		 
      runtime/debug.Stack()
      		 
      	/home/couchbase/cbdeps/go1.19.5/src/runtime/debug/stack.go:24 +0x65
      		 
      github.com/couchbase/sync_gateway/base.FatalPanicHandler()
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/sync_gateway/base/util.go:1524 +0x49
      		 
      panic({0x12666e0, 0xc03c28a1e0})
      		 
      	/home/couchbase/cbdeps/go1.19.5/src/runtime/panic.go:884 +0x212
      		 
      github.com/couchbase/gocbcore/v10.createAgent(0xc027b77680)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/godeps/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.3-0.20230412164057-d9c465de8911/agent.go:347 +0x2b85
      		 
      github.com/couchbase/gocbcore/v10.CreateAgent(...)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/godeps/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.3-0.20230412164057-d9c465de8911/agent.go:68
      		 
      github.com/couchbase/cbgt.setupGocbcoreAgent(0xc027b77680)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/godeps/pkg/mod/github.com/couchbase/cbgt@v1.3.2-0.20230405201040-c0ede9925e46/gocbcore_agents.go:88 +0x3c
      		 
      github.com/couchbase/cbgt.(*gocbcoreAgentsMap).createAgentsLOCKED(0xc000012990, {0xc000045a80, 0xb}, {0xc027af0ea0, 0x20}, {0xc000043b30, 0x4d}, {0xc000470780, 0x93}, 0xc03c0c9b60)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/godeps/pkg/mod/github.com/couchbase/cbgt@v1.3.2-0.20230405201040-c0ede9925e46/gocbcore_utils.go:247 +0x8cf
      		 
      github.com/couchbase/cbgt.(*gocbcoreAgentsMap).obtainAgents(0xc000012990, {0xc000045a80?, 0x0?}, {0xc027af0ea0, 0x20}, {0xc000043b30, 0x4d}, {0xc000470780?, 0x0?}, 0xc03c0c9b60)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/godeps/pkg/mod/github.com/couchbase/cbgt@v1.3.2-0.20230405201040-c0ede9925e46/gocbcore_utils.go:136 +0x1e5
      		 
      github.com/couchbase/cbgt.CBPartitions({0x44?, 0x7f4500000032?}, {0xc000045a80?, 0x4edf87?}, {0xc027af0ea0?, 0x20300d?}, {0xc000043b30?, 0x1?}, {0xc000470780, 0x93}, ...)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/godeps/pkg/mod/github.com/couchbase/cbgt@v1.3.2-0.20230405201040-c0ede9925e46/gocbcore_utils.go:295 +0x70
      		 
      github.com/couchbase/sync_gateway/base.SGGoCBFeedPartitions({0x1313372, 0x10}, {0xc000045a80, 0xb}, {0xc027af0ea0, 0x20}, {0xc03c05e3f0?, 0x20300d?}, {0xc000470780, 0x93}, ...)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/sync_gateway/base/dcp_feed_type.go:209 +0xb8
      		 
      github.com/couchbase/cbgt.DataSourcePartitions({0x1313372, 0x10}, {0xc000045a80, 0xb}, {0xc027af0ea0, 0x20}, {0xc03c05e3f0, 0x30}, {0xc000470780, 0x93}, ...)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/godeps/pkg/mod/github.com/couchbase/cbgt@v1.3.2-0.20230405201040-c0ede9925e46/feed.go:133 +0xf3
      		 
      github.com/couchbase/cbgt.DataSourcePrepParams({0x1313372, 0x10}, {0xc000045a80, 0xb}, {0xc027af0ea0, 0x20}, {0xc03c05e3f0, 0x30}, {0xc000470780, 0x93}, ...)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/godeps/pkg/mod/github.com/couchbase/cbgt@v1.3.2-0.20230405201040-c0ede9925e46/feed.go:147 +0x89
      		 
      github.com/couchbase/cbgt.(*Manager).CreateIndexEx(0xc03c160800, 0xc027d4a7f8)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/godeps/pkg/mod/github.com/couchbase/cbgt@v1.3.2-0.20230405201040-c0ede9925e46/manager_api.go:131 +0x4ce
      		 
      github.com/couchbase/cbgt.(*Manager).CreateIndex(0x2bc7?, {0x1313372?, 0x0?}, {0xc000045a80?, 0x0?}, {0xc027af0ea0?, 0x0?}, {0xc03c05e3f0?, 0x0?}, {0x131b33f, ...}, ...)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/godeps/pkg/mod/github.com/couchbase/cbgt@v1.3.2-0.20230405201040-c0ede9925e46/manager_api.go:42 +0x153
      		 
      github.com/couchbase/sync_gateway/base.createCBGTIndex({_, _}, _, {_, _}, {_, _}, {_, _}, {{0xc000121380, ...}, ...}, ...)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/sync_gateway/base/dcp_sharded.go:192 +0x736
      		 
      github.com/couchbase/sync_gateway/base.(*CbgtContext).StartManager(_, {_, _}, {_, _}, {_, _}, {_, _}, {{0xc000121380, ...}, ...}, ...)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/sync_gateway/base/dcp_sharded.go:388 +0x205
      		 
      github.com/couchbase/sync_gateway/base.StartShardedDCPFeed({_, _}, {_, _}, {_, _}, {_, _}, {0x16913b8, 0xc0006c79a0}, ...)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/sync_gateway/base/dcp_sharded.go:81 +0x2e5
      		 
      github.com/couchbase/sync_gateway/db.(*importListener).StartImportFeed(0xc027cc5d40, 0xc000022600)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/sync_gateway/db/import_listener.go:122 +0x969
      		 
      github.com/couchbase/sync_gateway/db.NewDatabaseContext({_, _}, {_, _}, {_, _}, _, {0xc0168c1310, 0xc037863978, 0x12c, ...})
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/sync_gateway/db/database.go:648 +0x20ad
      		 
      github.com/couchbase/sync_gateway/rest.(*ServerContext)._getOrAddDatabaseFromConfig(_, {_, _}, {0x175bf6fe452c0000, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/sync_gateway/rest/server_context.go:702 +0x12f1
      		 
      github.com/couchbase/sync_gateway/rest.(*ServerContext)._reloadDatabaseWithConfig(_, {_, _}, {0x175bf6fe452c0000, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/sync_gateway/rest/server_context.go:361 +0xd2
      		 
      github.com/couchbase/sync_gateway/rest.(*ServerContext)._applyConfig(_, {{_, _}}, {0x175bf6fe452c0000, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/sync_gateway/rest/config.go:1680 +0x445
      		 
      github.com/couchbase/sync_gateway/rest.(*ServerContext)._applyConfigs(0xc0001ec000?, {0x168dd18, 0xc000044030}, 0xc0005cae01?, 0xc0?)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/sync_gateway/rest/config.go:1608 +0x157
      		 
      github.com/couchbase/sync_gateway/rest.(*ServerContext).fetchAndLoadConfigs(0xc0001ec000, {0x168dd18, 0xc000044030}, 0x1)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/sync_gateway/rest/config.go:1323 +0x60f
      		 
      github.com/couchbase/sync_gateway/rest.(*ServerContext).initializeCouchbaseServerConnections(0xc0001ec000, {0x168dd18?, 0xc000044030})
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/sync_gateway/rest/server_context.go:1850 +0x2b7
      		 
      github.com/couchbase/sync_gateway/rest.SetupServerContext({0x168dd18, 0xc000044030}, 0xc000512d80, 0x64?)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/sync_gateway/rest/config.go:1264 +0x399
      		 
      github.com/couchbase/sync_gateway/rest.serverMainPersistentConfig({0x168dd18, 0xc000044030}, 0xc000192000, 0xc000512900)
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/sync_gateway/rest/main.go:145 +0x589
      		 
      github.com/couchbase/sync_gateway/rest.serverMain({0x168dd18, 0xc000044030}, {0xc0000360c0, 0x4, 0x4})
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/sync_gateway/rest/main.go:58 +0x145
      		 
      github.com/couchbase/sync_gateway/rest.ServerMain()
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/sync_gateway/rest/main.go:32 +0x45
      		 
      main.main()
      		 
      	/home/couchbase/jenkins/workspace/sgw-unix-build/3.1.0/enterprise/sync_gateway/main.go:24 +0x17

      The presumed line of the bug is https://github.com/couchbase/cbgt/blob/e240972a811cee52010177a8145c3e82616457dc/gocbcore_agents.go#L146

      Attachments

        Activity

          People

            tor.colvin Tor Colvin
            adamf Adam Fraser
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty