Uploaded image for project: 'Couchbase Go SDK'
  1. Couchbase Go SDK
  2. GOCBC-1084

multinode cluster not able to use TLS when specified local host

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Incomplete
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: library
    • Story Points:
      1

      Description

      Repro:

      setup single node cluster with explicitly specifying hostname as ip address (if leave blank it picks as 127.0.0.1 that works).
      OR
      setup multi node cluster which host ip become hostnames by default.

      Also enable the encryption

      As hostname setup TLS config has actual hosts not loop back address 127.0.0.1

      connect the gocbcore using connection string couchbases://127.0.0.1 with http boot strap
      with UseTLS=true
      The connection fails with

      _time=2021-04-15T18:08:45.621-07:00 _level=INFO _msg=(GOCBCORE) SDK Version: gocbcore/v9.1.3
      _time=2021-04-15T18:08:45.622-07:00 _level=INFO _msg=(GOCBCORE) Creating new agent: {MemdAddrs:[] HTTPAddrs:[<sd>127.0.0.1:18091<sd>] BucketName:<md>default<md> UserAgent:default UseTLS:true NetworkType: Auth:0x3c0a550 TLSRootCAProvider:0x1fa29d0 UseMutationTokens:false UseCompression:false UseDurations:false DisableDecompression:false UseOutOfOrderResponses:false DisableXErrors:false DisableJSONHello:false DisableSyncReplicationHello:false UseCollections:true CompressionMinSize:0 CompressionMinRatio:0 HTTPRedialPeriod:0s HTTPRetryDelay:0s HTTPMaxWait:0s CccpMaxWait:0s CccpPollPeriod:0s ConnectTimeout:10s KVConnectTimeout:7s KvPoolSize:8 MaxQueueSize:32768 HTTPMaxIdleConns:0 HTTPMaxIdleConnsPerHost:0 HTTPIdleConnectionTimeout:0s Tracer:<nil> NoRootTraceSpans:false DefaultRetryStrategy:0xc0000e2330 CircuitBreakerConfig:{Enabled:false VolumeThreshold:0 ErrorThresholdPercentage:0 SleepWindow:0s RollingWindow:0s CompletionCallback:<nil> CanaryTimeout:0s} UseZombieLogger:false ZombieLoggerInterval:0s ZombieLoggerSampleSize:0 AuthMechanisms:[]}
      _time=2021-04-15T18:08:45.622-07:00 _level=DEBUG _msg=(GOCBCORE) CCCP Looper starting.
      _time=2021-04-15T18:08:45.622-07:00 _level=INFO _msg=(GOCBCORE) CCCPPOLL: No nodes available to poll, return upstream
      _time=2021-04-15T18:08:45.622-07:00 _level=DEBUG _msg=(GOCBCORE) HTTP Looper starting.
      _time=2021-04-15T18:08:45.622-07:00 _level=DEBUG _msg=(GOCBCORE) Http Picked: https://127.0.0.1:18091.
      _time=2021-04-15T18:08:45.622-07:00 _level=DEBUG _msg=(GOCBCORE) HTTP Hostname: 127.0.0.1.
      _time=2021-04-15T18:08:45.622-07:00 _level=DEBUG _msg=(GOCBCORE) Requesting config from: https://127.0.0.1:18091//pools/default/bs/default.
      _time=2021-04-15T18:08:45.622-07:00 _level=TRACE _msg=(GOCBCORE) Writing HTTP request to https://127.0.0.1:18091/pools/default/bs/default ID=42a7ae90-530e-4d52-a48b-dc404a2d6e1f
      _time=2021-04-15T18:08:45.628-07:00 _level=TRACE _msg=(GOCBCORE) Received HTTP Response for ID=42a7ae90-530e-4d52-a48b-dc404a2d6e1f, errored
      _time=2021-04-15T18:08:45.628-07:00 _level=WARN _msg=(GOCBCORE) Failed to connect to host. Get https://127.0.0.1:18091/pools/default/bs/default: x509: certificate is valid for 172.23.97.79, not 127.0.0.1
      _time=2021-04-15T18:08:45.628-07:00 _level=DEBUG _msg=(GOCBCORE) Pick Failed.
      _time=2021-04-15T18:08:45.628-07:00 _level=DEBUG _msg=(GOCBCORE) Looper waiting...
      _time=2021-04-15T18:08:48.806-07:00 _level=DEBUG _msg= Credentials for user <ud>@prometheus</ud>
      

      Server components (N1QL, FTS) only know local address http://127.0.0.1:8091 we can pass that to gocbcore. gocbcore should get cluster config. Based on cluster config derive actual host of the nodes and connect to it (useTLS set use SSL)

      nodeServices gives actual hostname even run on local host

      curl -u Administrator:password http://localhost:8091/pools/default/nodeServices
      {"rev":44,"nodesExt":[{"services":{"capi":8092,"capiSSL":18092,"indexAdmin":9100,"indexHttp":9102,"indexHttps":19102,"indexScan":9101,"indexStreamCatchup":9104,"indexStreamInit":9103,"indexStreamMaint":9105,"kv":11210,"kvSSL":11207,"mgmt":8091,"mgmtSSL":18091,"n1ql":8093,"n1qlSSL":18093,"projector":9999},"thisNode":true,"hostname":"172.23.97.79"}],"clusterCapabilitiesVer":[1,0],"clusterCapabilities":{"n1ql":["enhancedPreparedStatements"]}}
      

      I have checked with Abhinav Dangeti they never set useTLS on config. Without that TLS never used.

      NOTE: If i am right at present only remote nodes only encrypted not local nodes (this allows config receive faster on non-ssl). There is no option pass that info to gocbcore.

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            brett19 Brett Lawson added a comment -

            Hey Sitaram Vemulapalli,

            We are going to discuss this in the team meeting as the underlying request here will actually affect a number of SDKs being used by server components. I'm going to drop the affects-cc-testing component from this though, as the workaround of fetching the config over PLAIN and creating a TLS connection string works fine, we just can't currently support the request that local node connections happen over PLAIN rather than TLS. Let me know if I am mistaken and there are still some aspects to this that are affecting CC.

            Cheers, Brett

            Show
            brett19 Brett Lawson added a comment - Hey Sitaram Vemulapalli , We are going to discuss this in the team meeting as the underlying request here will actually affect a number of SDKs being used by server components. I'm going to drop the affects-cc-testing component from this though, as the workaround of fetching the config over PLAIN and creating a TLS connection string works fine, we just can't currently support the request that local node connections happen over PLAIN rather than TLS. Let me know if I am mistaken and there are still some aspects to this that are affecting CC. Cheers, Brett
            Hide
            Sitaram.Vemulapalli Sitaram Vemulapalli added a comment -

            Brett Lawson N1QL has its own cluster map and moved on use that to connect actual host under SSL. N1QL has its own requirements dictated by ns_server. Underneath component doesn't support we need to find a workaround and move on or do ourself. Due to time constraint i used workaround time being.

            Show
            Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - Brett Lawson N1QL has its own cluster map and moved on use that to connect actual host under SSL. N1QL has its own requirements dictated by ns_server. Underneath component doesn't support we need to find a workaround and move on or do ourself. Due to time constraint i used workaround time being.
            Hide
            brett19 Brett Lawson added a comment -

            Hey Sitaram Vemulapalli,

            Can you quickly describe what that workaround is (if its not as I described) for the benefit of this ticket?

            Cheers, Brett

            Show
            brett19 Brett Lawson added a comment - Hey Sitaram Vemulapalli , Can you quickly describe what that workaround is (if its not as I described) for the benefit of this ticket? Cheers, Brett
            Hide
            Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - - edited

            While making SSL connection, N1QL will get the cluster config and find the hostname of the current node, SSL port and pass that as connection string.

            curl -u Administrator:password http://localhost:8091/pools/default/nodeServices
            {"rev":44,"nodesExt":[{"services":{"capi":8092,"capiSSL":18092,"indexAdmin":9100,"indexHttp":9102,"indexHttps":19102,"indexScan":9101,"indexStreamCatchup":9104,"indexStreamInit":9103,"indexStreamMaint":9105,"kv":11210,"kvSSL":11207,"mgmt":8091,"mgmtSSL":18091,"n1ql":8093,"n1qlSSL":18093,"projector":9999},"thisNode":true,"hostname":"172.23.97.79"}],"clusterCapabilitiesVer":[1,0],"clusterCapabilities":{"n1ql":["enhancedPreparedStatements"]}}
            

            In above case "thisNode":true, "hostname": "172.23.97.79" and "mgmtSSL":18091
            connection string will be couchbases://172.23.97.79?network=default&bootstrap_on=http (because SSL port is default port)
            If default port is not 18091 http://172.23.97.7:port?network=default&bootstrap_on=http

            In addition UseTLS=true set on agent config

            Show
            Sitaram.Vemulapalli Sitaram Vemulapalli added a comment - - edited While making SSL connection, N1QL will get the cluster config and find the hostname of the current node, SSL port and pass that as connection string. curl -u Administrator:password http: //localhost:8091/pools/default/nodeServices { "rev" : 44 , "nodesExt" :[{ "services" :{ "capi" : 8092 , "capiSSL" : 18092 , "indexAdmin" : 9100 , "indexHttp" : 9102 , "indexHttps" : 19102 , "indexScan" : 9101 , "indexStreamCatchup" : 9104 , "indexStreamInit" : 9103 , "indexStreamMaint" : 9105 , "kv" : 11210 , "kvSSL" : 11207 , "mgmt" : 8091 , "mgmtSSL" : 18091 , "n1ql" : 8093 , "n1qlSSL" : 18093 , "projector" : 9999 }, "thisNode" : true , "hostname" : "172.23.97.79" }], "clusterCapabilitiesVer" :[ 1 , 0 ], "clusterCapabilities" :{ "n1ql" :[ "enhancedPreparedStatements" ]}} In above case "thisNode":true, "hostname": "172.23.97.79" and "mgmtSSL":18091 connection string will be couchbases://172.23.97.79?network=default&bootstrap_on=http (because SSL port is default port) If default port is not 18091 http://172.23.97.7:port?network=default&bootstrap_on=http In addition UseTLS=true set on agent config
            Hide
            brett19 Brett Lawson added a comment -

            This issue is more or less resolved at this point, but there is still a related issue with regards to being able to connect to the cluster generally using TLS, but having the localhost connection still use non-TLS for performance reasons. I have filed GOCBC-1090 which covers this more specific request, and we will triage and handle that behaviour request over there instead.

            Show
            brett19 Brett Lawson added a comment - This issue is more or less resolved at this point, but there is still a related issue with regards to being able to connect to the cluster generally using TLS, but having the localhost connection still use non-TLS for performance reasons. I have filed GOCBC-1090 which covers this more specific request, and we will triage and handle that behaviour request over there instead.

              People

              Assignee:
              brett19 Brett Lawson
              Reporter:
              Sitaram.Vemulapalli Sitaram Vemulapalli
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty