Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49376

Alternative port for kv service rejected on non-kv node

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 7.1.0
    • ns_server
    • Untriaged
    • 1
    • Unknown

    Description

      I'm trying to set an alternative port for each node in a cluster to ensure that sdk bootstrapping can occur regardless of which node is selected.  Problem is, that when a node is running only the query service, an alternate port kv cannot be provided. 

      Attempts to set the kv & query port actually returns a 200 OK response:

       curl -v -X PUT -u Administrator:- http://test-couchbase-cluster-0002.test-couchbase-cluster.default.svc:8091/node/controller/setupAlternateAddresses/external -d hostname=172.18.0.2 -d kv=9000 -d n1ql=9050
      ...
      * Mark bundle as not supporting multiuse
      < HTTP/1.1 200 OK
      

       

      But only the n1ql port is set, not kv:

       {"services":{"mgmt":8091,"mgmtSSL":18091,"n1ql":8093,"n1qlSSL":18093},"hostname":"test-couchbase-cluster-0002.test-couchbase-cluster.default.svc","alternateAddresses":{"external":{"hostname":"172.18.0.2","ports":{"n1ql":9050}}}}

       

      Maybe this is expected behavior, although it can be misleading with a 200 response being returned?

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Yeah this seems like the expected behavior to some extent. We probably can't just return the external hosts for services we aren't even running when you hit the pools/default/nodeServices endpoint.. it's not like you are hitting the 'clusterServices' endpoint (I know that doesn't exist). The only thing that is problematic is the 200 response but I'm not sure how important that is or isn't – so  Meni Hillel is this important enough to fix? I guess we could just return error if we are attempting to set this value on nodes where we don't have those services.. 

            Also, Tommie McAfee what version is this affecting for you? It looks like this is the case in NEO which is what I used to verify.. 

            bryan.mccoid Bryan McCoid added a comment - Yeah this seems like the expected behavior to some extent. We probably can't just return the external hosts for services we aren't even running when you hit the pools/default/nodeServices endpoint.. it's not like you are hitting the 'clusterServices' endpoint (I know that doesn't exist). The only thing that is problematic is the 200 response but I'm not sure how important that is or isn't – so  Meni Hillel  is this important enough to fix? I guess we could just return error if we are attempting to set this value on nodes where we don't have those services..  Also, Tommie McAfee  what version is this affecting for you? It looks like this is the case in NEO which is what I used to verify.. 
            bryan.mccoid Bryan McCoid added a comment - - edited

            I should also mention, that the value DOES get set.. you just aren't seeing it in your request because it's filtered out.

            (n_1@127.0.0.1)5> ns_config:search({node,'n_1@127.0.0.1',alternate_addresses}).
            {value,[{external,[{hostname,"fake.host.blahblah.com"},
                               {ports,[{query_port,9051},{memcached_port,9999}]}]}]}
            (n_1@127.0.0.1)6>
            

            $ curl -v -u Administrator:asdasd http://127.0.0.1:9001/pools/default/nodeServices
            {"rev":87,"nodesExt":[{"services":{"capi":9500,"capiSSL":19500,"kv":12000,"kvSSL":11998,"mgmt":9000,"mgmtSSL":19000,"projector":10000},"hostname":"192.168.1.12","alternateAddresses":{"external":{"hostname":"127.0.0.1","ports":{"kv":9000}}}},{"services":{"mgmt":9001,"mgmtSSL":19001,"n1ql":9498,"n1qlSSL":19498},"thisNode":true,"alternateAddresses":{"external":{"hostname":"fake.host.blahblah.com","ports":{"n1ql":9051}}}}],"clusterCapabilitiesVer":[1,0],"clusterCapabilities":{"n1ql":["enhancedPreparedStatements"]},"revEpoch":1}
            

            This is an intriguing use case, though.. what was the original use-case for this feature? Maybe it can be justified to return these values..

            EDIT: We can fix this with a 1 line fix, we just have to figure out if that's what we actually want.. I think it's a good idea to change if it makes your life easier, since I'm not sure that we gain anything from the supposed "consistency" of filtering out services that aren't on this node.. 

            Anyways let me know the version you are referring to and assign back to me – thanks

            bryan.mccoid Bryan McCoid added a comment - - edited I should also mention, that the value DOES get set.. you just aren't seeing it in your request because it's filtered out. (n_1 @127 .0. 0.1 ) 5 > ns_config:search({node, 'n_1@127.0.0.1' ,alternate_addresses}). {value,[{external,[{hostname, "fake.host.blahblah.com" }, {ports,[{query_port, 9051 },{memcached_port, 9999 }]}]}]} (n_1 @127 .0. 0.1 ) 6 > $ curl -v -u Administrator:asdasd http: //127.0.0.1:9001/pools/default/nodeServices { "rev" : 87 , "nodesExt" :[{ "services" :{ "capi" : 9500 , "capiSSL" : 19500 , "kv" : 12000 , "kvSSL" : 11998 , "mgmt" : 9000 , "mgmtSSL" : 19000 , "projector" : 10000 }, "hostname" : "192.168.1.12" , "alternateAddresses" :{ "external" :{ "hostname" : "127.0.0.1" , "ports" :{ "kv" : 9000 }}}},{ "services" :{ "mgmt" : 9001 , "mgmtSSL" : 19001 , "n1ql" : 9498 , "n1qlSSL" : 19498 }, "thisNode" : true , "alternateAddresses" :{ "external" :{ "hostname" : "fake.host.blahblah.com" , "ports" :{ "n1ql" : 9051 }}}}], "clusterCapabilitiesVer" :[ 1 , 0 ], "clusterCapabilities" :{ "n1ql" :[ "enhancedPreparedStatements" ]}, "revEpoch" : 1 } This is an intriguing use case, though.. what was the original use-case for this feature? Maybe it can be justified to return these values.. EDIT: We can fix this with a 1 line fix, we just have to figure out if that's what we actually want.. I think it's a good idea to change if it makes your life easier, since I'm not sure that we gain anything from the supposed "consistency" of filtering out services that aren't on this node..  Anyways let me know the version you are referring to and assign back to me – thanks

            Got it, the use case here is to ensure SDK's can retrieve cluster config from any node via Global CCCP (https://github.com/couchbaselabs/sdk-rfcs/blob/master/rfc/0048-sdk3-bootstrapping.md#cluster-connection-bootstrapping-sequence) 

            As it's my understanding that every node actually runs kv service for this purpose, even if the node is advertised as query only.

            So this would be to have accurate representation of what ports are reachable.  As it is currently, I'm only able to do bootstrapping against nodes that include data service, which I thought was no longer a limitation.  Testing with server 7.0

            Alternatively I agree that it doesn't make sense to accept an alternate port for index or search on query only service. 

             

            tommie Tommie McAfee added a comment - Got it, the use case here is to ensure SDK's can retrieve cluster config from any node via Global CCCP ( https://github.com/couchbaselabs/sdk-rfcs/blob/master/rfc/0048-sdk3-bootstrapping.md#cluster-connection-bootstrapping-sequence)   As it's my understanding that every node actually runs kv service for this purpose, even if the node is advertised as query only. So this would be to have accurate representation of what ports are reachable.  As it is currently, I'm only able to do bootstrapping against nodes that include data service, which I thought was no longer a limitation.  Testing with server 7.0 Alternatively I agree that it doesn't make sense to accept an alternate port for index or search on query only service.   
            bryan.mccoid Bryan McCoid added a comment -

             Dave Finlay  explained that we may not want to allow these values to be returned if that node doesn't have that service, but that we should block those requests (return an error) on configured nodes (but allow, still, on non-configured nodes for future use on that node).  

            "As it's my understanding that every node actually runs kv service for this purpose, even if the node is advertised as query only." – is this true? 

            So your request for these to be shown on nodes that don't have that service, does not extend beyond the kv service? Regardless it seems like we are going the other way and creating some validation and error responses to disallow users to set those values in the first place, when they are invalid. 

            bryan.mccoid Bryan McCoid added a comment -   Dave Finlay   explained that we may not want to allow these values to be returned if that node doesn't have that service, but that we should block those requests (return an error) on configured nodes (but allow, still, on non-configured nodes for future use on that node).   "As it's my understanding that every node actually runs kv service for this purpose, even if the node is advertised as query only." – is this true?   So your request for these to be shown on nodes that don't have that service, does not extend beyond the kv service? Regardless it seems like we are going the other way and creating some validation and error responses to disallow users to set those values in the first place, when they are invalid. 
            dfinlay Dave Finlay added a comment - - edited

            Hey Tommie McAfee:

            Every node runs an instance of memcached; but not of KV. Memcached is run as on all nodes it plays the role of the audit daemon, accepting & validating audits and writing them to the log. On non-KV nodes, memcached only runs auditing - and you can't bootstrap a bucket connection on these nodes.

            dfinlay Dave Finlay added a comment - - edited Hey Tommie McAfee : Every node runs an instance of memcached; but not of KV. Memcached is run as on all nodes it plays the role of the audit daemon, accepting & validating audits and writing them to the log. On non-KV nodes, memcached only runs auditing - and you can't bootstrap a bucket connection on these nodes.

            OK sounds like the port filtering is correct,  thanks for clarifying.

            tommie Tommie McAfee added a comment - OK sounds like the port filtering is correct,  thanks for clarifying.

            Tommie McAfee Can you share with us if at the time you've set the alternative ports, the node was already 1) not provisioned, 2) provisioned, not part of the cluster, 3) provisioned and already part of the cluster?

            meni.hillel Meni Hillel (Inactive) added a comment - Tommie McAfee Can you share with us if at the time you've set the alternative ports, the node was already 1) not provisioned, 2) provisioned, not part of the cluster, 3) provisioned and already part of the cluster?

            Build couchbase-server-7.1.0-1735 contains ns_server commit 25c75ce with commit message:
            MB-49376: Alternative external service address validation

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1735 contains ns_server commit 25c75ce with commit message: MB-49376 : Alternative external service address validation

            Verified on 7.1.0-2246 that trying to set alternate port for a service that does not exist on a node is disallowed and returns an error like the following:

            Cannot set external ports ["kv"] as services are unavailable on the node.

            sumedh.basarkod Sumedh Basarkod added a comment - Verified on 7.1.0-2246 that trying to set alternate port for a service that does not exist on a node is disallowed and returns an error like the following: Cannot set external ports ["kv"] as services are unavailable on the node.

            People

              sumedh.basarkod Sumedh Basarkod
              tommie Tommie McAfee
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty