Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-2293

Expose index/view NodePort service on data pods

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.2.0
    • 2.2.1
    • operator, testing
    • None
    • 1

    Description

      The data service only pods are not exposing NodePorts for the view service on 8092.
      test.yaml

      This creates two pods, one for data and one for index. Only the index pod exposes a NodePort service for 8092.
       
      kubectl get svc
      cb-example-0000                       NodePort    10.96.83.144    <none>  8091:31447/TCP,18091:31343/TCP,11210:31937/TCP,11207:32437/TCP
      cb-example-0001                       NodePort    10.96.208.44    <none>  8091:32601/TCP,18091:32422/TCP,11210:32747/TCP,11207:31558/TCP,8092:30492/TCP,18092:32543/TCP

      kubectl describe pod cb-example-0000  

      Name:         cb-example-0000

      ...

      Labels:       app=couchbase

                    couchbase_cluster=cb-example

                    couchbase_node=cb-example-0000

                    couchbase_node_conf=data

                    couchbase_server=true

                    couchbase_service_data=enabled

       

      Attachments

        1. test.yaml
          0.8 kB
        2. ViewsSDK.py
          1 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            We are currently enabling the admin and data services for all services (8091 & 11210 - with TLS ones as well).

            patrick.stephens Patrick Stephens (Inactive) added a comment - We are currently enabling the admin and data services for all services (8091 & 11210 - with TLS ones as well).

            Existing 2.2 GA using the above YAML:

            cb-example-0000 NodePort ... 8091:30580/TCP,18091:32506/TCP,11210:32293/TCP,11207:32664/TCP

            cb-example-0001 NodePort ... 8091:31940/TCP,18091:30126/TCP,11210:30429/TCP,11207:31843/TCP,8092:31012/TCP,18092:30149/TCP

            With changes from Gerrit:

            cb-example-0000 NodePort ... 11210:32700/TCP,11207:30187/TCP,8091:32373/TCP,18091:30511/TCP,8092:32130/TCP,18092:30850/TCP

            cb-example-0001 NodePort ... 11210:31743/TCP,11207:32040/TCP,8091:32355/TCP,18091:30746/TCP

             

            patrick.stephens Patrick Stephens (Inactive) added a comment - Existing 2.2 GA using the above YAML: cb-example-0000 NodePort ... 8091:30580/TCP,18091:32506/TCP,11210:32293/TCP,11207:32664/TCP cb-example-0001 NodePort ... 8091:31940/TCP,18091:30126/TCP,11210:30429/TCP,11207:31843/TCP,8092:31012/TCP,18092:30149/TCP With changes from Gerrit: cb-example-0000 NodePort ... 11210:32700/TCP,11207:30187/TCP,8091:32373/TCP,18091:30511/TCP,8092:32130/TCP,18092:30850/TCP cb-example-0001 NodePort ... 11210:31743/TCP,11207:32040/TCP,8091:32355/TCP,18091:30746/TCP  

            Had a quick discussion with Binh Le, and capturing his advice here for posterity. I've updated my review as well.


             
            matt  11:02 AM
            hey Binh, need an opinion on something for indexer-- you're the indexer PM, is that right?
             
            Binh Le  11:03 AM
            Yes. But I am hosting a call right now. I can answer your slack later if thats ok
             
            matt  11:08 AM
            yes, I'll drop the question here, and just reply later when you have a moment…
             
            question is: should port 9102 be exposed to admins/clients/public when running under Kubernetes.
             
            in a K8S deployment, we have the concept of exposed services.  As you know, index service doesn't have any network services (ironically).  It was originally going to, but that's a discussion for another day…
             
            There is a bug right now where index as an exposed service is exposing the views port, because that had historically been called an index.  we are fixing that because views really always go with KV.  As we fix that, should we expose 9102?  I can see from docs that there are a couple of publicly documented REST endpoints on that port number:
            https://docs.couchbase.com/server/current/rest-api/rest-index-service.html (edited) 
             
             
            docs.couchbase.com
            Index Service API | Couchbase Docs
            The Index service REST API provides configuration options for the Index service.
             
            matt  12:25 PM
            any chance you're out?
             
            Binh Le  12:48 PM
            I am here Matt. The stat endpoint is important for GSI,as the info is needed to manually manage the index - as the UI doesn’t support everytrhing. Closing the port would not make sense.
             
            matt  12:48 PM
            well, it's closed now :)
            12:48
            but, by closed I mean from internal to Kubernetes to the outside world
            12:49
            possibly to clients, depending on the user's deployment approach
            12:51
            two questions:
            	.	uses for it are just these stats things?  are there situations where support or others may request a customer to send a request to it directly that aren't documented?
            	.	are there any security considerations?  other ports have some security considerations too, but is there anything here that shouldn't be exposed to administrators/client apps?  exposing the port is pretty coarse grained in that we don't quite know how it will be used, but all of that is covered in our docs
             
            Binh Le  12:54 PM
            I recalled having to use this stat myself to determine the replica id, so that it can be moved or delete
             
            matt  12:54 PM
            oh, that wasn't available through the N1QL interface?
             
            Binh Le  12:54 PM
            so 1 - is Yes.
            12:56
            The replica id is needed for the ALTER INDEX
            12:57
            And I don’t know of another way to get the replicaid other than from the index stat
             
            matt  1:23 PM
            how about 2?
             
            Binh Le  1:26 PM
            I am not aware of any security issue we have with this. The API is for full admin and Query Manage Index (edited) 
             
            matt  1:27 PM
            ok, sounds like your guidance then is we should expose it.  any other checks we should do before going about it with the Operator?  thanks for taking the time to talk through this
             
            Binh Le  1:28 PM
            What i don’t know Matt - is what is the implication of K8S in this. But I guess, K8S is simply how customer deploy CB. So they still need access to the stats
             
            matt  1:30 PM
            effectively, yes.  details hare here: https://docs.couchbase.com/operator/current/concept-couchbase-networking.html#exposed-features
             
            matt  1:39 PM
            technically, the way that works is you add Admin || XDCR || Client as exposed features:
            https://docs.couchbase.com/operator/current/resource/couchbasecluster.html#couchbaseclusters-spec-networking-exposedfeatures
             
            matt  2:32 PM
            so, it sounds like we'll want the index port in client.
            2:33
            would you mind if I copy this discussion into a comment on a JIRA ticket for posterity?
             
            Binh Le  2:34 PM
            The stat endpoint is needed for troubleshooting, etc
             
            Binh Le  2:43 PM
            sure Matt
            

            ingenthr Matt Ingenthron added a comment - Had a quick discussion with Binh Le , and capturing his advice here for posterity. I've updated my review as well.   matt  11:02 AM hey Binh, need an opinion on something for indexer-- you're the indexer PM, is that right?   Binh Le  11:03 AM Yes. But I am hosting a call right now. I can answer your slack later if thats ok   matt  11:08 AM yes, I'll drop the question here, and just reply later when you have a moment…   question is: should port 9102 be exposed to admins/clients/public when running under Kubernetes.   in a K8S deployment, we have the concept of exposed services.  As you know, index service doesn't have any network services (ironically).  It was originally going to, but that's a discussion for another day…   There is a bug right now where index as an exposed service is exposing the views port, because that had historically been called an index.  we are fixing that because views really always go with KV.  As we fix that, should we expose 9102?  I can see from docs that there are a couple of publicly documented REST endpoints on that port number: https://docs.couchbase.com/server/current/rest-api/rest-index-service.html (edited)      docs.couchbase.com Index Service API | Couchbase Docs The Index service REST API provides configuration options for the Index service.   matt  12:25 PM any chance you're out?   Binh Le  12:48 PM I am here Matt. The stat endpoint is important for GSI,as the info is needed to manually manage the index - as the UI doesn’t support everytrhing. Closing the port would not make sense.   matt  12:48 PM well, it's closed now :) 12:48 but, by closed I mean from internal to Kubernetes to the outside world 12:49 possibly to clients, depending on the user's deployment approach 12:51 two questions: . uses for it are just these stats things?  are there situations where support or others may request a customer to send a request to it directly that aren't documented? . are there any security considerations?  other ports have some security considerations too, but is there anything here that shouldn't be exposed to administrators/client apps?  exposing the port is pretty coarse grained in that we don't quite know how it will be used, but all of that is covered in our docs   Binh Le  12:54 PM I recalled having to use this stat myself to determine the replica id, so that it can be moved or delete   matt  12:54 PM oh, that wasn't available through the N1QL interface?   Binh Le  12:54 PM so 1 - is Yes. 12:56 The replica id is needed for the ALTER INDEX 12:57 And I don’t know of another way to get the replicaid other than from the index stat   matt  1:23 PM how about 2?   Binh Le  1:26 PM I am not aware of any security issue we have with this. The API is for full admin and Query Manage Index (edited)    matt  1:27 PM ok, sounds like your guidance then is we should expose it.  any other checks we should do before going about it with the Operator?  thanks for taking the time to talk through this   Binh Le  1:28 PM What i don’t know Matt - is what is the implication of K8S in this. But I guess, K8S is simply how customer deploy CB. So they still need access to the stats   matt  1:30 PM effectively, yes.  details hare here: https://docs.couchbase.com/operator/current/concept-couchbase-networking.html#exposed-features   matt  1:39 PM technically, the way that works is you add Admin || XDCR || Client as exposed features: https://docs.couchbase.com/operator/current/resource/couchbasecluster.html#couchbaseclusters-spec-networking-exposedfeatures   matt  2:32 PM so, it sounds like we'll want the index port in client. 2:33 would you mind if I copy this discussion into a comment on a JIRA ticket for posterity?   Binh Le  2:34 PM The stat endpoint is needed for troubleshooting, etc   Binh Le  2:43 PM sure Matt

            Current understanding is data port services need to be available on all nodes:

            • 8092 - views, but needed for all for XDCR
            • 11210 - SDK bootstrapping needed for all (K8S-1713)

            Index needs the 9102 service as well.

            TLS variants also, just not included here for simplicity.

            patrick.stephens Patrick Stephens (Inactive) added a comment - - edited Current understanding is data port services need to be available on all nodes: 8092 - views, but needed for all for XDCR 11210 - SDK bootstrapping needed for all ( K8S-1713 ) Index needs the 9102 service as well. TLS variants also, just not included here for simplicity.

            Testing Process:

            CB Configuration: 2 Pods with Server 6.6.2, 1 pod(cb-example-0000) running data service , another(cb-example-0001) running index service.

            When Cluster was deployed with Operator 2.2 using test.yaml attached , Status of services and corresponding Ports available:         ----

            cb-example-0000  NodePort    10.81.204.22  <none>  8091:31105/TCP,18091:32279/TCP,11210:32246/TCP,11207:32142/TCP                                                                                                                                      
            cb-example-0001  NodePort    10.81.203.26 <none>   8091:30137/TCP,18091:30343/TCP,11210:32510/TCP, 11207:30551/TCP,8092:31500/TCP,18092:32680/TCP  

            Using same test.yaml, Cluster when deployed with Operator 2.2.1 (build 125) , Status of services and corresponding Ports:               ----(ii)

            cb-example-0000 NodePort 10.81.205.30 <none>    8091:30457/TCP,18091:31109/TCP,11210:31186/TCP, 11207:32017/TCP,8092:31179/TCP,18092:31868/TCP                                                                                                                                                                                   
            cb-example-0001 NodePort 10.81.194.234 <none>   8091:32327/TCP,18091:32388/TCP,11210:32369/TCP, 11207:30204/TCP,8092:32348/TCP,18092:30368/TCP  

            This verifies the first part of the ticket where Views Port(8092) is now available on data pods with Operator 2.2.1

             

            prateek.kumar Prateek Kumar added a comment - Testing Process: CB Configuration: 2 Pods with Server 6.6.2, 1 pod(cb-example-0000) running data service , another(cb-example-0001) running index service. When Cluster was deployed with Operator 2.2 using test.yaml attached , Status of services and corresponding Ports available:         ---- cb-example- 0000   NodePort    10.81 . 204.22   <none>  8091 : 31105 /TCP, 18091 : 32279 /TCP, 11210 : 32246 /TCP, 11207 : 32142 /TCP                                                                                                                                      cb-example- 0001   NodePort    10.81 . 203.26  <none>  8091 : 30137 /TCP, 18091 : 30343 /TCP, 11210 : 32510 /TCP, 11207 : 30551 /TCP, 8092 : 31500 /TCP, 18092 : 32680 /TCP  Using same test.yaml, Cluster when deployed with Operator 2.2.1 (build 125) , Status of services and corresponding Ports:               ----(ii) cb-example- 0000  NodePort  10.81 . 205.30  <none>  8091 : 30457 /TCP, 18091 : 31109 /TCP, 11210 : 31186 /TCP, 11207 : 32017 /TCP, 8092 : 31179 /TCP, 18092 : 31868 /TCP                                                                                                                                                                                   cb-example- 0001  NodePort  10.81 . 194.234 <none>  8091 : 32327 /TCP, 18091 : 32388 /TCP, 11210 : 32369 /TCP, 11207 : 30204 /TCP, 8092 : 32348 /TCP, 18092 : 30368 /TCP  This verifies the first part of the ticket where Views Port(8092) is now available on data pods with Operator 2.2.1  

            The second part was to test if we can upsert Views design docs using SDK in either Operator setup which translates to:

            When bootstrapping using alternative addresses, the SDK should use them as a result, and if 8092 isn't exposed it should fail to upsert design docs.

            We used Python3 SDK to verify this hypothesis. (SDK script attached, ViewsSDK.py)

            Connection String Used: <NodeIP>:<NodePort> of data pods

            • in 2.2.0: <NodeIP>:31105 (refer from above comment).
            • in 2.2.1: <NodeIP>:30457 (refer (ii) )ViewsSDK.py

            Upsert Function used: https://github.com/couchbase/couchbase-python-client/blob/55d37b4c0fd15e208f8a2f5a02d45265af398975/couchbase/management/views.py#L142

            2.2.0 Observation:

            Traceback (most recent call last):
              File "/test/test1.py", line 22, in <module>
                view_manager.upsert_design_document(
              File "/usr/local/lib/python3.9/site-packages/couchbase/exceptions.py", line 1473, in wrapped
                return func(*args, **kwargs)
              File "/usr/local/lib/python3.9/site-packages/couchbase/management/views.py", line 166, in upsert_design_document
                self._http_request(False, **args)
              File "/usr/local/lib/python3.9/site-packages/couchbase/management/views.py", line 89, in _http_request
                return self._bucket._http_request(**kwargs)
            couchbase.exceptions.LCB_0x400 (generated, catch: BaseException, KeyValueException, QueryException, AnalyticsException, SearchException, ViewException): <Key='_design/dev_landmarks', RC=0x400[LCB_ERR_CONNECTION_REFUSED (1024)], HTTP Request failed. Examine 'objextra' for full result, Results=1, C Source=(src/pycbc_http.c,212), OBJ=HttpResult<rc=0x400[LCB_ERR_CONNECTION_REFUSED (1024)], value=None, http_status=0, url=_design/dev_landmarks, tracing_context=0, tracing_output=None>, Context={'response_code': 0, 'path': '/default/_design/dev_landmarks', 'response_body': '', 'endpoint': '10.142.0.17:8092', 'type': 'HTTPErrorContext'}, Tracing Output={":nokey:0": null}> 

            2.2.1 Observation:

            Found design doc: landmarks w/ 2 views. 

             

            This confirms that Views port wasn't available for any views related Ops on data pod in 2.2.0 but is now present in 2.2.1

            prateek.kumar Prateek Kumar added a comment - The second part was to test if we can upsert Views design docs using SDK in either Operator setup which translates to: When bootstrapping using alternative addresses, the SDK should use them as a result, and if 8092 isn't exposed it should fail to upsert design docs. We used Python3 SDK to verify this hypothesis. (SDK script attached, ViewsSDK.py) Connection String Used: <NodeIP>:<NodePort> of data pods in 2.2.0: <NodeIP>:31105 (refer from above comment). in 2.2.1: <NodeIP>:30457 (refer (ii) ) ViewsSDK.py Upsert Function used: https://github.com/couchbase/couchbase-python-client/blob/55d37b4c0fd15e208f8a2f5a02d45265af398975/couchbase/management/views.py#L142 2.2.0 Observation: Traceback (most recent call last):   File "/test/test1.py" , line 22 , in <module>     view_manager.upsert_design_document(   File "/usr/local/lib/python3.9/site-packages/couchbase/exceptions.py" , line 1473 , in wrapped     return func(*args, **kwargs)   File "/usr/local/lib/python3.9/site-packages/couchbase/management/views.py" , line 166 , in upsert_design_document     self._http_request(False, **args)   File "/usr/local/lib/python3.9/site-packages/couchbase/management/views.py" , line 89 , in _http_request     return self._bucket._http_request(**kwargs) couchbase.exceptions.LCB_0x400 (generated, catch : BaseException, KeyValueException, QueryException, AnalyticsException, SearchException, ViewException): <Key= '_design/dev_landmarks' , RC= 0x400 [LCB_ERR_CONNECTION_REFUSED ( 1024 )], HTTP Request failed. Examine 'objextra' for full result, Results= 1 , C Source=(src/pycbc_http.c, 212 ), OBJ=HttpResult<rc= 0x400 [LCB_ERR_CONNECTION_REFUSED ( 1024 )], value=None, http_status= 0 , url=_design/dev_landmarks, tracing_context= 0 , tracing_output=None>, Context={ 'response_code' : 0 , 'path' : '/default/_design/dev_landmarks' , 'response_body' : '' , 'endpoint' : '10.142.0.17:8092' , 'type' : 'HTTPErrorContext' }, Tracing Output={ ":nokey:0" : null }> 2.2.1 Observation: Found design doc: landmarks w/ 2 views.   This confirms that Views port wasn't available for any views related Ops on data pod in 2.2.0 but is now present in 2.2.1
            patrick.stephens Patrick Stephens (Inactive) added a comment - Great work Prateek Kumar !

            Patrick Stephens gave a fix for the attached CBSE which was verified through SDK test in Operator framework. Details in previous comments.

            Closing out this ticket.

            prateek.kumar Prateek Kumar added a comment - Patrick Stephens  gave a fix for the attached CBSE which was verified through SDK test in Operator framework. Details in previous comments. Closing out this ticket.

            People

              prateek.kumar Prateek Kumar
              patrick.stephens Patrick Stephens (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty