Description
The data service only pods are not exposing NodePorts for the view service on 8092.
test.yaml
This creates two pods, one for data and one for index. Only the index pod exposes a NodePort service for 8092.
kubectl get svc
cb-example-0000 NodePort 10.96.83.144 <none> 8091:31447/TCP,18091:31343/TCP,11210:31937/TCP,11207:32437/TCP
cb-example-0001 NodePort 10.96.208.44 <none> 8091:32601/TCP,18091:32422/TCP,11210:32747/TCP,11207:31558/TCP,8092:30492/TCP,18092:32543/TCP
kubectl describe pod cb-example-0000
Name: cb-example-0000
...
Labels: app=couchbase
couchbase_cluster=cb-example
couchbase_node=cb-example-0000
couchbase_node_conf=data
couchbase_server=true
couchbase_service_data=enabled
Attachments
- test.yaml
- 0.8 kB
- ViewsSDK.py
- 1 kB
Issue Links
Activity
Existing 2.2 GA using the above YAML:
cb-example-0000 NodePort ... 8091:30580/TCP,18091:32506/TCP,11210:32293/TCP,11207:32664/TCP
cb-example-0001 NodePort ... 8091:31940/TCP,18091:30126/TCP,11210:30429/TCP,11207:31843/TCP,8092:31012/TCP,18092:30149/TCP
With changes from Gerrit:
cb-example-0000 NodePort ... 11210:32700/TCP,11207:30187/TCP,8091:32373/TCP,18091:30511/TCP,8092:32130/TCP,18092:30850/TCP
cb-example-0001 NodePort ... 11210:31743/TCP,11207:32040/TCP,8091:32355/TCP,18091:30746/TCP
Had a quick discussion with Binh Le, and capturing his advice here for posterity. I've updated my review as well.
|
matt 11:02 AM
|
hey Binh, need an opinion on something for indexer-- you're the indexer PM, is that right?
|
|
Binh Le 11:03 AM
|
Yes. But I am hosting a call right now. I can answer your slack later if thats ok
|
|
matt 11:08 AM
|
yes, I'll drop the question here, and just reply later when you have a moment…
|
|
question is: should port 9102 be exposed to admins/clients/public when running under Kubernetes.
|
|
in a K8S deployment, we have the concept of exposed services. As you know, index service doesn't have any network services (ironically). It was originally going to, but that's a discussion for another day…
|
|
There is a bug right now where index as an exposed service is exposing the views port, because that had historically been called an index. we are fixing that because views really always go with KV. As we fix that, should we expose 9102? I can see from docs that there are a couple of publicly documented REST endpoints on that port number:
|
https://docs.couchbase.com/server/current/rest-api/rest-index-service.html (edited)
|
|
|
docs.couchbase.com
|
Index Service API | Couchbase Docs
|
The Index service REST API provides configuration options for the Index service.
|
|
matt 12:25 PM
|
any chance you're out?
|
|
Binh Le 12:48 PM
|
I am here Matt. The stat endpoint is important for GSI,as the info is needed to manually manage the index - as the UI doesn’t support everytrhing. Closing the port would not make sense.
|
|
matt 12:48 PM
|
well, it's closed now :)
|
12:48
|
but, by closed I mean from internal to Kubernetes to the outside world
|
12:49
|
possibly to clients, depending on the user's deployment approach
|
12:51
|
two questions:
|
. uses for it are just these stats things? are there situations where support or others may request a customer to send a request to it directly that aren't documented?
|
. are there any security considerations? other ports have some security considerations too, but is there anything here that shouldn't be exposed to administrators/client apps? exposing the port is pretty coarse grained in that we don't quite know how it will be used, but all of that is covered in our docs
|
|
Binh Le 12:54 PM
|
I recalled having to use this stat myself to determine the replica id, so that it can be moved or delete
|
|
matt 12:54 PM
|
oh, that wasn't available through the N1QL interface?
|
|
Binh Le 12:54 PM
|
so 1 - is Yes.
|
12:56
|
The replica id is needed for the ALTER INDEX
|
12:57
|
And I don’t know of another way to get the replicaid other than from the index stat
|
|
matt 1:23 PM
|
how about 2?
|
|
Binh Le 1:26 PM
|
I am not aware of any security issue we have with this. The API is for full admin and Query Manage Index (edited)
|
|
matt 1:27 PM
|
ok, sounds like your guidance then is we should expose it. any other checks we should do before going about it with the Operator? thanks for taking the time to talk through this
|
|
Binh Le 1:28 PM
|
What i don’t know Matt - is what is the implication of K8S in this. But I guess, K8S is simply how customer deploy CB. So they still need access to the stats
|
|
matt 1:30 PM
|
effectively, yes. details hare here: https://docs.couchbase.com/operator/current/concept-couchbase-networking.html#exposed-features
|
|
matt 1:39 PM
|
technically, the way that works is you add Admin || XDCR || Client as exposed features:
|
https://docs.couchbase.com/operator/current/resource/couchbasecluster.html#couchbaseclusters-spec-networking-exposedfeatures
|
|
matt 2:32 PM
|
so, it sounds like we'll want the index port in client.
|
2:33
|
would you mind if I copy this discussion into a comment on a JIRA ticket for posterity?
|
|
Binh Le 2:34 PM
|
The stat endpoint is needed for troubleshooting, etc
|
|
Binh Le 2:43 PM
|
sure Matt
|
Current understanding is data port services need to be available on all nodes:
- 8092 - views, but needed for all for XDCR
- 11210 - SDK bootstrapping needed for all (
K8S-1713)
Index needs the 9102 service as well.
TLS variants also, just not included here for simplicity.
Testing Process:
CB Configuration: 2 Pods with Server 6.6.2, 1 pod(cb-example-0000) running data service , another(cb-example-0001) running index service.
When Cluster was deployed with Operator 2.2 using test.yaml attached , Status of services and corresponding Ports available: ----
cb-example-0000 NodePort 10.81.204.22 <none> 8091:31105/TCP,18091:32279/TCP,11210:32246/TCP,11207:32142/TCP |
cb-example-0001 NodePort 10.81.203.26 <none> 8091:30137/TCP,18091:30343/TCP,11210:32510/TCP, 11207:30551/TCP,8092:31500/TCP,18092:32680/TCP |
Using same test.yaml, Cluster when deployed with Operator 2.2.1 (build 125) , Status of services and corresponding Ports: ----(ii)
cb-example-0000 NodePort 10.81.205.30 <none> 8091:30457/TCP,18091:31109/TCP,11210:31186/TCP, 11207:32017/TCP,8092:31179/TCP,18092:31868/TCP |
cb-example-0001 NodePort 10.81.194.234 <none> 8091:32327/TCP,18091:32388/TCP,11210:32369/TCP, 11207:30204/TCP,8092:32348/TCP,18092:30368/TCP |
This verifies the first part of the ticket where Views Port(8092) is now available on data pods with Operator 2.2.1
The second part was to test if we can upsert Views design docs using SDK in either Operator setup which translates to:
When bootstrapping using alternative addresses, the SDK should use them as a result, and if 8092 isn't exposed it should fail to upsert design docs.
We used Python3 SDK to verify this hypothesis. (SDK script attached, ViewsSDK.py)
Connection String Used: <NodeIP>:<NodePort> of data pods
- in 2.2.0: <NodeIP>:31105 (refer
from above comment).
- in 2.2.1: <NodeIP>:30457 (refer (ii) )ViewsSDK.py
Upsert Function used: https://github.com/couchbase/couchbase-python-client/blob/55d37b4c0fd15e208f8a2f5a02d45265af398975/couchbase/management/views.py#L142
2.2.0 Observation:
Traceback (most recent call last):
|
File "/test/test1.py", line 22, in <module> |
view_manager.upsert_design_document(
|
File "/usr/local/lib/python3.9/site-packages/couchbase/exceptions.py", line 1473, in wrapped |
return func(*args, **kwargs) |
File "/usr/local/lib/python3.9/site-packages/couchbase/management/views.py", line 166, in upsert_design_document |
self._http_request(False, **args)
|
File "/usr/local/lib/python3.9/site-packages/couchbase/management/views.py", line 89, in _http_request |
return self._bucket._http_request(**kwargs) |
couchbase.exceptions.LCB_0x400 (generated, catch: BaseException, KeyValueException, QueryException, AnalyticsException, SearchException, ViewException): <Key='_design/dev_landmarks', RC=0x400[LCB_ERR_CONNECTION_REFUSED (1024)], HTTP Request failed. Examine 'objextra' for full result, Results=1, C Source=(src/pycbc_http.c,212), OBJ=HttpResult<rc=0x400[LCB_ERR_CONNECTION_REFUSED (1024)], value=None, http_status=0, url=_design/dev_landmarks, tracing_context=0, tracing_output=None>, Context={'response_code': 0, 'path': '/default/_design/dev_landmarks', 'response_body': '', 'endpoint': '10.142.0.17:8092', 'type': 'HTTPErrorContext'}, Tracing Output={":nokey:0": null}> |
2.2.1 Observation:
Found design doc: landmarks w/ 2 views. |
This confirms that Views port wasn't available for any views related Ops on data pod in 2.2.0 but is now present in 2.2.1
Patrick Stephens gave a fix for the attached CBSE which was verified through SDK test in Operator framework. Details in previous comments.
Closing out this ticket.
We are currently enabling the admin and data services for all services (8091 & 11210 - with TLS ones as well).