Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-2365

[OCP] Operator unable to connect to K8s API Server with Istio STRICT

    XMLWordPrintable

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.2.1
    • not-targeted
    • None
    • None
    • 3

    Description

      Setup Details:

      1. OCP Cluster version: 4.6.1
      2. ServiceMesh 2.0.7 installed on the Cluster. 
      3. Sidecar annotations added to the DAC and Operator Deployments.(sidecar.istio.io/inject: "true")
      4. Peer Authentication Rule created to run DAC with ISTIO PERMISSIVE.(peerauth-dac.yaml, namespace: default)
      5. Destination Rule created to use mTLS when sending requests to other services in the mesh for DAC.(destirule-dac.yaml, namespace: default)
      6. Created DAC deployment, Istio sidecar comes up with Status as Running

        Prateeks-MacBook-Pro:bin prateekkumar$ oc get pods NAME   READY   STATUS    RESTARTS   AGE couchbase-operator-admission-db8f6c4b4-79ncf   2/2     Running   0          125m  

      7. Created a new namespace test-istio-0
      8. Peer Authentication Rule created to run Operator with ISTIO Strict.(peerauth-op.yaml, namespace: test-istio-0)
      9. Destination Rule created to use mTLS when sending requests to other services in the mesh for Operator.(destirule-op.yaml, namespace: test-istio-0)
      10. Created Operator Deployment, Istio sidecar comes up, but Operator pod doesn't. Error:

        Prateeks-MacBook-Pro:bin prateekkumar$ oc get pods -n test-istio-0
        NAME                                  READY   STATUS             RESTARTS   AGE
        couchbase-operator-5fbc545565-fp9qq   1/2     CrashLoopBackOff   7          16m. 

      11. Prateeks-MacBook-Pro:bin prateekkumar$ oc logs pods/couchbase-operator-5fbc545565-fp9qq -n test-istio-0 couchbase-operator {"level":"info","ts":1629794425.0975158,"logger":"main","msg":"couchbase-operator","version":"2.2.1 (build 126)","revision":"b75530987818a959ec1f8984da92b5e2d3f615f7"} {"level":"error","ts":1629794435.1664348,"logger":"controller-runtime.manager","msg":"Failed to get API Group-Resources","error":"an error on the server (\"\") has prevented the request from succeeding","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/couchbase/go/pkg/mod/github.com/go-logr/zapr@v0.3.0/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/manager.New\n\t/home/couchbase/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.4/pkg/manager/manager.go:279\nmain.main\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/cmd/operator/main.go:83\nruntime.main\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/golangHc7tn/go1.16.3/src/runtime/proc.go:225"} {"level":"error","ts":1629794435.1665149,"logger":"main","msg":"Error initializing manager","error":"an error on the server (\"\") has prevented the request from succeeding","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/couchbase/go/pkg/mod/github.com/go-logr/zapr@v0.3.0/zapr.go:132\nmain.main\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/cmd/operator/main.go:91\nruntime.main\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/golangHc7tn/go1.16.3/src/runtime/proc.go:225"} 

      Since automatic mTLS Strict is not used, both Peer Authentication Rule and Destination Rule are required for deployments. (https://docs.openshift.com/container-platform/4.6/service_mesh/v2x/ossm-security.html#ossm-security-mtls-sidecars-incoming-services_ossm-security) 

      cbopinfo attached. Triage in progress.

      Attachments

        1. cbopinfo-20210824T150226+0530.tar.gz
          183 kB
        2. destirule-dac.yaml
          0.2 kB
        3. destirule-op.yaml
          0.2 kB
        4. peerauth-dac.yaml
          0.2 kB
        5. peerauth-op.yaml
          0.1 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            It looks like the operator cannot talk to the API server so it probably needs a rule to allow that.

            I'm not sure I follow the specifics here as to what we're trying to do - enable mTLS globally?

            That would be like so: https://docs.openshift.com/container-platform/4.6/service_mesh/v2x/ossm-security.html#ossm-security-enabling-strict-mtls_ossm-security

            We then need to tweak it to allow the DAC in permissive mode and any ingress the user wants.

            What we seem to have here is mTLS enabled for the default namespace and enabling it for connections to the cluster service. The DAC then is in permissive mode.

            patrick.stephens Patrick Stephens (Inactive) added a comment - - edited It looks like the operator cannot talk to the API server so it probably needs a rule to allow that. I'm not sure I follow the specifics here as to what we're trying to do - enable mTLS globally? That would be like so: https://docs.openshift.com/container-platform/4.6/service_mesh/v2x/ossm-security.html#ossm-security-enabling-strict-mtls_ossm-security We then need to tweak it to allow the DAC in permissive mode and any ingress the user wants. What we seem to have here is mTLS enabled for the default namespace and enabling it for connections to the cluster service. The DAC then is in permissive mode.

            Yes, We are trying to enable mTLS STRICT globally other than for DAC deployment.

            A peer authentication rule followed by destination rule was created to run DAC with mTLS PERMISSIVE as indicated in peerauth-dac and destirule-dac yaml file attached and it brings up the DAC pod with sidecar successfully.

            Similarly Operator was stated to run in STRICT mode with required peer authentication and destination rules created. This results in the error mentioned in the description.

            I'm not sure why irrespective of peer auth and destination rules in place would Operator fail to connect to API server? 

            prateek.kumar Prateek Kumar added a comment - Yes, We are trying to enable mTLS STRICT globally other than for DAC deployment. A peer authentication rule followed by destination rule was created to run DAC with mTLS PERMISSIVE as indicated in peerauth-dac and destirule-dac yaml file attached and it brings up the DAC pod with sidecar successfully. Similarly Operator was stated to run in STRICT mode with required peer authentication and destination rules created. This results in the error mentioned in the description. I'm not sure why irrespective of peer auth and destination rules in place would Operator fail to connect to API server? 

            Ah, this is down to that unsupported Istio version (EOL) in Openshift service mesh 

            patrick.stephens Patrick Stephens (Inactive) added a comment - Ah, this is down to that unsupported Istio version (EOL) in Openshift service mesh 

            Removing the fix version, PM to update the ticket with the next release version it'll be targeted for. 

            CC: Roshani Sanghavi

            prateek.kumar Prateek Kumar added a comment - Removing the fix version, PM to update the ticket with the next release version it'll be targeted for.  CC: Roshani Sanghavi

            People

              roshani.sanghavi Roshani Sanghavi (Inactive)
              prateek.kumar Prateek Kumar
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty