Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-2365

[OCP] Operator unable to connect to K8s API Server with Istio STRICT

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • .minor
    • 2.2.1
    • None
    • None
    • 3

    Description

      Setup Details:

      1. OCP Cluster version: 4.6.1
      2. ServiceMesh 2.0.7 installed on the Cluster. 
      3. Sidecar annotations added to the DAC and Operator Deployments.(sidecar.istio.io/inject: "true")
      4. Peer Authentication Rule created to run DAC with ISTIO PERMISSIVE.(peerauth-dac.yaml, namespace: default)
      5. Destination Rule created to use mTLS when sending requests to other services in the mesh for DAC.(destirule-dac.yaml, namespace: default)
      6. Created DAC deployment, Istio sidecar comes up with Status as Running

        Prateeks-MacBook-Pro:bin prateekkumar$ oc get pods NAME   READY   STATUS    RESTARTS   AGE couchbase-operator-admission-db8f6c4b4-79ncf   2/2     Running   0          125m  

      7. Created a new namespace test-istio-0
      8. Peer Authentication Rule created to run Operator with ISTIO Strict.(peerauth-op.yaml, namespace: test-istio-0)
      9. Destination Rule created to use mTLS when sending requests to other services in the mesh for Operator.(destirule-op.yaml, namespace: test-istio-0)
      10. Created Operator Deployment, Istio sidecar comes up, but Operator pod doesn't. Error:

        Prateeks-MacBook-Pro:bin prateekkumar$ oc get pods -n test-istio-0
        NAME                                  READY   STATUS             RESTARTS   AGE
        couchbase-operator-5fbc545565-fp9qq   1/2     CrashLoopBackOff   7          16m. 

      11. Prateeks-MacBook-Pro:bin prateekkumar$ oc logs pods/couchbase-operator-5fbc545565-fp9qq -n test-istio-0 couchbase-operator {"level":"info","ts":1629794425.0975158,"logger":"main","msg":"couchbase-operator","version":"2.2.1 (build 126)","revision":"b75530987818a959ec1f8984da92b5e2d3f615f7"} {"level":"error","ts":1629794435.1664348,"logger":"controller-runtime.manager","msg":"Failed to get API Group-Resources","error":"an error on the server (\"\") has prevented the request from succeeding","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/couchbase/go/pkg/mod/github.com/go-logr/zapr@v0.3.0/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/manager.New\n\t/home/couchbase/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.4/pkg/manager/manager.go:279\nmain.main\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/cmd/operator/main.go:83\nruntime.main\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/golangHc7tn/go1.16.3/src/runtime/proc.go:225"} {"level":"error","ts":1629794435.1665149,"logger":"main","msg":"Error initializing manager","error":"an error on the server (\"\") has prevented the request from succeeding","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/couchbase/go/pkg/mod/github.com/go-logr/zapr@v0.3.0/zapr.go:132\nmain.main\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/cmd/operator/main.go:91\nruntime.main\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/golangHc7tn/go1.16.3/src/runtime/proc.go:225"} 

      Since automatic mTLS Strict is not used, both Peer Authentication Rule and Destination Rule are required for deployments. (https://docs.openshift.com/container-platform/4.6/service_mesh/v2x/ossm-security.html#ossm-security-mtls-sidecars-incoming-services_ossm-security) 

      cbopinfo attached. Triage in progress.

      Attachments

        1. cbopinfo-20210824T150226+0530.tar.gz
          183 kB
        2. destirule-dac.yaml
          0.2 kB
        3. destirule-op.yaml
          0.2 kB
        4. peerauth-dac.yaml
          0.2 kB
        5. peerauth-op.yaml
          0.1 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            malarky Chris Malarky
            prateek.kumar Prateek Kumar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty