Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-2756

Operator send error message in loop about forbidden access to leases.coordination.k8s.io

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Not a Bug
    • 2.3.0
    • None
    • operator
    • 1

    Description

      Deploying a fresh new CB cluster on K8s with CAO 2.3.0 (no upgrade, just try to create a basic CB cluster from scratch).

      THe CAO keeps sending error message in loop about forbidden access to leases.coordination.k8s.io :

       

      {"level":"info","ts":1655253115.0067184,"logger":"main","msg":"couchbase-operator","version":"2.3.0 (build 301)","revision":"207fa7b86260bc7c75c1c1868cda753370991e64"}
      {"level":"info","ts":1655253115.6125796,"logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":"0.0.0.0:8383"}
      {"level":"info","ts":1655253115.613606,"msg":"Starting server","path":"/metrics","kind":"metrics","addr":"[::]:8383"}
      {"level":"info","ts":1655253115.6137633,"msg":"attempting to acquire leader lease dev-couchbase/couchbase-operator...\n"}
      {"level":"error","ts":1655253115.6206286,"msg":"error retrieving resource lock dev-couchbase/couchbase-operator: leases.coordination.k8s.io \"couchbase-operator\" is forbidden: User \"system:serviceaccount:dev-couchbase:eks-helm-couchbase-operator\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"dev-couchbase\"\n","stacktrace":"k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire.func1\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:250\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:133\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:249\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).Run\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:206\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startLeaderElection.func3\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/internal.go:642"}
      {"level":"error","ts":1655253118.9478855,"msg":"error retrieving resource lock dev-couchbase/couchbase-operator: leases.coordination.k8s.io \"couchbase-operator\" is forbidden: User \"system:serviceaccount:dev-couchbase:eks-helm-couchbase-operator\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"dev-couchbase\"\n","stacktrace":"k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire.func1\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:250\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:133\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:249\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).Run\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:206\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startLeaderElection.func3\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/internal.go:642"}
      {"level":"error","ts":1655253122.0192041,"msg":"error retrieving resource lock dev-couchbase/couchbase-operator: leases.coordination.k8s.io \"couchbase-operator\" is forbidden: User \"system:serviceaccount:dev-couchbase:eks-helm-couchbase-operator\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"dev-couchbase\"\n","stacktrace":"k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire.func1\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:250\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:133\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:249\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).Run\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:206\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startLeaderElection.func3\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/internal.go:642"}
      {"level":"error","ts":1655253125.6897628,"msg":"error retrieving resource lock dev-couchbase/couchbase-operator: leases.coordination.k8s.io \"couchbase-operator\" is forbidden: User \"system:serviceaccount:dev-couchbase:eks-helm-couchbase-operator\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"dev-couchbase\"\n","stacktrace":"k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire.func1\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:250\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:133\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:249\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).Run\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:206\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startLeaderElection.func3\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/internal.go:642"} 

      Current case is using Public Networking with external DNS.

      (AWS / Cloudflare as DDNS and Namecheap as DNS provider)

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          I think you need to do 'helm repo update' ?

          Because this doesn't look like a 2.3 helm install, the Role resource has a 2.2 label

              helm.sh/chart: couchbase-operator-2.2.106 

          And so it's missing the Rbac role which includes leases:

          ref: https://github.com/couchbase-partners/helm-charts/blob/a567370a2b69a7119fda5c54bc8535a6830142b7/charts/couchbase-operator/templates/operator-deployment.yaml#L213

           

          tommie Tommie McAfee added a comment - I think you need to do 'helm repo update' ? Because this doesn't look like a 2.3 helm install, the Role resource has a 2.2 label     helm.sh/chart: couchbase-operator- 2.2 . 106 And so it's missing the Rbac role which includes leases: ref: https://github.com/couchbase-partners/helm-charts/blob/a567370a2b69a7119fda5c54bc8535a6830142b7/charts/couchbase-operator/templates/operator-deployment.yaml#L213  

          Hi Tommie McAfee ,

          You were right : I forgot to update the helm repo.

          Thanks a lot Tommie McAfee.

           

          Now it works perfectly.

          We can close this ticket.

          fabrice.leray Fabrice Leray added a comment - Hi Tommie McAfee , You were right : I forgot to update the helm repo. Thanks a lot Tommie McAfee .   Now it works perfectly. We can close this ticket.

          Read the Helm documentation and update helm repo + CRD accordingly !

          My fault  

          fabrice.leray Fabrice Leray added a comment - Read the Helm documentation and update helm repo + CRD accordingly ! My fault  

          People

            fabrice.leray Fabrice Leray
            fabrice.leray Fabrice Leray
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty