Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-2756

Operator send error message in loop about forbidden access to leases.coordination.k8s.io

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Major
    • None
    • 2.3.0
    • operator
    • 1

    Description

      Deploying a fresh new CB cluster on K8s with CAO 2.3.0 (no upgrade, just try to create a basic CB cluster from scratch).

      THe CAO keeps sending error message in loop about forbidden access to leases.coordination.k8s.io :

       

      {"level":"info","ts":1655253115.0067184,"logger":"main","msg":"couchbase-operator","version":"2.3.0 (build 301)","revision":"207fa7b86260bc7c75c1c1868cda753370991e64"}
      {"level":"info","ts":1655253115.6125796,"logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":"0.0.0.0:8383"}
      {"level":"info","ts":1655253115.613606,"msg":"Starting server","path":"/metrics","kind":"metrics","addr":"[::]:8383"}
      {"level":"info","ts":1655253115.6137633,"msg":"attempting to acquire leader lease dev-couchbase/couchbase-operator...\n"}
      {"level":"error","ts":1655253115.6206286,"msg":"error retrieving resource lock dev-couchbase/couchbase-operator: leases.coordination.k8s.io \"couchbase-operator\" is forbidden: User \"system:serviceaccount:dev-couchbase:eks-helm-couchbase-operator\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"dev-couchbase\"\n","stacktrace":"k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire.func1\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:250\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:133\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:249\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).Run\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:206\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startLeaderElection.func3\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/internal.go:642"}
      {"level":"error","ts":1655253118.9478855,"msg":"error retrieving resource lock dev-couchbase/couchbase-operator: leases.coordination.k8s.io \"couchbase-operator\" is forbidden: User \"system:serviceaccount:dev-couchbase:eks-helm-couchbase-operator\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"dev-couchbase\"\n","stacktrace":"k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire.func1\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:250\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:133\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:249\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).Run\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:206\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startLeaderElection.func3\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/internal.go:642"}
      {"level":"error","ts":1655253122.0192041,"msg":"error retrieving resource lock dev-couchbase/couchbase-operator: leases.coordination.k8s.io \"couchbase-operator\" is forbidden: User \"system:serviceaccount:dev-couchbase:eks-helm-couchbase-operator\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"dev-couchbase\"\n","stacktrace":"k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire.func1\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:250\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:133\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:249\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).Run\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:206\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startLeaderElection.func3\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/internal.go:642"}
      {"level":"error","ts":1655253125.6897628,"msg":"error retrieving resource lock dev-couchbase/couchbase-operator: leases.coordination.k8s.io \"couchbase-operator\" is forbidden: User \"system:serviceaccount:dev-couchbase:eks-helm-couchbase-operator\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"dev-couchbase\"\n","stacktrace":"k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire.func1\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:250\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tk8s.io/apimachinery@v0.23.2/pkg/util/wait/wait.go:133\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:249\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).Run\n\tk8s.io/client-go@v0.23.2/tools/leaderelection/leaderelection.go:206\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startLeaderElection.func3\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/manager/internal.go:642"} 

      Current case is using Public Networking with external DNS.

      (AWS / Cloudflare as DDNS and Namecheap as DNS provider)

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            fabrice.leray Fabrice Leray
            fabrice.leray Fabrice Leray
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty