Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-2689

Fix Persistence Race after deleting CB cluster.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Minor
    • None
    • 2.3.0
    • operator
    • None
    • 1

    Description

      Create DAC, Operator Pod with the latest build and apply the default couchbase-cluster.yaml file. This will create a 3 node CB cluster with default bucket and secret cb-example-auth.

      Status of the pods: __ 

      Prateeks-MacBook-Pro:~ prateekkumar$ kubectl get pods
      NAME                                            READY   STATUS    RESTARTS   AGE
      cb-example-0000                                 1/1     Running   0          7m47s
      cb-example-0001                                 1/1     Running   0          6m50s
      cb-example-0002                                 1/1     Running   0          5m59s
      couchbase-operator-5b4cb9f599-l7r7p             1/1     Running   0          12m
      couchbase-operator-admission-65469748f6-tbg2x   1/1     Running   0          13m 

      Delete the couchbase-cluster.yaml file.

      Error seen in Operator logs: 

      {"level":"info","ts":1648022239.4163053,"logger":"cluster","msg":"Reconciliation failed","cluster":"default/cb-example","error":"required resource missing: unable to get admin secret cb-example-auth","stack":"github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcileAdminPassword\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/security.go:20\ngithub.com/couchbase/couchbase-operator/pkg/cluster.reconcileFuncList.run\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:54\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/reconcile.go:163\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:481\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).Update\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:524\ngithub.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/controller/controller.go:90\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
       
      {"level":"info","ts":1648022239.4254403,"logger":"cluster","msg":"Resource updated","cluster":"default/cb-example","diff":"  (\n  \t\"\"\"\n  \t... // 31 identical lines\n  \t  status: \"True\"\n  \t  type: Balanced\n+ \t- lastTransitionTime: \"2022-03-23T07:57:19Z\"\n+ \t  lastUpdateTime: \"2022-03-23T07:57:19Z\"\n+ \t  message: 'required resource missing: unable to get admin secret cb-example-auth'\n+ \t  reason: ErrorEncountered\n+ \t  status: \"True\"\n+ \t  type: Error\n  \tcurrentVersion: 7.0.3\n  \tmembers:\n  \t... // 6 identical lines\n  \t\"\"\"\n  )\n"}
      {"level":"info","ts":1648022239.6910443,"logger":"cluster","msg":"unable to update status","cluster":"default/cb-example","error":"admission webhook \"couchbase-operator-admission.default.svc\" denied the request: validation failure list:\nsecret cb-example-auth referenced by spec.security.adminSecret must exist"}
       
      {"level":"info","ts":1648022239.7046678,"logger":"cluster","msg":"Resource updated","cluster":"default/cb-example","diff":"  (\n  \t\"\"\"\n  \t... // 31 identical lines\n  \t  status: \"True\"\n  \t  type: Balanced\n+ \t- lastTransitionTime: \"2022-03-23T07:57:19Z\"\n+ \t  lastUpdateTime: \"2022-03-23T07:57:19Z\"\n+ \t  message: 'required resource missing: unable to get admin secret cb-example-auth'\n+ \t  reason: ErrorEncountered\n+ \t  status: \"True\"\n+ \t  type: Error\n  \tcurrentVersion: 7.0.3\n  \tmembers:\n  \t... // 6 identical lines\n  \t\"\"\"\n  )\n"}
       
      {"level":"error","ts":1648022239.8348606,"logger":"cluster","msg":"Status update failed","cluster":"default/cb-example","error":"admission webhook \"couchbase-operator-admission.default.svc\" denied the request: validation failure list:\nsecret cb-example-auth referenced by spec.security.adminSecret must exist","stacktrace":"github.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile.func1\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:415\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:499\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).Update\n\tgithub.com/couchbase/couchbase-operator/pkg/cluster/cluster.go:524\ngithub.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile\n\tgithub.com/couchbase/couchbase-operator/pkg/controller/controller.go:90\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"} 

      Not sure if this needs to be fixed hence no fix version provided but just in case. Possible duplicate of K8S-2661 where we saw this condition when trying to create CB pods, here we see this error after deleting CB pods.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            simon.murray Simon Murray
            prateek.kumar Prateek Kumar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty