Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-1853

cluster re-creation race condition

    XMLWordPrintable

Details

    • Page
    • Resolution: Fixed
    • Minor
    • not-targeted
    • 2.1.0
    • documentation
    • None
    • 1

    Description

      2.1.0-239

      If you create a cluster, delete the cluster, wiating until all cb pods are deleted, then immeadiately re-create the same cluster (possibly with a different spec, but same name etc) you will see the following error message in the operator logs and the cluster will fail to come up:

      {
      "level":"info",
      "ts":1608067227.8315897,
      "logger":"cluster",
      "msg":"Reconciliation failed",
      "cluster":"default/cb-example-perf",
      "error":"Operation cannot be fulfilled on secrets \"cb-example-perf\": StorageError: invalid object, Code: 4, Key: /registry/secrets/default/cb-example-perf, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: eb635fd3-7b27-4de9-adbb-77d1140d07d8, UID in object meta: ",
      "stack":"github.com/couchbase/couchbase-operator/pkg/cluster/persistence.(*persistentStorageImpl).flush\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/persistence/persistence.go:164\ngithub.com/couchbase/couchbase-operator/pkg/cluster/persistence.(*persistentStorageImpl).Clear\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/persistence/persistence.go:153\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).create\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/cluster.go:231\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).reconcile\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/reconcile.go:117\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).runReconcile\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/cluster.go:392\ngithub.com/couchbase/couchbase-operator/pkg/cluster.(*Cluster).Update\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/cluster/cluster.go:418\ngithub.com/couchbase/couchbase-operator/pkg/controller.(*CouchbaseClusterReconciler).Reconcile\n\t/home/couchbase/jenkins/workspace/couchbase-k8s-microservice-build/couchbase-operator/pkg/controller/controller.go:90\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/couchbase/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/couchbase/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/couchbase/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.17.5-beta.0/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.17.5-beta.0/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/couchbase/go/pkg/mod/k8s.io/apimachinery@v0.17.5-beta.0/pkg/util/wait/wait.go:88"}
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            simon.murray Simon Murray
            korrigan.clark Korrigan Clark (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty