Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-1461

InitContainer is not idempotent

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 2.0.1
    • 1.0.0, 1.1.0, 1.2.0, 1.2.1, 1.2.2, 2.0.0
    • operator
    • None
    • 1

    Description

      The InitContainer, that should only be attached when the default PVC is first created, copies the couchbase/etc directory from the container on startup onto the default PV.
      This is where Couchbase Server stores its data directory settings so this special InitContainer handling was added in https://github.com/couchbase/couchbase-operator/commit/a8ef3845be8e7a23cf7e52b306bad2dac99d9cf0 to handle pod recovery.

      In normal circumstances, the flow is:

      • The etc directory is copied from the container onto the PV (this is default config).
      • The Operator updates the storage path using some REST API commands, updating the config on the PV.
      • On restore, the config from the PV is then used with the updated storage paths set by the operator, rather than the default paths.

      The problem is there are circumstances where this InitContainer runs after the PVC has already been created.
      One such circumstance would be a backup using Velero, where the pod definition is restored directly, including any InitContainers.
      In the field we have also observed other (unexplained) situations where the InitContainer is run on an existing pod.

      In this case, the 'updated' config stored on the PV is overwritten by the cp command in the InitContainer when the pod is recovered/restarted.
      Unlike the flow outlined above, the Operator is now unable to set storage paths via REST API as the pod is in the cluster, so we've just overwritten /mnt/data as the storage path with the default of /opt/couchbase/var/lib/couchbase.
      That means Couchbase Server is looking for data in the wrong directory (and is using the wrong PV) and we've essentially 'lost' (though can recover) all of the data on the data PV.

      If the InitContainer was idempotent, it could run multiple times without any issues, it would also simplify the code significantly.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              simon.murray Simon Murray
              matt.carabine Matt Carabine (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty