Details
-
Improvement
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
None
Description
From our discussions at the Supportability Design meeting, we agreed that we would not be able to solve the problem of collecting logs from Couchbase Server pods which have been deleted by the operator in the 1.0 timeframe.
A compromise was made to instead make it easy for users to setup persistent volumes for logs so that at least these get re-attached to incoming replacement pods. Alternatively, they can attach the persistent volume to their own pods to access the logs if needed.
While this does not solve the problem entirely, it is a very acceptable solution for most cases.
It was my understanding at the time of that meeting that it would be easy to map specific mount points (e.g. /opt/couchbase/var/lib/couchbase/logs) to persistent volumes.
From reviewing the documentation for persistent volumes it appears that is impossible to setup PVCs for logs (reference: https://github.com/couchbase/couchbase-operator/blob/master/docs/user/modules/ROOT/pages/couchbase-cluster-config.adoc#specserverspodvolumemounts).
It is possible to setup a ‘default’ volume mount, which mounts the entirety of /opt/couchbase/var/lib/couchbase, but this is not really acceptable to most users who require high performance from their disks (i.e. won’t want to use PVs for data or indexes). These users will instead want their data mounted on the local ephemeral storage, which is their only option for local storage before host storage becomes a persistent volume option within Kubernetes.
You could argue that we’re trying to solve a problem that won’t exist when host storage becomes GA within Kubernetes, but we do not know exactly when that will be.
I do not know the exact level of work required to add in a way of mapping ONLY the logs directory to a persistent volume, but I feel we have to get this added to 1.0 of the operator if at all possible.
Attachments
Activity
I think this is a request for better supportability for clusters that use ephemeral storage. We definitely missed this use case when putting together the persistent volumes design, but I don't think it will be a common use case for customers. My recommendation is that we only support users with persistent volumes for 1.0.
Description for release notes:
Known Issue: It is not possible to attach a persistent volume to the Couchbase logs directory for deployments when Couchbase data is not written to a persistent volume. This means that important debugging information may be unavailable during failure situations.
Workaround: Users should always deploy clusters in production with persistent volumes.
Good points here, I think we should definitely do this. Seems pretty simple to implement, and I'll experiment and see if there is anything being overlooked. One thing I know is that If user wants logs and Pod recovery they have to persist the 'default' path (/opt/couchbase/var/lib/couchbase) .... which means they'll end up having a mount inside of a mount. From doing a quick search, this doesn't seem to be a problem, but I remember we specifically did not want to mount the data volumes inside of the 'default' volumes in case something could go wrong with that approach.
The other aspect is that the user who wants Logs only persisted would not be able to recover the Pod. But they will be able to get the logs from the volume, which I'm understanding is the ask here. We'll simply create a new Pod as is done today and attach another persistent volume for the log path.