From our discussions at the Supportability Design meeting, we agreed that we would not be able to solve the problem of collecting logs from Couchbase Server pods which have been deleted by the operator in the 1.0 timeframe.
A compromise was made to instead make it easy for users to setup persistent volumes for logs so that at least these get re-attached to incoming replacement pods. Alternatively, they can attach the persistent volume to their own pods to access the logs if needed.
While this does not solve the problem entirely, it is a very acceptable solution for most cases.
It was my understanding at the time of that meeting that it would be easy to map specific mount points (e.g. /opt/couchbase/var/lib/couchbase/logs) to persistent volumes.
From reviewing the documentation for persistent volumes it appears that is impossible to setup PVCs for logs (reference: https://github.com/couchbase/couchbase-operator/blob/master/docs/user/modules/ROOT/pages/couchbase-cluster-config.adoc#specserverspodvolumemounts).
It is possible to setup a ‘default’ volume mount, which mounts the entirety of /opt/couchbase/var/lib/couchbase, but this is not really acceptable to most users who require high performance from their disks (i.e. won’t want to use PVs for data or indexes). These users will instead want their data mounted on the local ephemeral storage, which is their only option for local storage before host storage becomes a persistent volume option within Kubernetes.
You could argue that we’re trying to solve a problem that won’t exist when host storage becomes GA within Kubernetes, but we do not know exactly when that will be.
I do not know the exact level of work required to add in a way of mapping ONLY the logs directory to a persistent volume, but I feel we have to get this added to 1.0 of the operator if at all possible.