Problem Definition

The #cp-mcr team are currently investigating snapshot backup/restore for Capella, in brief, that consists creating atomic snapshots of - on disk - node state (e.g. /opt/couchbase/var/lib/couchbase).

For the golden path restore, we have a workable/tested solution that requires no intervention from couchbase-server, however, we're currently working through the non-golden path which might.

There's a strong desire to handle the following two cases:

1. Soft Topology Change :: The cluster has changed topology, it must be fixed up to match the topology in the backup (i.e. the case where a customer has scaled up their cluster since their backup).
2. Cluster Clone :: Entirely new infrastructure is provisioned to match the topology in the backup.

The main areas of concern, are currently:

1. SRV Record :: The front door for the user
2. Hostnames :: The cluster node/hostnames
3. TLS Certificates :: Created by Capella based on the clusters "base" hostname (source)
4. Cluster/Node UUIDs :: A discussion I intend to address outside of this ticket

For the soft topology change case, Capella can save the hostnames and re-provision the infrastructure to match.

This is not the case for the cluster clone scenario as both clusters will need to coexist.

Potential Solutions

I can see two clear solutions to these problems, although I'm sure there's more:

1. Capella Remapping :: We use the /etc/hosts file to remap hostnames, and then produce a certificate which has multiple SAN entries.
2. ns_server Remapping :: We provide ns_server with a map from hostname to hostname and the config is rewritten.

For the first solution, I believe we have two problems:

1. Certificates :: We'll have a cluster with a certificate which is signed for hostnames from another cluster.
2. Hostnames :: Intra-node communication will continue with old hostnames where external connections would use new hostnames which is likely to become a debugging nightmare (i.e. due to logs).

For the latter solution:

1. chronicle :: Node names are currently stored - in part - in chronicle which rules out a trivial config rewrite.
2. Services :: Unknown unknowns on how/whether other services store node hostnames (i.e. does this extend further than ns_server?).

Request

Please could we investigate having a supported way to remap/rewrite hostnames in ns_server so that we can boot from snapshotted volumes running on new infrastructure.

We could provide a payload such as the following:

{

    "host1": "host3",

    "host2": "host4"

}

This could be used to cause:

1. host1 to remap itself
2. host1 to remap the other nodes in the cluster

There may be other information required that we could gather/provide.

Considerations

There's a desire to have this functionality work for the supported versions deployed in Capella (i.e. 7.1.x and 7.2.x).