[Cluster Clone/Node Remap] Analytics service is unavailable on the cloned cluster

Steps:

1. Initialise a 2 node cluster with services kv:cbas on the first node and kv:cbas on the second node. TLS is enabled and encryption level is set to strict.
2. Create 3 buckets - 2 Magma and 1 CouchStore. Load some documents into the buckets.
3. A dataverse 'mydataverse' is created. 3 datasets are created in 'mydataverse' with the following cbas queries. Everything is successfully created.

   CREATE DATASET mydataset0 ON `bucket-2`.`_default`.`_default`

   CREATE DATASET mydataset1 ON `bucket-1`.`_default`.`_default`

   CREATE DATASET mydataset2 ON `bucket-0`.`_default`.`_default`

   CONNECT LINK Local;

1. Perform node remapping process from the respective source to destination nodes:
   172.23.122.140 => 172.23.122.155
   172.23.122.144 => 172.23.122.154
2. The following config_remap command was used. Command is executed without the --regenerate-bucket-uuids flag. Therefore, bucket UUID remains the same on the source and the destination cluster.

   /opt/couchbase/bin/escript /opt/couchbase/bin/escript-wrapper --initargs-path /opt/couchbase/var/lib/couchbase/initargs -- /opt/couchbase/bin/config_remap --initargs-path /opt/couchbase/var/lib/couchbase/initargs --output-path /output_dir --remap ns_1@172.23.122.144 ns_1@172.23.122.154 --remap ns_1@172.23.122.140 ns_1@172.23.122.155 --regenerate-cookie --regenerate-cluster-uuid

1. After the completion of this process, server was started on the cloned nodes, and when tried to query the cbas service, it returned the following error:

   USE mydataverse; SELECT VALUE d.DataverseName || "." || d.DatasetName FROM Metadata.`Dataset` d WHERE d.DataverseName = "mydataverse";

   2024-06-25 05:15:40,194 | infra | DEBUG | MainThread | [Rest_Connection:_urllib_request:305] POST https://172.23.122.155:18095/analytics/service body: {"pretty": true, "client_context_id": null, "statement": "USE mydataverse; SELECT VALUE d.DataverseName || \".\" || d.DatasetName FROM Metadata.`Dataset` d WHERE d.DataverseName = \"mydataverse\";"} headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': u'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/json'} error: 503 reason: [{u'msg': u'Analytics Service is temporarily unavailable', u'code': 23000}] {

   "requestID": "59071db9-d212-4cdc-bd18-f9b2ec3731fc",

   "clientContextID": "null",

   "errors": [{

   "code": 23000, "msg": "Analytics Service is temporarily unavailable" }

   ],

   "status": "fatal",

   "metrics": {

   "elapsedTime": "48.053328ms",

   "executionTime": "13.990506ms",

   "compileTime": "0ns",

   "queueWaitTime": "0ns",

   "resultCount": 0,

   "resultSize": 0,

   "processedObjects": 0,

   "bufferCacheHitRatio": "0.00%",

   "errorCount": 1

   }

   } auth: Administrator:password

1. On the UI, Analytics page displays messages such as this

   Failed to get bucket insights.

   Try refreshing the bucket insights.

1. Cb-collect logs of both the source and cloned cluster has been attached.