panic observed when restoring FTS indexes to a newly initialized cluster using the new (collection aware) endpoint

Description:

Restoring full-text search indexes and aliases to a different cluster fails.

I think this is also affecting all other services as the individual service restoring tests are failing.

Steps to reproduce:

Pre-requisites: Two single node clusters A and B.

On Cluster A:

1. Initialise the single node in this cluster with all services apart from analytics and load the travel-sample bucket.
2. Create any full-text search indexes and aliases via the UI. I created an index named 'my_index' on the 'travel-sample' bucket targeting the 'type' attribute. I also created two aliases 'my_alias1' and 'my_alias2' targeting 'my_index'.
3. I took a backup of cluster A using cbbackupmgr.

   [root@node1-cheshire-cat-testing-centos7 home]# /opt/couchbase/bin/cbbackupmgr backup -c 10.112.210.101 -a my_archive -r my_repo --username Administrator --password password

   Backing up to '2021-01-19T17_02_37.221825587Z'

   Copied all data in 5.995276735s (Avg. 8.29MB/Sec)                                                                            63182 items / 41.46MB

   [========================================================================================================================================] 100.00%

   Backup completed successfully

   Backed up bucket "travel-sample" succeeded

   Mutations backed up: 63182, Mutations failed to backup: 0

   Deletions backed up: 0, Deletions failed to backup: 0

   Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0

1. When restoring to cluster A, the operation succeeds: 

   [root@node1-cheshire-cat-testing-centos7 home]# /opt/couchbase/bin/cbbackupmgr restore -c 10.112.210.101 -a my_archive -r my_repo --username Administrator --password password

   Restoring backup '2021-01-19T17_02_37.221825587Z'

   Copied all data in 9.92824559s (Avg. 4.60MB/Sec)                                                                             63182 items / 41.43MB

   [========================================================================================================================================] 100.00%

On Cluster B:

1. Initialise the single node in this cluster with all services apart from analytics.
2. When restoring to a different cluster, the operation fails: 

   [root@node1-cheshire-cat-testing-centos7 home]# /opt/couchbase/bin/cbbackupmgr restore -c 10.112.210.104 -a my_archive -r my_repo --username Administrator --password password --auto-create-buckets

   Restoring backup '2021-01-19T17_02_37.221825587Z'

   Transferring full text indexes for bucket 'travel-sample'                                                                             0 items / 0B

   [                                                                                                                                          ] 0.29%

   Error restoring cluster: error executing 'POST' request to '/api/v1/bucket/travel-sample/backup' response body was empty, check the logs for more details

What happens:

The restore of the full-text search aliases and indexes fails.

What I expected to happen:

I expected the full-text search aliases to be restored successfully.

The backup logs show:

[root@node1-cheshire-cat-testing-centos7 home]# tail my_archive/logs/backup-0.log 

2021-01-19T17:11:43.207+00:00 (REST) (Attempt 1) (POST) Dispatching request to 'http://10.112.210.104:8091/pools/default/buckets/travel-sample/scopes/%40ensureManifest/1'

2021-01-19T17:11:43.327+00:00 (REST) (Attempt 1) (POST) (200) Received response from 'http://10.112.210.104:8091/pools/default/buckets/travel-sample/scopes/%40ensureManifest/1'

2021-01-19T17:11:43.327+00:00 (Plan) (Bucket) Successfully created/edited scopes/collections for bucket 'travel-sample' | {"number":2,"duration":"137.151455ms"}

2021-01-19T17:11:43.327+00:00 (Plan) (Analytics) Transferring Analytics metadata for bucket 'travel-sample'

2021-01-19T17:11:43.327+00:00 (Plan) (Analytics) Successfully transferred Analytics metadata for bucket 'travel-sample' | {"number":3,"duration":"28.447µs"}

2021-01-19T17:11:43.327+00:00 (Plan) (Search) Transferring full text index definitions for bucket 'travel-sample'

2021-01-19T17:11:43.327+00:00 (REST) (Attempt 1) (POST) Dispatching request to 'http://10.112.210.104:8094/api/v1/bucket/travel-sample/backup?prevIndexUUID=%2A'

2021-01-19T17:11:43.333+00:00 ERRO: (REST) (Attempt 1) (POST) Failed to dispatch request to 'http://10.112.210.104:8094/api/v1/bucket/travel-sample/backup?prevIndexUUID=%2A': Post "http://10.112.210.104:8094/api/v1/bucket/travel-sample/backup?prevIndexUUID=%2A": EOF -- rest.(*Request).execute() at request.go:204

2021-01-19T17:11:43.333+00:00 (Stats) Stopping stat collection

2021-01-19T17:11:43.334+00:00 (Cmd) Error restoring cluster: failed to execute cluster operations: failed to execute bucket operation for bucket 'travel-sample': failed to transfer full text indexes for bucket 'travel-sample': failed to set full text index definitions: failed to post FTS index definitions: failed to execute request: http client failed to dispatch/receive request/response: error executing 'POST' request to '/api/v1/bucket/travel-sample/backup' response body was empty, check the logs for more details