[backup][2i] cluster fails to restore

Note: this error is seen as far back as 4471

Cannot get a local repro, but on jenkins this test fails saying the backup manager failed to restore from the backup (on jenkins the failure is also intermittent, but it happens more often than not)

./testrunner -i /tmp/testexec.15656.ini -p dataset=bigdata,GROUP=P2,get-cbcollect-info=True -t 2i.gsi_alter_index_replicas.GSIAlterIndexesTests.test_backup_restore_unhealthy_replica,doc-per-day=1000,nodes_init=4,services_init=kv:n1ql-index-index-index,groups=simple,reset_services=True,GROUP=P2

This test does the following
1 kv+n1ql node, 3 index nodes
it loads 400k docs,
it creates this index CREATE INDEX idx1 ON default(_id) USING GSI WITH

now it takes a backup of the cluster as is
it wipes this cluster and reinitializes a new cluster 1 kv+n1ql node, 3 index nodes
create default bucket again
creates the same index again CREATE INDEX idx1 ON default(_id) USING GSI WITH

failsover a node that contains one of the replicas of the index
rebalances out this node
restores from the backup at this point
restore fails

We see this error:
[2019-12-04 15:02:10,361] - [remote_util:2991] INFO - Error restoring cluster: Internal server error while executing "POST http://172.23.105.251:9102/restoreIndexMetadata?bucket=default" check the server logs for more details

Looking at the index logs on this timestamp we see this snippet for 172.23.105.251:
2019-12-04T15:02:10.354-08:00 [Error] requestHandler.makeCreateIndexRequest(): create index request fails. Error=Index default.idx1 already exists

Will attach logs from all the servers.