Details
-
Bug
-
Resolution: Fixed
-
Major
-
6.5.0
-
6.5.0-4908
-
Untriaged
-
Unknown
Description
Note: this error is seen as far back as 4471
Cannot get a local repro, but on jenkins this test fails saying the backup manager failed to restore from the backup (on jenkins the failure is also intermittent, but it happens more often than not)
./testrunner -i /tmp/testexec.15656.ini -p dataset=bigdata,GROUP=P2,get-cbcollect-info=True -t 2i.gsi_alter_index_replicas.GSIAlterIndexesTests.test_backup_restore_unhealthy_replica,doc-per-day=1000,nodes_init=4,services_init=kv:n1ql-index-index-index,groups=simple,reset_services=True,GROUP=P2
This test does the following
1 kv+n1ql node, 3 index nodes
it loads 400k docs,
it creates this index CREATE INDEX idx1 ON default(_id) USING GSI WITH
now it takes a backup of the cluster as is
it wipes this cluster and reinitializes a new cluster 1 kv+n1ql node, 3 index nodes
create default bucket again
creates the same index again CREATE INDEX idx1 ON default(_id) USING GSI WITH
failsover a node that contains one of the replicas of the index
rebalances out this node
restores from the backup at this point
restore fails
We see this error:
[2019-12-04 15:02:10,361] - [remote_util:2991] INFO - Error restoring cluster: Internal server error while executing "POST http://172.23.105.251:9102/restoreIndexMetadata?bucket=default" check the server logs for more details
Looking at the index logs on this timestamp we see this snippet for 172.23.105.251:
2019-12-04T15:02:10.354-08:00 [Error] requestHandler.makeCreateIndexRequest(): create index request fails. Error=Index default.idx1 already exists
Will attach logs from all the servers.
Attachments
Issue Links
- Clones
-
MB-37175 [backup][2i] cluster fails to restore
-
- Closed
-