[System Test Provisioned][Backup] Failed to read secrets

Test Steps

1. Deploy a 7 node AWS cluster running on 7.6 server version with following cluster configuration -
   1. KV - 3
   2. Index - 2
   3. Query - 2
2. Create a bucket named default having 1 scope and 10 collections apart from _default scope and _default collection.
3. Enable scheduled backups for this particular bucket with weekly full backups and incremental backups with 4 hour interval.
4. Load 100 million documents in each of the 10 collections.

Observation

First full Backup taken for this particular bucket failed with following error.

failed to execute cluster operations: failed to execute bucket operation for bucket 'default': failed to transfer bucket data for bucket 'default': failed to transfer key value data: failed to transfer key value data: stream closed with an unexpected error: DCP stream closed unexpectedly, check the logs for more information

Note

Cluster was not undergoing any failover or rebalance operation when the backup operation failed.