Details
-
Bug
-
Resolution: Fixed
-
Critical
-
4.6.3, 5.0.0
-
Untriaged
-
Unknown
Description
Build : 5.0.0-3104
This is a system test where we generate a lot of incremental backups with constant KV ops, and merge these backups. When merging a merged backup with several incremental backups, merge fails with the error :
Backup Meta `/disk3/backup/backup/.merge/other-3ebf39ba0b75e24c10024712fd58c61e/range.json` not found
No. of nodes in src cluster : 3 (64GB RAM x 32 cores)
No. of Items in cluster : ~166M (some items expired)
No. of buckets : 2
Steps :
1. Start data loading on 2 buckets (65% creates, 25% updates, 10% deletes, 15% items have TTL set to 10 mins). This will run throughout the test
2. After 5 mins, take 1st backup
3. Rebalance out a node
4. After 5 mins, take 2nd backup
5. Rebalance in a node
6. After 5 mins, take 3rd backup
7. Failover and add back a node
8. After 5 mins, take 4th backup
9. Failover and rebalance out a node
10. After 5 mins, take 5th backup
11. Flush bucket
12. After 5 mins, take 6th backup
13. Merge all backups
14. Repeat Steps 2-12
15. At this point, there will be 7 backups (1 merged, 6 incremental)
16. Merge all backups
Merge fails with the above mentioned error. backup.log attached.
Attachments
Issue Links
- relates to
-
MB-24924 Backup fails to finish presumably due to network errors
- Closed