Details
-
Bug
-
Resolution: Fixed
-
Critical
-
4.6.3, 5.0.0
-
Untriaged
-
Unknown
Description
Build : 5.0.0-3104
This is a system test where we generate a lot of incremental backups with constant KV ops, and merge these backups. When merging a merged backup with several incremental backups, merge fails with the error :
Backup Meta `/disk3/backup/backup/.merge/other-3ebf39ba0b75e24c10024712fd58c61e/range.json` not found
No. of nodes in src cluster : 3 (64GB RAM x 32 cores)
No. of Items in cluster : ~166M (some items expired)
No. of buckets : 2
Steps :
1. Start data loading on 2 buckets (65% creates, 25% updates, 10% deletes, 15% items have TTL set to 10 mins). This will run throughout the test
2. After 5 mins, take 1st backup
3. Rebalance out a node
4. After 5 mins, take 2nd backup
5. Rebalance in a node
6. After 5 mins, take 3rd backup
7. Failover and add back a node
8. After 5 mins, take 4th backup
9. Failover and rebalance out a node
10. After 5 mins, take 5th backup
11. Flush bucket
12. After 5 mins, take 6th backup
13. Merge all backups
14. Repeat Steps 2-12
15. At this point, there will be 7 backups (1 merged, 6 incremental)
16. Merge all backups
Merge fails with the above mentioned error. backup.log attached.
Attachments
Issue Links
- relates to
-
MB-24924 Backup fails to finish presumably due to network errors
- Closed
For Gerrit Dashboard: MB-24952 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
79735,2 | MB-24952: Report an error is there is one | master | backup | Status: MERGED | +2 | +1 |
79922,2 | MB-24952: Fix deadlock when an unexpected DCP error is received | master | backup | Status: MERGED | +2 | +1 |
79923,2 | MB-24952: Use the snap end seqno as the end seqno if it's larger | master | backup | Status: MERGED | +2 | +1 |
79935,3 | MB-24952: [Backport] Fix deadlock when an unexpected DCP error is received | 4.6.3 | backup | Status: MERGED | +2 | +1 |
79936,3 | MB-24952: [Backport] Use the snap end seqno as the end seqno if it's larger | 4.6.3 | backup | Status: MERGED | +2 | +1 |
79938,1 | MB-24952: Report an error is there is one | 4.6.3 | backup | Status: ABANDONED | 0 | 0 |