Major Description
We have about 360 tests in P0 and runtime is now 4hours. These tests are crucial for validating major changes and can delay release process if we cannot get a reasonable turnaround on runs.
The purpose here is to cover more features than sanity, however the current set of tests covers multiple permutations of the same feature, for instance:
TestBackupFullIncremental – sanity 7m 23s
|
TestBackupFullIncrementalS3 – sanity 7m 10s
|
TestBackupFullIncrementalAzure – sanity 9m 32s
|
TestBackupFullIncrementalGCP – sanity
|
Where we can just test TestBackupFullIncremental and later introduce additional variations in P1. This will us more foresight into what goes where regarding P0 vs P1. Then all else goes into P2.
Future future, I'm sure we can avoid setup in some cases by reusing an already provisioned cluster. Especially tests that validate the changing of some API settings. In these cases instead of doing full provisioning setup, just reset the server api settings and run the next tests.