engine_testapp: Allow >1 attempt for a test to pass

We (unfortunately) have some tests driven by engine_testapp which intermittently fail - for example ep_testsuite. While much has been done to improve these test issues (particulary on newer branches), the fact remains that some tests are flaky and need retrying to pass.

For commit-validation jobs, this means that the whole job will fail, and has to be manually retried. Not only is the repeat job time-consuming (CV job runtimes vary, but say ~10mins on average), but it also requires human intervention to trigger the retry.

To attempt to improve this, we should add the ability for engine_testapp to attempt a
test multiple times before declaring it a failure. For branches under maintenance (3.1.x, sherlock) which we are unlikely to spend much effort in fixing intermittent failures, this should significantly speed up our ability to get changes through CV.