Add max testing duration
Transferred from http://code.google.com/p/scalacheck/issues/detail?id=83 #82 requests a timeout for any particular test. It would also be nice if Test.Params had an option for max duration of the entire testing run. For example, instead of specifying minSuccessfulTests and maxDiscardedTests, I'd like to just specify that testing go for 8 hours. There are any number of ways this feature could interact with the other parameters, as discussed on the mailing list. All would be fine with me!
I think the following behaviour makes most sense:
The test runner takes the following parameters: propertyTimeout (see #5) minSuccessfulTests maxDiscardRatio (see #11) minTotalDuration (this issue)
The test runner will keep evaluating one property over and over again until:
It takes longer than propertyTimeout to do one evaluation --> failure
The ratio between the discarded evaluations and successful evaluations goes above maxDiscardRatio --> failure
The number of successful evaluations has passed minSuccessfulTests AND minTotalDuration milliseconds has passed since the first property evaluation was done --> success
Note: This means that if you set minTotalDuration to > 0, it will take at least this time for EACH of your properties to pass, since the test parameters only concerns ONE property, not property collections.