quickcheck
quickcheck copied to clipboard
quickCheckWithResult with low maxSuccess fails unexpectedly
Contrary to expectation, lowering the maxSuccess parameter may cause tests to fail:
> let p k k' = k /= k' ==> (k :: Int) /= k'
> quickCheckWithResult stdArgs p
+++ OK, passed 100 tests; 11 discarded.
> quickCheckWithResult stdArgs{maxSuccess=2} p
+++ OK, passed 2 tests; 11 discarded.
> quickCheckWithResult stdArgs{maxSuccess=1} p
*** Gave up! Passed only 0 tests; 10 discarded tests.
The explanation seems to be due to a combination of:
- discard ratio defaults to 10, which means we allow only 10 discarded results in total for the case
maxSuccess=1(maxSuccess=2gets 20 discards upfront) -
computeSizedoesn't ramp up the size quickly enough, it starts out[0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,2,2,2...regardless ofmaxSuccessparameter
Those computed sizes in particular seem a bit weird to me, but I can't say I have any kind of understanding of most of what's going on here. It does seem a bit at odds with the comment here https://github.com/nick8325/quickcheck/blob/7ff70fe9da61c95180ec90bd1ff074597efce37d/src/Test/QuickCheck/Test.hs#L228, which would make me expect [0,1,2,....