qcheck
qcheck copied to clipboard
0.18 `max_fail` regression
Consider this max_fail example from the documentation https://c-cube.github.io/qcheck/0.18/qcheck-core/QCheck/index.html
(which says "The 15 smallest counter-example lists will be printed" 😉):
With QCheck 0.17 three counterexamples are printed as expected:
utop # let test = QCheck.(
Test.make
~count:10_000 ~max_fail:3
(list small_nat)
(fun l -> l = List.sort compare l));;
val test : Test.t = QCheck.Test.Test <abstr>
utop # QCheck.Test.check_exn test;;
Exception:
test `anon_test_1` failed on ≥ 3 cases:
[1; 0] (after 27 shrink steps)
[1; 0] (after 16 shrink steps)
[1; 0] (after 13 shrink steps)
With QCheck 0.18 we only get one counterexample printed unfortunately - a regression:
utop # let test = QCheck.(
Test.make
~count:10_000 ~max_fail:3
(list small_nat)
(fun l -> l = List.sort compare l));;
val test : Test.t = QCheck2.Test.Test <abstr>
utop # QCheck.Test.check_exn test;;
Exception:
test `anon_test_1` failed on ≥ 1 cases: [1; 0] (after 13 shrink steps)
We should probably add a test for this case... 😉 ... or consider using mdx to ensure that the documentation's examples continue to work as intended.
mdx is cool, but having a unit test is always less flimsy :)