qcheck icon indicating copy to clipboard operation
qcheck copied to clipboard

0.18 `max_fail` regression

Open jmid opened this issue 4 years ago • 1 comments

Consider this max_fail example from the documentation https://c-cube.github.io/qcheck/0.18/qcheck-core/QCheck/index.html (which says "The 15 smallest counter-example lists will be printed" 😉):

With QCheck 0.17 three counterexamples are printed as expected:

utop # let test = QCheck.(
          Test.make
            ~count:10_000 ~max_fail:3
            (list small_nat)
            (fun l -> l = List.sort compare l));;
val test : Test.t = QCheck.Test.Test <abstr>
utop # QCheck.Test.check_exn test;;
Exception:
test `anon_test_1` failed on ≥ 3 cases:
[1; 0] (after 27 shrink steps)
[1; 0] (after 16 shrink steps)
[1; 0] (after 13 shrink steps)

With QCheck 0.18 we only get one counterexample printed unfortunately - a regression:

utop # let test = QCheck.(
          Test.make
            ~count:10_000 ~max_fail:3
            (list small_nat)
            (fun l -> l = List.sort compare l));;
val test : Test.t = QCheck2.Test.Test <abstr>
utop # QCheck.Test.check_exn test;;
Exception:
test `anon_test_1` failed on ≥ 1 cases: [1; 0] (after 13 shrink steps)

We should probably add a test for this case... 😉 ... or consider using mdx to ensure that the documentation's examples continue to work as intended.

jmid avatar Sep 22 '21 09:09 jmid

mdx is cool, but having a unit test is always less flimsy :)

c-cube avatar Sep 22 '21 11:09 c-cube