simonmichael
Hspec as an alternative test framework?
@simonmichael does anything speak against hspec as alternative test framework?
Downsides:
- three more dependencies (hspec, hspec-core, hspec-contrib)
- a bit more complex argument handling or a different way to pass arguments to hspec
Benefits:
- a
--fail-fastoption, see https://github.com/haskell/test-framework/issues/55 (I have many long running tests that I'd like to stop on first failure). - print cpu time
- colored diffs (?)
- dry-run, to see what test files are loaded
- re-run failed tests only
- read options from
.hspecconfig file - some more, e.g. html output
Sources:
- Compare
shelltest -hand http://hspec.github.io/options.html - https://hackage.haskell.org/package/hspec-core-2.7.2/docs/Test-Hspec-Core-Runner.html#g:2
- https://github.com/haskell/test-framework/issues/55
Thanks! I agree that we should move off test-framework, if only because it's no longer maintained.
Of the current big two, tasty and hspec, I think tasty is the most successful and the most future-proof, and I'd favour choosing that. But I'll review this when I can.
Hi @schoettl, when you have time could you rebase this against latest master.
I had a look at this. I see both runners are still supported for now, I guess there's no harm in keeping both until we have tested a bit more. Merging, thank you!
CI failed with stackage nightly/GHC 9.4, so we probably need some bounds or code changes for newer hspec. https://github.com/simonmichael/shelltestrunner/actions/runs/4199072857
Belatedly testing this; at least in the shelltestrunner and hledger repos, it seems to run a few tests but very slowly/hangingly. To reproduce in this repo, add --hspec in Makefile and run make test. @schoettl would you be able to look at this ?
Repro:
<
a
$ cat
a
$ shelltest a.test --hspec
It works with echo but not cat. Perhaps the fix is in shelltest.hs:runCommandWithInput.
Hi @simonmichael, thanks for testing and fixes. I plan to look at it this weekend.