Add --repeat command line argument

[capsocrates]

Google Test, for all of its painpoints, has a few features that Catch2 does not. One of them is the --gtest_repeat command line argument that lets you run a given configuration of tests N times (where N may also be some sentinel value for "infinite"). I would love to have this feature in Catch2. Yes, I can write a script that does it, but it is much easier if it is native to the command line tool so that I can easily repeat whatever combination and configuration of tests that I want without mucking around with external scripts.

One of the motivating use cases for this is to identify and/or reproduce spuriously failing tests.

[rsalinas]

I just missed this feature. It should be fairly easy to implement.

[capsocrates]

I don't know if it will be easy to implement or not. It will require resetting all state; I don't know how difficult that is to do on the back-end.

[piponazo]

Hi! I would also love to see this feature in Catch2. I find it super interesting when dealing with Flaky tests. I could even try to work on it if some maintainer of the project could give me some guidance :)

[akalsi87]

Bump!

[horenmar]

If someone wants to work on this, it indeed shouldn't be too hard. There are two parts to it

1. Make sure that a test case can be re-entered after being run to completion once. I don't remember on top of my head whether this is the case right now, but if not, it just means that when a test case is fully completed, Catch2 has to delete its internal bookkeeping on which SECTIONs inside that test have been run.
2. Add a handling of repeat somewhere in Session::runInternal.

[eiszapfen2000]

I had a quick look at this, and immediately stumbled into two questions relevant for an implementation:

• If I understand correctly, --gtest_repeat runs the set of selected tests n times repeatedly. It does not run each of the selected test cases n times, but runs the entire set, then runs it again, and again, asoasf?
• How would this interact with the EventListener. Would testRunStarting() be still called only once? Or would it be called for each repetition? Or do we need a new method in IEventListener to properly model this?