Run tests on multiple cores

[amigalemming]

There are packages like pqc and tasty-quickcheck that allow to run tests in parallel on multiple cores. Since they run tests on different properties in parallel, they do no longer show the increasing test counter. Would it be possible to run different tests on the same property in parallel? This would still enable the live counter. Maybe it would even save some memory if tests on the same property share some data. Maybe the parallel testing could be enabled by a new field of Args for the number of parallel testing threads?

[nick8325]

That would certainly be a nice feature!

It would require re-working the main QuickCheck loop in Test.QuickCheck.Test to spawn several tests and collect the reults. Possibly a bit of work, but something we probably ought to do.

[sol]

@amigalemming Just as a side note, it is possible to show progress information, even when running QuickCheck properties in parallel. hspec does exactly that.

The hairy details:

• We assume that the user runs a test suite with n tests.
• The user specifies N, the number of parallel tests to run at a time.
• Hspec's runner spawns one thread for each test, t₁ to tₙ.
• Each thread tₓ has an associated Mₓ :: MVar (Either Progress Result).
• Each thread waits on a semaphore with the capacity N.
• Once a thread tₓ is signaled to run, it runs the associated test; while doing so it writes progress information into Mₓ and once it's done the result (that is write if the MVar is empty, otherwise replace).
• The applications main thread folds over the list M₁..ₙ
• For each Mₓ it reads the current value and writes progress information as long as it gets a Left-value.
• Once it reads a Right-value it reports the result to the user and moves on to Mₓ₊₁.

With this approach we see progress information for the currently running test with the lowest x only. In practice this works out nicely.

/cc @feuerbach

[amigalemming]

It seems that QuickCheck-2.11 uses more cores, at least top reports CPU usage higher than 300%. If so, I think it is not a good idea to parallelize without permission by the user. How can I limit the number of cores/threads used?

[nick8325]

No, it doesn't parallelise automatically! Not sure what's going on there.

[Rewbert]

I have implemented this :) With some blood, sweat, and tears, it should be in 2.16. Stay tuned

[phadej]

How soon 2.16 is expected? If it's "soon", than I won't care updating to QuickCheck-2.15 at all in between.

[Rewbert]

@phadej There is no timeline. I think it will be a nightmare to merge, but worth it.

[Rewbert]

To clarify; I would not wait if I were you.

[Rewbert]

Here is a link to the paper describing the work done on this: https://arxiv.org/pdf/2404.16062