[FEATURE] Run test in random order

[fasenderos]

In order to improve the quality of user tests, it would be great to run tests in a random order based on a random seed to ensure that the tests are properly isolated from each other.

The use of a seed will allow to re-run tests in the same order to reproduce a failed case for a specific order.

Other test suites have this feature; for example, you can refer to jest randomize or vitest shuffle

[isaacs]

Do you mean the suite files, or the tests within a given file?

Tap doesn't defer your tests or put them in any sort of queue to be run at some later time, so "shuffling" is sort of a weird thing to consider. Tap just runs your test functions right away, unless it's already in the middle of something else.

For example:

import t from 'tap'
console.error('a')
t.test('b', t => {
  console.error('in b')
  t.end()
})
console.error('c')
t.test('d', t => {
  console.error('in d')
  t.end()
})
console.error('e')

Output:

a
TAP version 14
in b
# Subtest: b
    1..0
ok 1 - b # time=0.781ms

c
in d
# Subtest: d
    1..0
ok 2 - d # time=0.336ms

e
1..2
# { total: 2, pass: 2 }
# time=9.289ms

As you can see, it doesn't even know about the d subtest until the b subtest is completely finished and resolved. This determinism is baked into tap, so that definitely could not be changed by default, maybe as opt-in, and only if the tests are not sync.

If they're async (and maybe whatever shuffle opt-in forces them to be deferred and async), they could theoretically be shuffled. Today, if you set t.jobs to a number > 1, then it will run them in parallel, which shuffles them up a bit, but not quite what you're looking for, I think.

It's really not required for the tests within a given file to be isolated from one another, and it's pretty common for them not to be. For example, you might be testing specifically that a given series of actions has some effect, and using subtests as a sort of flow-control-plus-assertions is a handy way to do that.

[isaacs]

Approved feature: --shuffle flag will run test files in a random order, and print the seed that was used, which can also be set via --shuffle-seed=<number>. It won't have any effect on the contents of each individual test file, though.