Random tests should run in the same relative order when some are added or removed.

[kevincox]

Currently if you have a list of tests that you are running and you add or remove one the entire list gets shuffled differently. It would be much nicer if adding or removing tests kept the same random order. Effectively adding or removing tests from the list.

This makes debugging very difficult as you can't easily narrow down the lists of tests to find the one that is causing the problem.

[kevincox]

FWIW here is an implementation. However it should use an actual consistent hashing function rather then python's builtin hash which I don't think promises to be consistent. However it was enough for my problem. Just replace the old shuffle call with this line.

seed = str(config.option.random_seed)                                                           
def key(item):                                                                                  
    return hash(seed + str(hash(item)))

items.sort(key=key)