Unable to Retrieve Failing Test Information for Mutants

[serinesefardjelah]

I'm currently conducting research involving mutant generation using various tools. I wanted to use your tool as a baseline for comparison. However, I ran into a limitation: I couldn't find a way to identify which specific tests fail for each generated mutant.

I reviewed the documentation but didn't find any command-line options or configuration settings that would allow me to obtain this information. So far, I’ve only been able to retrieve the mutants and their overall state (e.g., Killed, Survived), but not the mapping between individual failing tests and specific mutants.

Could you please let me know if there’s a configuration or feature that enables this? This level of detail is essential for the analysis in my study.

Thank you in advance!

[boxed]

I don't remember if this is stored in the result json. You can check easily enough on a test project. In any case that would only be the first test to fail, not all, due to performance reasons.

[serinesefardjelah]

I checked the results.json output, and while it contains detailed test coverage mappings (which tests exercise which functions) and duration of each test, it doesn’t explicitly track which tests fail/pass for each mutant

[boxed]

Yea ok. Then you'd have to add this to mutmut if you need it.

[Otto-AA]

Hi @serinesefardjelah , I'd be curious how you use this information to compare the mutation tools.

To add this feature, I guess you would need to add a pytest plugin to the PytestRunner.run_tests method (which you can find here). The plugin usage would be similar to the StatsCollector, though probably with different hooks.