Generate tests for fine-grained targets instead of whole modules

[kitsiosk]

Is your feature request related to a problem? Please describe. In many cases, one may need to generate tests for a specific class, function/method, or even a block of lines within a funciton, for example if someone suspects a faulty behavior in that function.

Describe the solution you'd like Instead of passing the target module as parameter, a target function could be given.

Describe alternatives you've considered

Additional context I do not know if this is possible, given the random mutations that take place, but if you think it is, I would be happy to give it a shot.

[LuKrO2011]

Thanks for bringing this up — we were considering that feature previously as well. Sadly, this is not as trivial as it might seem.

There are currently two parameters that allow you to somewhat restrict the test generation: One could use ignore_methods or ignore_modules which blacklist code from being analyzed, meaning it's excluded from test_cluster generation and cannot be invoked by Pynguin. However, this can be problematic when such methods are required (e.g., to construct objects necessary for testing). Thus, for code that should remain reachable but just isn't a coverage goal, this mechanism falls short.

Currently, the whole module under test is instrumented for measuring coverage. Potential adjustments to the coverage measuring might be:

1. Partial Bytecode Instrumentation: Instead of instrumenting entire methods, we could insert goals selectively in specific bytecode locations.

2. Post-Instrumentation Bytecode Analysis: Analyze the bytecode to map blocks back to methods and use that to filter goals.

3. Switch to Python 3.12+ with sys.monitoring: Rework instrumentation using Python’s new monitoring capabilities.

Due to simplicity and logic (1) should probably be preferred over (2), though if (3) proves viable, it might be worth adopting that directly.

Whichever solution is applied takes a significant effort to implement, and while we keep this in mind, this feature is currently not our top priority.

[BergLucas]

As @LuKrO2011 said, I also think that option 2 is the hardest and least practical to implement. It would require changing the CoverageFunctions while having lost raw information from the SUT. Option 3 would also require huge changes to Pynguin's architecture, so it's not really a viable option for such a small feature. Given the current architecture, option 1 is therefore the "easiest" to implement, and would only require changing the InstrumentationAdapters so that they instrument only some parts of the SUT. However, this still requires quite a lot of work if we want every type of coverage present in Pynguin to work properly.

[kitsiosk]

Thanks both for your prompt and detailed feedback.

Soon I will clear up some time and I can start an exploration of option 1 in a local fork to see how it goes.

Until then, feel free to close this one if you wish.

[LuKrO2011]

We would appreciate that! I am happy to review any changes so that in the end your code can be merged into this repo. I will keep this issue open as an open feature request for now.

[kitsiosk]

Hi @LuKrO2011, @BergLucas,

I have some time to work on this now. Based on our previous discussion, you had reached the conclusion that

option 1 is therefore the "easiest" to implement, and would only require changing the InstrumentationAdapters so that they instrument only some parts of the SUT

Is this still relevant, now that #115 is opened, or did anything change?

Thanks a lot for your help!

P.S.: The need for fine-grained targets started from this paper ("the most common issue (10/20) is that the tests do not cover the changes"). I leave it here in case you'd like to read more and because it discusses yet another useful application of Pynguin.

[BergLucas]

Hi @kitsiosk,

Initially, I just wanted to fix #4 in PR #115. However, while refactoring the PR to make it cleaner in response to the reviews, I realised that it would be better to integrate the elements of this issue already, because otherwise we could end up having to change everything later to incorporate it. Therefore, a CLI parameter --only-cover has been introduced to enable the coverage of only specific classes or functions. However, from reading your paper, it may be necessary to have an even finer-grained target because if a patch is in a function with many branches, the scope could still be too large. Perhaps we could create another parameter that would allow the user to specify lines of code, or maybe you have other ideas to make it more convenient for your use case. I have the feeling that an annotation system like # pragma: no cover wouldn't be very practical in your case, but I may be wrong.

[kitsiosk]

Thanks, I'll give it a run with --only-cover and see how it goes.

More probably than not, I will eventually need to go for line-level targets though. In that case, would it be a straightforward extension of the --only-cover you added, or would it require deeper changes?

[BergLucas]

Hi @kitsiosk,

The fully-qualified names of the functions and the annotations are already converted into lines of code so that they can be matched with the code objects being instrumented in the InstrumentationAdapters. Therefore, it should be fairly straightforward to add additional lines of code from the CLI parameters.