se2p
Pynguin does not create test cases
Hello,
I tried to generate test cases for a private project. Unfortunately Pynguin was not able to create a working test case, but there were some failed test cases. I tried all documented strategies but the situation did not really change. The functions in the module I used are a bit complex and the code is a mixture of functional and procedural code, there is no object oriented code.
Do you have an idea how I can create test cases with Pynguin? Do I have to change the code or are there known problems with code similar to that one I described? Can the problem be solved with changing the parameters for Pynguin?
Hi,
Thank you for your interest in Pynguin. From your description it is difficult to tell, what the exact issue is. Could you please elaborate a bit?
Best would be, if you could provide us access to your code (or some minimal working example showing the same behaviour, if sharing the original code was not possible). Furthermore, I'd be interested in the exact version and command-line parameters for Pynguin that you are using. This would allow me to try to reproduce the issue and give you a better answer.
Thank you in advance.
Best, Stephan
Hi Stephan,
thank you for your reply. I will try to create some minimal example.
I used the current version on the main branch, i.e. commit number 2f9d72a. I used the provided Dockerfile with a small change, I added
RUN apt-get update && apt-get install --no-install-recommends -y \
libgl1 \
libgomp1 \
libusb-1.0-0 \
&& rm -rf /var/lib/apt/lists/*
in line 40. These are requirements we need for some python packages used in this project.
The paramerters used were:
docker run --mount <many mount options> pynguin-container --algorithm WHOLE_SUITE --project_path /input --output_path /output --module_name <name of module> -v --budget 60
Where <many mount options> are for mounting the correct directories as described in the pynguin-docker.sh file. The algrithm used was changed, but none resulted in passing test cases. Also the budget was choosen to be higher, but I am not sure what is a good choice for this parameter.
Hi,
A minimal example (or if possible the original code) would be very helpful.
The command line and also the change to the Dockerfile seem reasonable, hence I assume it is related to the Python module.
Please note: the failing test cases are test cases that were generated by Pynguin, but that raised an exception during their execution. This might be, because Pynguin was not able to come up with correct input values for the functions in the code. In this case, we might need to extend Pynguin to support the generation of further inputs.
It might, however, also be possible that the exceptions were raised on purpose, i.e., the code raises some exceptions explicitly. In this case, unfortunately, Pynguin is not yet able to handle this correctly. We are working on ideas to analyse the code for such exceptions in order to provide reasonable assertions for them, i.e., wrap the code, e.g., by a with pytest.raises(...) block to explicitly check for that exception.
However, to give a proper answer, it would be great to have some source code to play around with.
Best, Stephan
Hi Stephan,
I could create a minimal example that shows the behavior described above:
import numpy as np
def function(rows: int,
cols: int,
intrinsic: np.ndarray):
xx, yy = np.meshgrid(np.arange(cols), np.arange(rows))
c_x, c_y = intrinsic[0, 2], intrinsic[1, 2]
f_x, f_y = intrinsic[0, 0], intrinsic[1, 1]
I = np.zeros((rows, cols, 3), dtype=np.float32) # noqa: E741
I[:, :, 0] = (xx - c_x) * (1. / f_x)
I[:, :, 1] = (yy - c_y) * (1. / f_y)
I[:, :, 2] = 1.
return I
There could be better input checking, for not getting exceptions from numpy functions and further calculations. But as I understood your explanation, Pynguin is currently not able to handle exceptions, I think this is the problem of the code I want to test.
Is it possible to give hints for Pynguin to choose only arguments, that are valid and do not raise exceptions? I have something in mind like restricting ints to positive values only.
I also saw tests for numpy.ndarray. This is not really important for the code we want to test, we assume that numpy is already tested. Can I exclude certain functions or modules from the test generation?
Best, Jonas
Hi Jonas,
Sorry for the late reply, I finally managed to dig into this a bit.
First of all, thank you for the minimal example. I've now investigated on this using Pynguin v0.9.2, with the following command-line arguments (used a fresh virtual environment with Python 3.8, and latest numpy in it)
pynguin \
--algorithm MOSA \
--project-path . \
--output-path /tmp/pynguin-results \
--module-name numpy_github \
--budget 60 \
--seed 42 \
--statistics-backend CONSOLE \
--log-file /tmp/pynguin-results/log.txt -v
The first thing I noted is that—as you already mentioned—Pynguin generated only one failing test case, which looks like the following
# Automatically generated by Pynguin.
import numpy_github as module0
def test_case_0:
try:
var0 = -3169
var1 = None
var2 = module0.func(var0, var0, var1)
except BaseException:
pass
The generated log file tells me that Pynguin parses the function parameter's signature to (int, int, np.ndarray), which is correct. It then, however, struggles to figure out how to generate an np.ndarray instance. The underlying inspect library parses this class' constructor signature to something like (self, /, *args, **kwargs) with the types args: typing.List[typing.Any], kwargs: typing.Dict[str, typing.Any]. If I understand numpy correctly here (I am by far no expert with that library), it would be necessary to at least specify a shape to generate a valid ndarray object? Pynguin fails to do so, however, as it does not know about the shape parameter. This truly is something that should be solvable by improving the code parsing, which currently only relies on Python's inspect library.
Failing to generate a valid np.ndarray object causes an exception latest when the intrinsics variable is accessed in the second line of the function's body. For that case, as you have seen, the aforementioned test is exported.
Regarding your further questions:
Is it possible to give hints for Pynguin to choose only arguments, that are valid and do not raise exceptions? I have something in mind like restricting ints to positive values only.
Unfortunately no, at least not yet. This would, however, be a nice thing (similar to libraries like hypothesis allow to restrict values using its assume function. I put this on our feature list.
I also saw tests for numpy.ndarray. This is not really important for the code we want to test, we assume that numpy is already tested. Can I exclude certain functions or modules from the test generation?
I would be interested in a concrete generated test code snippet here. Pynguin should only generate tests for functions/classes/methods that are declared in the module under test. It, however, needs to generate imported object types, too, to fulfill the requirements of the code under test. Does this maybe come from trying to do that?
Anyway, this is another item for our to-do list (we've seen such a request before).
Overall, I need to admit that Pynguin has serious struggles with such a complex library as numpy as a dependency. We need to improve our tool here. Unfortunately, in the current state I see no easy way to make it work for your example.
Best, Stephan
Hi Stephan,
thank you for your reply. This helps a lot to understand the problem. In my opinion it would really be great if Pynguin could create also test with complex libraries. Maybe it is possible to give hints for Pynguin how the constructor of classes like np.ndarray should be used?
As you asked for an example of a test case for np.ndarray, here is one:
import numpy as module0
import testing as module1
def test_case_0():
try:
var0 = module0.ndarray()
except BaseException:
pass
def test_case_1():
try:
var0 = None
var1 = 5104
var2 = [var0]
var3 = module0.ndarray(*var2)
assert var3 is not None
var4 = module1.functionvar0, var1, var3)
except BaseException:
pass
This example was created with the following command line:
sudo docker run --mount type=bind,source="/home/scoobe3d/prog/python_polarization",target=/input,readonly --mount type=bind,source="/tmp/pynguin",target=/output -v /home/scoobe3d/prog/python_polarization:/package:ro pynguin:0.0.1 --algorithm MIO --project_path /input --output_path /output --module-name dmo_testing -v
Things like test_case_0 and the first part of test_case_1 occured some times, but I have the impression that this is also dependent on the algorithm used.
Thank you for your detailed explanation and the time for investigating this problem. I really like the Pynguin project, keep up the good work on it.
Best Jonas
Hi Jonas,
Thank you for the code snippet. Without further investigation, I have an assumption for test_case_0: When Pynguin generates test cases it basically adds/remotes/mutates statements. Afterwards, it executes them. In this case, it starts the sequence of statements with the call to np.ndarray(); it omits any arguments since (as I elaborated previously) it assumes the arguments are *args and **kwargs, which could be omitted. Unfortunately, numpy does not allow this:
>>> import numpy as np
>>> np.ndarray()
Traceback (most recent call last):
File "<input>", line 1, in <module>
TypeError: ndarray() missing required argument 'shape' (pos 1)
Hence, Pynguin sees an exception, truncates all possibly following statements from the test case, and yields the aformentioned test case to the output.
Things like test_case_0 and the first part of test_case_1 occured some times, but I have the impression that this is also dependent on the algorithm used.
Yes, both this and the used seed for the random-number generator. Here, test generation is a random process.
In my opinion it would really be great if Pynguin could create also test with complex libraries. Maybe it is possible to give hints for Pynguin how the constructor of classes like np.ndarray should be used?
I am currently working on such things, not directly targetting numpy, but providing a better inference and analysis of the code under test and its dependencies. This will, however, take its time. I try to remind myself of referencing this issue, when it's done :wink:
Best, Stephan