pynguin seems not work on `arrow` project

[denini08]

Describe the bug

Im trying to use pynguing to run on the arrow project, but it doesn't seem to work. Could you tell me if I'm doing something wrong or is it a bug in Pynguin?

To Reproduce

I used docker for reproducibility:

1. run docker

docker run -it --rm python:3.10.14-slim-bullseye bash

2. install git

apt update && apt install -y git

3. clone the repo

git clone --depth=1 https://github.com/arrow-py/arrow
cd arrow

4. install the dependencies

pip3 install --upgrade pip setuptools wheel
pip3 install .[dev,test,tests,testing]
# Install additional requirements if available (within root + 2 nest levels excluding env/ folder)
find . -maxdepth 3 -type d -name "env" -prune -o -type f -name "*.txt" -print | while read -r file; do
    if [ -f "$file" ]; then
        pip3 install -r "$file"
    fi
done
pip3 install pytest pynguin

5. run the test generator

export PYNGUIN_DANGER_AWARE=true
pynguin -v --project-path . --output-path ./tests2 --module-name arrow.arrow

Screenshots

Software Version:

• OS: docker with python:3.10.14-slim-bullseye
• Python version 3.10.14
• Pynguin Version 0.40.0

additional information I plan to use Pynguin to create tests in several repositories that I am studying

[stephanlukasczyk]

Dear @denini08

Please excuse my late reply, I've totally missed the notification for your issue.

First, thank you for your interest in Pynguin and the report.

I have good and bad news: the bad news is that we see this issue in the execution every now and then. The problem is that the execution of a generated test is isolated in a separate thread, which for some reason could not be terminated correctly. Worse, it did not even return an execution result. We still have not fully understood why this happens and what the underlying module's properties cause it. Thus, we unfortunately have no real solution for this.

The good news is that @BergLucas is working on this issue and has some solution. You can try this out by using https://github.com/BergLucas/pynguin/tree/improvement/subprocess-execution instead of upstream Pynguin. Instead of installing Pynguin into your Docker container (step 4 of your report), do the following:

pip install poetry
git clone https://github.com/BergLucas/pynguin /pynguin
cd pynguin
git checkout improvement/subprocess-execution
poetry build
pip install dist/pynguin-0.41.0.dev0-py3-none-any.whl

To execute Pynguin, add --subprocess to its command line (step 5):

pynguin -v --project-path . --output-path ./tests2 --module-name arrow.arrow --subprocess

With this approach I was able to generate tests. But there's a caveat of the approach: it is slower than Pynguin's thread-based execution, which means that you have to wait longer for a result. I am discussing this with @BergLucas , what we can do about speed and also about integrating the approach into Pynguin's main repository. For now, you could try it out such that you can continue with your work.

[pietrolechthaler]

Hi @denini08 @stephanlukasczyk,

I'm currently working on a research project that involves generating test cases for a large number of Python repositories, and I ran into exactly the same issue while trying to run Pynguin.

I was wondering if there have been any updates on this issue, or if the workaround using the --subprocess flag from @BergLucas’s improvement/subprocess-execution branch is still considered the best solution.

Thanks in advance for any guidance or updates you can provide!

[BergLucas]

Hi @pietrolechthaler ,

Yeah, there have been updates. My branch has been merged into the official Pynguin repo after a bit of refactoring (#82), so in theory, it's better to use the main branch of the official repo now.

We have validated it with the usual Pynguin benchmark (the one used in the papers). However, after the PR was merged, we started working on another benchmark specialised on C-modules, which are often the cause of crashes, and we noticed that there are still some minor bugs with the subprocess execution in some particular cases. We're currently working on debugging and fixing them with @LuKrO2011.

I guess that the next version of Pynguin on PyPi will wait for the few bug fixes we're going to do. However, if you can't wait for these fixes, I suppose you can use the repo's main branch without real problems.

[BergLucas]

Works on main and v0.43.0.

I'm pretty sure this has been fixed since #82 and v0.41.0.

[LuKrO2011]

Based on your setup, @denini08, I executed the following:

git clone --depth=1 https://github.com/arrow-py/arrow
cd arrow
python3 -m venv .venv
source .venv/bin/activate
pip3 install .[dev,test,tests,testing]
find . -maxdepth 3 -type d -name "env" -prune -o -type f -name "*.txt" -print | while read -r file; do
    if [ -f "$file" ]; then
        pip3 install -r "$file"
    fi
done
pip3 install pynguin
export PYNGUIN_DANGER_AWARE=true
pynguin -v --project-path . --output-path ./tests2 --module-name arrow.arrow

I used the latest (0.43.0) Pynguin version and there was no issue. Thus, I can confirm the comment of @BergLucas that this issue is fixed.