Low coverage generating tests on a ROS Node

[banzo]

We are trying to generate tests for our ROS project. Running Pynguin on a simple vehicle class shows a coverage of 0.187500 and the test generated is not very useful:

# Test cases automatically generated by Pynguin (https://www.pynguin.eu).
# Please check them before you use them.
import pytest
import vehicle as module_0


@pytest.mark.xfail(strict=True)
def test_case_0():
    module_0.Vehicle()

We are guessing that Pynguin gets lost at one point, and are looking for some insight on what we can do.

To Reproduce We made a minimal example here.

Expected behavior We would expect the coverage to be a bit higher, with some relevant tests (test on the speed_profile or even the quickstart example).

Software Version (please complete the following information):

• OS: Docker Ubuntu 22.04 + ROS Humble
• Python version: 3.10.12
• Pynguin Version: 0.34.0

[BergLucas]

Hi @banzo,

This can be explained by 2 reasons:

• Functions with the name main are not analysed by Pynguin because, in principle, their purpose is to be executed when the module is run, so they are generally tested end-to-end rather than via unit tests.
• To instantiate the Vehicle class, you need to call the rclpy.init function and not pass any arguments. However, since Pynguin has no information on possible *args and **kwargs, it will test random values or not call the rclpy.init function before instantiating a Vehicle.

As a result, there's only a small chance that a Vehicle will be instantiated, and so the code coverage remains at 18.75%, which corresponds to the code executed when the module is imported + the coverage of the constructor up to the point where exceptions are created.

Here are the exceptions that are usually thrown by the constructor:

• NotInitializedException('rclpy.init() has not been called', 'cannot create node')
• TypeError("Node.__init__() got an unexpected keyword argument '6$D%'.V'")
• TypeError('Node.__init__() takes 2 positional arguments but 6 were given')
• ...

By adding a small function create_vehicle, I managed to get at least 80% of code coverage because I've forced Pynguin to sometimes instantiate a valid vehicle and start communication before allowing it to call methods on the vehicle, but it's not a perfect solution:

def create_vehicle() -> Vehicle:
    init(args=None)
    return Vehicle()

In any case, I'm not sure that, without modification, Pynguin can easily generate good tests for this type of library because there's an external component (the Docker container) and Pynguin can't control it. In my master's thesis, I started working on a fork of Pynguin with a plugin system that lets you add extra knowledge to generate better tests, but I haven't yet had time to propose a merge of the changes on this repo. However, a plugin that forces Pynguin to initialize communication before each test and cut it off as soon as the test has been run could potentially solve this problem.

I hope this answers your questions.

Have a nice day!