SonarQube passed ignoring test coverage threshold

[Famondir]

Hi all, today I set up PyBuilder with SonarQube. It works fine. But even though I have not met my test coverage threshold (neither the one stated in PyBuilder e.g. 70 % nor the standard one in SonarQube 80 %). When I just run sonar-scanner outside of the PyBuilder pipeline it notes me that the criteria are violated in SonarQube marking it as a Failure.

It would be great to communicate the threshold from PyBuilder or adjust the used quality gate somehow. Maybe this is already possible and you can tell me how?

[arcivanov]

Could you please provide your build.py?

[Famondir]

Sure. I tried it with this version using pytest:

#   -*- coding: utf-8 -*-
from pybuilder.core import use_plugin, init

use_plugin("python.core")
# use_plugin("python.unittest")
use_plugin('pypi:pybuilder_pytest')
use_plugin("python.flake8")
# use_plugin("python.coverage")
use_plugin('pypi:pybuilder_pytest_coverage')
use_plugin("python.distutils")
use_plugin("python.sonarqube")


name = "Music-Suite"
default_task = "publish"


@init
def set_properties(project):
    project.set_property("pytest_coverage_xml", True)
    project.get_property("pytest_extra_args").append("-x")
    project.set_property("pytest_coverage_break_build_threshold", 70)

    project.set_property("sonarqube_project_key", "musicsuitmusicsuite:2024e")
    # project.set_property("sonarqube_project_name", "Music Suite")

And using the default test suite like this:

#   -*- coding: utf-8 -*-
from pybuilder.core import use_plugin, init

use_plugin("python.core")
use_plugin("python.unittest")
# use_plugin('pypi:pybuilder_pytest')
use_plugin("python.flake8")
use_plugin("python.coverage")
# use_plugin('pypi:pybuilder_pytest_coverage')
use_plugin("python.distutils")
use_plugin("python.sonarqube")


name = "Music-Suite"
default_task = "publish"


@init
def set_properties(project):
    # project.set_property("pytest_coverage_xml", True)
    # project.get_property("pytest_extra_args").append("-x")
    # project.set_property("pytest_coverage_break_build_threshold", 70)

    project.set_property("sonarqube_project_key", "musicsuitmusicsuite:2024e")
    # project.set_property("sonarqube_project_name", "Music Suite")

[arcivanov]

Well, I'm confused here. Does the coverage break without sonar cube? Because PyTest Coverage has nothing to do with SQ.

[Famondir]

When I run pyb instead of pyb run_sonar_analysis then the indication of not met thresholds is printed correctly in the terminal output.

And when I run the sonar scanner manually with sonar-scanner -Dsonar.projectKey=musicsuitmusicsuite:2024e -Dsonar.sources=. -Dsonar.host.url=http://localhost:9000 -Dsonar.token=sqp_xyz the not met threshold is violating the qualitygates correctly.

But when I run the sonar-scanner with PyBuilder it reports no quality gate issue. But I think I found the problem. After limiting the sonar-runner only on the src/main/python folder I got no issue there as well, because it only looked for new code and not the missing tests for the already existing code as it seems.

So when I added some more new code both ways report issues.

[arcivanov]

I'll keep this open as it might be a bug, but I'll need to create a repro case for it.