osquery icon indicating copy to clipboard operation
osquery copied to clipboard

ci: Add code coverage for macOS and Linux

Open Smjert opened this issue 3 years ago • 1 comments

  • Add the OSQUERY_ENABLE_COVERAGE CMake option, to enable building with code coverage using LLVM. This also adds a generate_coverage target to be run after the tests have run.

  • Add a helper for declaring a test, so that it's possible to automatically set the coverage profile file path in the environment variables.

  • Modify the python tests helpers so that when the LLVM_PROFILE_FILE env var is detected, we append a pid to it, to avoid the profile file to be overriden by multiple executions of the osqueryd binary.

  • Update the docker image to install lcov, also create the Linux unprivileged_user user directly in the image itself, instead of recreating it everytime during the build.

  • Correct user owning the submodule cache on Linux. It should be root; the unprivileged user is only needed for the tests to work, so only the build artifacts needs to have that user. Also fix an issue with the user owning the sources, since a recent security fix on git makes it complain if the sources are owned by a different user.

Depends on #7672 and #7671

Smjert avatar Jul 03 '22 20:07 Smjert

Overview

This PR adds the support for generating code coverage using clang/LLVM, It sets the necessary compilation flags so that the binaries that are run spit out a profraw file (by default named default.profraw), which contains the profiling of that binary (which paths it took).

Since we want to know this information for the tests and we don't want each of them to overwrite the same file, we can pass an environment variable, LLVM_PROFILE_FILE, which provides the path to the profraw file we want to write.

Later these profraw files needs to be indexed via llvm-profdata and merged into one, for convenience. This operation creates a osquery.profdata file.

Then, this file can immediately be used/analyzed by the llvm-cov tool, which can show coverage information via text/CLI. It can also export an HTML, but it's unstructured, in the sense that's basically a dump of all the regions of code, without a hierarchy of folders.

So to have better output we leverage lcov, which can generate a nicer report. Starting from the .profdata file we need to export the coverage information for each binary in a format that's understood by lcov.

Then, after having created a .lcov file for each binary, we can merge them back into one file using the lcov utility. Finally, we can generate the report via the genthml command.

This is just a rough overview of what's happening, but as the commit states, there's a helper target generare_coverage that does this for you and creates the HTML report in <build folder>/coverage/html-out. It's important to remember that one needs to run the tests first, to generate the .profraw files.

CI job

Finally the PR adds a CI job that will run once a day on the master branch to generate the coverage and will upload an artifact for both Linux and macOS with the HTML report.

The job is not being done inline with the PRs because it doesn't use ccache, so it take ~1h 30m to build and would slow down things. It doesn't use ccache because it doesn't share compilation flags with the normal builds, so it will be another archive in the Github Cache to upload, which takes space that's limited (5GB total across all jobs).

An example of the output can be seen in my CI job: https://github.com/Smjert/osquery/actions/runs/2605832136

Smjert avatar Jul 03 '22 22:07 Smjert

I want to move this forward when I have some free time, but I want to further split this up in multiple PRs where possible, since it's big.

Smjert avatar Dec 09 '22 12:12 Smjert