How to get coverage from Bazel by running the program instead of writing unit tests?

[realTaki]

Description of the feature request:

Except for the unit test for every single function, it sometimes needs testing in some scenes like the simulation of the user's input as a black box testing. To ensure the testing scene is complex enough to find vulnerabilities, I want to know the coverage of some testing scenes.

I want it like using LLVM flags -fprofile-instr-generate -fcoverage-mapping and running the test program directly, interacting with the program, and then getting coverage. I read from Bazel's documents know the coverage command runs the unit test to get coverage, but can I directly get coverage from a running program?

What underlying problem are you trying to solve with this feature?

simulate the user's input as a black box testing and get coverage of the testing case.

Which operating system are you running Bazel on?

Ubuntu 18.04

What is the output of bazel info release?

release 5.3.2

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

https://stackoverflow.com/questions/75311103/how-to-get-coverage-from-bazel-by-running-the-program-instead-of-writing-unit-te

Any other information, logs, or outputs that you want to share?

No response

[c-mita]

It depends on the rules in question.

With something like cc_test/cc_binary, building with --collect_code_coverage will produce instrumented binaries, but the coverage data will not be gathered into a single location, and no conversion or further handling of the coverage data is produced.

Assuming GCC, you'll end up with "xxx.o.gcno" files spread around your bazel-out/.../bin/_objs directory. Executing the resulting binary will generate gcda files for each of those gcno files (the location depends on the version of GCC you're using and where you execute the binary).

I think LLVM is similar, but I can't remember off the top of my head.

Java rules are another matter; the jar produced by bazel build --collect_code_coverage ... may be instrumented, but the wrapping script needs to be called with specific arguments in a specific environment for coverage report generation to actually work, and I don't think that's a simple matter.

If you're simulating the user input, can that be encapsulated into something that is then handled by another custom test rule which then coordinates the execution of the target binary and uses collect_cc_coverage.sh to handle everything?

[github-actions[bot]]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs. If you think this issue is still relevant and should stay open, please post any comment here and the issue will no longer be marked as stale.

[bcsgh]

BTW: it seems that bazel run --collect_code_coverage doesn't result in the same build as bazel coverage. At a minimum, it doesn't pick up the default computed value for --instrumentation_filter , but I'm also seeing indication that just copying the reported value for --instrumentation_filter from the output of bazel coverage doesn't work as expected (the logs are reporting coverage for other stuff as well).