Support baseline coverage

[iirina]

Right now baseline code coverage collection is broken, see CoverageCommand#L61.

First there needs to be an assessment of the baseline coverage status for Java and C++ (it's not clear whether it's broken for both).

[wchen9911]

@iirina do we have any ETA when this would be fixed?

[iirina]

Unfortunately this issue is not scheduled for this quarter. The soonest is Q4 but I can't make any promises.

[castler]

@meisterT since you unassigned @iirina I am assuming she is no longer working on the coverage topics. Since I had quite some headache which the baseline generation (and I still think that it needs to be improved because it is not working correctly) I would like to ask if external contributions in this area are right now welcome or more certainly could not be reviewed. If the former I would try to dive a little more into this topic for the C++ part.

[meisterT]

@c-mita is the best person to answer your question

[c-mita]

I don't think we're really ready to take PRs in this area. The means to output fleshed out baseline coverage files, including line numbers etc, will differ between languages and Bazel needs some mechanism to facilitate that.

I don't really like the idea of just sticking ad-hoc magic into the C++ rules until we've got some idea of what that would look like, unless there's a compelling reason to do so.

@lberki any thoughts?

[sventiffe]

Adding a team label for better tracking and bumping down to P4 as per the last comment from @c-mita. If we do not even accept PRs in this area, should we leave it open as P4 or close it?

[limdor]

@sventiffe @c-mita I see that the priority has been lowered and rules-cpp tag added. But I am a bit concerned if baseline coverage is working at all in Bazel with any language. If that is the case would mean that bazel coverage is not really useful on it's own. I understand that we do not want to stick ad-hoc magic for C++ rules, but is there a language that baseline coverage is working? Could we have as a much higher priority a description on how bazel coverage and the baseline coverage should work?

This could be added in the documentation together with a table showing for what languages/compilers is supported. Currently it is not clear from the documentation that what you pass to instrumentation_filter is used in any language for the baseline coverage. https://docs.bazel.build/versions/master/command-line-reference.html#flag--instrumentation_filter

Once it is defined how it is supposed to work right now, we could create several example to see the support in the different languages. I already started creating some example and I would like to extend them for additional languages but right now I do not know what are the expectations. Who can help here? The most important question for me would be: What is the command line that should be executed to calculate the code coverage considering baseline coverage? Does it exist nowadays?

[alokpr]

bazel-coverage for cc targets is currently unusable without the baseline. If this is not going to be fixed anytime soon, it should be at least documented for the coverage command. I wasted a lot of time trying to figure out why some of my source files were not getting included in the coverage report even after setting the --instrumentation_filter

[kof02guy]

Is there'a plan for this? Or is there some other work round to get all files included in the coverage report for C++?

[github-actions[bot]]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[fmeum]

@bazelbuild/triage Not stale

@c-mita Do you have an update on the status and/or plans for this feature?

[c-mita]

No update or even concrete plans as of yet.

There are a couple of questions that would need to be answered:

• How do people imagine asking Bazel to generate baseline coverage for a binary? As part of bazel build? So bazel build --generate_baseline_coverage [top_level_binary] would generate a report for everything in the tree rooted at that binary? Or perhaps on an individual target level? So something like bazel build --baseline_coverage //... would be used to generate a whole-project baseline coverage report?
• How will different languages support baseline coverage?

Note that bazel coverage is not really relevant here; that's just a special case of bazel test - the point of baseline coverage is to examine things that aren't tests.

On language support; I'm pretty sure I could write something that produces a baseline coverage report for a C++ target. Similarly, I can produce a tool that takes the deploy.jar from a java_binary (built with --collect_code_coverage) and outputs a baseline coverage report.

But handling the language boundaries, tying everything together into a Bazel command, and working out how external rules would support this, is another issue.

[c-mita]

I suspect we'll just define a flag and leave it for rule authors to determine how best to support their language (either generating baseline reports for each _library rule (I can imagine cc_library working that way) or for wherever they determine their "language boundaries" to be.

Then we just need a way to collate all this information together and a merged report can easily be generated.

[fmeum]

Maybe this could be done by having a well-known output group, similar to how validations use _validation and toggle whether it is requested via a flag? Then ruleset authors could decide whether to generate this output per library or per binary target.

[dieortin]

Any updates?

[c-mita]

Unfortunately not right now. Towards the end of last year I had hoped to focus on following up https://github.com/bazelbuild/bazel/discussions/19869 and baseline coverage could be considered either as part of that or as a sequel, but unfortunately I haven't been able to get around to that yet.