BuildQualityChecks - Code Coverage checks should run per file (not averaged)

[CIPop]

Describe the context

• Extension: BuildQualityChecks
• Environment: Azure DevOps Services (cloud)
• Pipeline type: yaml

Describe the problem and expected behavior

C code coverage analyzer makes an average for the entire project. The expectation is that each file is over the required limits across the project.

E.g. for a good project where current coverage is 90%, adding a few small files with 0% coverage will result in a passing CI gate (reducing the coverage to let's say 80%).

The problem this creates is that new files must now have more coverage than expected.

Example from our https://github.com/Azure/azure-sdk-for-c repo:

Our yaml is:

  - task: mspremier.BuildQualityChecks.QualityChecks-task.BuildQualityChecks@6
    displayName: Check line coverage
    inputs:
      checkCoverage: true
      coverageFailOption: fixed
      coverageType: line
      # 90% minimum line coverage
      coverageThreshold: 90
    condition: eq(variables['AZ_SDK_CODE_COV'], 1)

  - task: mspremier.BuildQualityChecks.QualityChecks-task.BuildQualityChecks@6
    displayName: Check branch coverage
    inputs:
      checkCoverage: true
      coverageFailOption: fixed
      coverageType: branch
      # 70% minimum branch coverage
      coverageThreshold: 70
    condition: eq(variables['AZ_SDK_CODE_COV'], 1)

[ReneSchumacher]

Hi @CIPop,

currently, the requested behavior is not supported by BQC since we're not parsing/understanding the coverage data directly. BQC merely works based on the coverage summary data provided by Azure DevOps (through its API). This summary data does not contain information about individual files.

The API has limited support for more detailed coverage data, but it only seems to work for .NET coverage data (.coverage files) and I have never seen more details than module level (i.e., assembly). Thus, we would probably need to move to our own coverage parsing logic, something I've been trying to avoid so far. 😉

[CIPop]

@ReneSchumacher, is there any way / example on how to implement our own verification yet still integrate with your system? I'm thinking I could write my own parser then, somehow pass a single calculated (min of all files instead of total) percentage result to BQC.

[ReneSchumacher]

Hi again,

unfortunately, there's now way to inject coverage values into BQC. Instead of adding something like this, I'd rather invest some time in adding custom coverage parsers to the task itself. Is there a specification for the coverage file format you're using?

[CIPop]

I don't have details but the tools that are being used are gcov and lcov: https://github.com/Azure/azure-sdk-for-c/blob/bcb2e6f50fec297a261109045f5f5c09dd8ae921/CONTRIBUTING.md#code-coverage-reports

They can be configured to produce various reports (the one I pasted above is HTML).

I'd rather invest some time in adding custom coverage parsers to the task itself. Is there a specification for the coverage file format you're using?

Allowing 3rd parties to write scripts to parse would be great!