Raise coverage error after collating multiple results files

[arich]

Hi there,

We currently have some custom code similar to what's in https://github.com/simplecov-ruby/simplecov/issues/920 that I'm trying to replace with simply using the public SimpleCov.collate method. We run our specs in separate CI containers and merge the .resultset-n.json files together in a separate task afterwards.

I'm looking for the coverage task to fail and raise an error if the final merged coverage is below ENV['MIN_COVERAGE'] or SimpleCov.minimum_coverage setting, similar to how SimpleCov does after running specs and coverage is below SimpleCov.minimum_coverage. (ENV['MIN_COVERAGE'] is probably our own personal implementation detail but I thought I'd include it anyway.)

This is the relevant code that we're running:

  puts 'Merging coverage...'
  glob = Dir['./coverage_results/.resultset*.json']
  SimpleCov.collate glob
  results = SimpleCov::Result.from_hash(JSON.parse(File.read('./coverage/.resultset.json'))).first
  puts "Covered: #{results.covered_percent} %"
  return if ENV['MIN_COVERAGE'].to_f <= results.covered_percent

  raise "Minimum coverage set to #{ENV['MIN_COVERAGE']} but merged coverage was #{results.covered_percent}."

This code works as is, but I want to use public supported methods wherever possible. We did something pretty similar to https://github.com/simplecov-ruby/simplecov/issues/920 previously and I'm trying to get us fully onto public methods if possible.

So, is this supported already? I thought I'd ask before diving into the code too hard. I noticed that run_exit_tasks! is called in the collate method and expected that would do it. Am I missing something? https://github.com/simplecov-ruby/simplecov/blob/818bc2547842a90c607b4fec834320766a8686de/lib/simplecov.rb#L96 If not I'm happy to close this as is, or I can try to put up a PR.

• Include how you run your tests and which testing framework or frameworks you are running. RSpec and cucumber on some apps, mostly Rails, some gems.
• Include the SimpleCov version you are running in your report. 0.19.1
• Include your ruby -e "puts RUBY_DESCRIPTION". Mostly 2.6-2.7

Thanks!

[PragTob]

Hey @arich,

thanks for the report and sorry for the long wait. Hasn't been an easy time.

You are right, collate should run this exit tasks (imo) or if there's a good reason not to run them then there should be a public API to switch it. Also, thanks for wanting to get on the public API :green_heart:

And it intends to, as you correctly remark it is calling run_exit_tasks!.

    def run_exit_tasks!
      error_exit_status = exit_status_from_exception

      at_exit.call

      exit_and_report_previous_error(error_exit_status) if previous_error?(error_exit_status)
      process_results_and_report_error if ready_to_process_results?
    end

The processing you speak of happens in the last line. I assume that ready_to_process_results? errornously errors out:

    def ready_to_process_results?
      final_result_process? && result?
    end

I can't see how or why at first glance. result? relies on @result which whould be correctly set.

So it's probably down to:

    def final_result_process?
      # checking for ENV["TEST_ENV_NUMBER"] to determine if the tests are being run in parallel
      !defined?(ParallelTests) || !ENV["TEST_ENV_NUMBER"] || ParallelTests.last_process?
    end

But that looks correct unless something is off (There's a TEST_ENV_NUMBER but ParallelTest claims it's not the last process).

Or maybe I went down the wrong hole and the code gets triggered but just misbehaves for whatever reason. Should hopefully not be too hard to write a test reproducing the scenario and going from there :)