Coverage results generated before tests finish for a gem under development.

[ThomasOwens]

I was able to put together a very simple repository that demonstrates the issue, including committing the coverage files.

https://github.com/ThomasOwens/simplecov-bug-demo

I'm building a gem, which was initialized using the bundler gem tooling. After initializing the gem, I added a dependency on simplecov for development and configured it per instructions. However, when I run commands (like bundle exec rake, which is configured to run tests and rubocop), I get output, as shown below, that indicates the coverage report is being generated before test execution.

At first, I thought it had to do with the initialization of application code before simplecov, but it looks like the only thing being initialized is the standard version.rb file. It appears to be a timing of generation versus test based on the console output.

In the reproduction example repository, I would expect no coverage of lib/simplecov-bug-demo/version.rb, because this file is required in the gemspec. However, I would expect lines 7-8 of lib/simplecov-bug-demo/commands/command.rb to be covered. However, neither line is covered. Only the module, class, and method declaration lines are covered, which is what I would expect for simply loading the file after executing simplecov.

Without a .simplecov file, the console output of bundle exec rake looks like:

Coverage report generated for Minitest to /workspaces/simplecov-bug-demo/coverage.
Line Coverage: 75.0% (9 / 12)
Run options: --seed 44598

# Running:

..

Finished in 0.001718s, 1164.1532 runs/s, 1164.1532 assertions/s.

2 runs, 2 assertions, 0 failures, 0 errors, 0 skips

With a .simplecov file, the console output of bundle exec rake is:

root@72365261d8df:/workspaces/simplecov-bug-demo# bundle exec rake
Warning: Error occurred while trying to load /workspaces/simplecov-bug-demo/.simplecov. Error message: uninitialized constant Simplecov
Run options: --seed 20890

# Running:

..

Finished in 0.001560s, 1281.7818 runs/s, 1281.7818 assertions/s.

2 runs, 2 assertions, 0 failures, 0 errors, 0 skips

I'm using Ruby 3.3 inside a devcontainer built on top of a Ruby 3.3 docker image. Minitest is at version 5.25.1, Rake at 13.2.1, and simplecov at 0.22.0.

The underlying cause may be the same as #1099. However, that issue does talk specifically about Rails, which is not involved in my project.

[ThomasOwens]

I asked about this in a Stack Overflow question and got an answer from mitch_. The solution was, as described, to add SimpleCov.external_at_exit = true before the SimpleCov.start in the test_helper.rb file. I also did have to convert the start to a block and explicitly filter the vendor and test directories.

If this doesn't represent a bug in SimpleCov, it would be worth updating the readme to describe how to integrate SimpleCov with minitest. If there's a more appropriate solution, it would be good to get that out into the world as well.

[liveh2o]

I asked about this in a Stack Overflow question and got an answer from mitch_. The solution was, as described, to add SimpleCov.external_at_exit = true before the SimpleCov.start in the test_helper.rb file. I also did have to convert the start to a block and explicitly filter the vendor and test directories.

If this doesn't represent a bug in SimpleCov, it would be worth updating the readme to describe how to integrate SimpleCov with minitest. If there's a more appropriate solution, it would be good to get that out into the world as well.

This worked for me as well. (Ruby 2.7, SimpleCov 0.22.0)

[pboling]

This isn't a bug, exactly, but it is confusing. If Minitest failed to load the SimpleCov plugin, or if SimpleCov wasn't defined when it did, the fix didn't get applied, which is why you had to manually apply it.

I am having the same issue, and I'm not sure why the "normal" scenario isn't working, but the workaround for me was similar. The ones above lacked an important part, which is clear if you look at how it is intended to work (ref):

# How minitest plugins. See https://github.com/simplecov-ruby/simplecov/pull/756 for why we need this.
# https://github.com/seattlerb/minitest#writing-extensions
module Minitest
  def self.plugin_simplecov_init(_options)
    if defined?(SimpleCov)
      SimpleCov.external_at_exit = true

      Minitest.after_run do
        SimpleCov.at_exit_behavior
      end
    end
  end
end

I have my SimpleCov.start inside my .simplecov file, so it now has:

require "kettle/soup/cover/config" # This lives in the kettle-soup-cover gem!
require "minitest/simplecov_plugin" # This lives in the simplecov gem!

Minitest.plugin_simplecov_init({})

SimpleCov.start

It's working great!