SimulateCoverage discrepancy w.r.t. Coverage#line_stub

[avmnu-sng]

SimpleCov::SimulateCoverage should use the Coverage::line_stub and then apply the nocov token, otherwise, there is a discrepancy in the coverage result.

Consider the following file foo.rb:

class Foo
  module_function

  def foo
    Bar.bar
  rescue BarError => e
    Logger.error(__method__, :failed, e)
  ensure
    Baz.baz
  end
end

The following script is used to compare the results:

require 'coverage'
require 'simplecov'

puts RUBY_DESCRIPTION
puts SimpleCov::VERSION

sc_stub = SimpleCov::SimulateCoverage.call('foo.rb')['lines']
rl_stub = Coverage.line_stub('foo.rb')

puts "SimpleCov stub => #{sc_stub.to_json}"
puts "Ruby lib stub  => #{rl_stub.to_json}"

mismatch = rl_stub.zip(sc_stub).each_with_object([]).with_index do |((rl, sc), lines), i|
  lines << i + 1 if rl != sc
end

puts "Mismatch lines => #{mismatch.to_json}"

The output is:

ruby 2.7.0p0 (2019-12-25 revision 647ee6f091) [x86_64-darwin19]
0.21.2
SimpleCov stub => [0,0,null,0,0,0,0,0,0,0,0]
Ruby lib stub  => [0,0,null,0,0,null,0,null,0,null,null]
Mismatch lines => [6,8,10,11]

The mismatched lines are rescue BarError => e, ensure, and both the end.

[PragTob]

:wave:

Hi there, thanks.

I wasn't aware this existed, seemingly it got introduced in 2.6 so will need some juggling around to make it work for us (+ I fear JRuby doesn't have it) so we can't get rid of all of it but that seems like a great improvement.

Thanks!

[avmnu-sng]

@PragTob https://github.com/jruby/jruby/blob/master/lib/ruby/stdlib/coverage.rb for JRuby. We can probably borrow these implementations to use in 2.5. My point is that we have incorrect data compared to when using Ruby-provided coverage and stubbed coverage, so incorporating the stub coverage logic brings in consistency in both cases.

[PragTob]

@avmnu-sng I know what you mean and I know it's been a problem as it is our own poor implementation basically. I'm all for consistency, hence it's great and thanks for bringing it to my attention.

Not sure we can borrow the implementation for 2.5 as it seems somewhat JRuby specific.

[avmnu-sng]

@PragTob I meant borrow CRuby and JRuby implementations. For example, I did this in rspec-tracer.

[PragTob]

Ah, I don't think we have to borow anything in JRuby as versions there are recent enough with our support so I got confused. I wasn't sure one could borrow CRuby's impl as I assumed it was in C, but that seems helpful. Thanks!

[avmnu-sng]

https://github.com/ruby/ruby/blob/master/ext/coverage/lib/coverage.rb The line_stub is Ruby code that I borrowed without any modifications.