coverage icon indicating copy to clipboard operation
coverage copied to clipboard

Coverage not reported correctly for too many line hits

Open Mr-Pepe opened this issue 1 year ago • 1 comments

I was told to report this issue here.

Steps to reproduce

  • Run dart create dart_test
  • Add the following to lib/dart_test.dart
    String doSomething(bool flag) {
      if (flag == true) {
        return 'true';
      }
    
      if (flag == false) {
        return 'false';
      }
    
      throw ArgumentError('Invalid flag');
    }
    
  • Add the following to test/dart_test_test.dart (sorry for the name...)
    import 'package:dart_test/dart_test.dart';
    import 'package:test/test.dart';
    
    void main() {
      test('Test', () {
        const nTests = 100000;
        for (var i = 0; i < nTests; i++) {
          expect(doSomething(true), equals('true'));
        }
        expect(doSomething(false), equals('false'));
      });
    }
    
  • Execute tests with dart run coverage:test_with_coverage && genhtml coverage/lcov.info -o coverage/html

Expected behavior

The line if (flag == false) should always be counted as covered, regardless of nTests.

Observed behavior

For nTests = 10, the line is counted as covered.

❯ dart run coverage:test_with_coverage && genhtml coverage/lcov.info -o coverage/html
The Dart VM service is listening on http://127.0.0.1:8181/
The Dart DevTools debugger and profiler is available at: http://127.0.0.1:8181/devtools?uri=ws://127.0.0.1:8181/ws
00:00 +1: All tests passed!                                                                                                                                                                            
Found 1 entries.
Found common filename prefix "/home/felipe/tmp/dart_test"
Generating output.
Processing file lib/dart_test.dart
  lines=4 hit=3
Overall coverage rate:
  lines......: 75.0% (3 of 4 lines)
  functions......: no data found

For nTests = 100000, the line is counted as not covered.

❯ dart run coverage:test_with_coverage && genhtml coverage/lcov.info -o coverage/html
The Dart VM service is listening on http://127.0.0.1:8181/
The Dart DevTools debugger and profiler is available at: http://127.0.0.1:8181/devtools?uri=ws://127.0.0.1:8181/ws
00:00 +1: All tests passed!                                                                                                                                                                            
Found 1 entries.
Found common filename prefix "/home/felipe/tmp/dart_test"
Generating output.
Processing file lib/dart_test.dart
  lines=4 hit=2
Overall coverage rate:
  lines......: 50.0% (2 of 4 lines)
  functions......: no data found

Placing the second expect expect(doSomething(false), equals('false')); before the loop leads to the line always being counted as covered.

System

[✓] Flutter (Channel stable, 3.19.2, on Arch Linux 6.7.6-arch1-2, locale en_US.UTF-8)
    • Flutter version 3.19.2 on channel stable at /home/felipe/snap/flutter/common/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 7482962148 (5 days ago), 2024-02-27 16:51:22 -0500
    • Engine revision 04817c99c9
    • Dart version 3.3.0
    • DevTools version 2.31.1

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /home/felipe/Android/Sdk/
    • Platform android-34, build-tools 34.0.0
    • Java binary at: /home/felipe/Applications/android-studio/jbr/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)
    • All Android licenses accepted.

[✓] Chrome - develop for the web
    • CHROME_EXECUTABLE = chromium

[✓] Linux toolchain - develop for Linux desktop
    • clang version 10.0.0-4ubuntu1
    • cmake version 3.16.3
    • ninja version 1.10.0
    • pkg-config version 0.29.1

[✓] Android Studio (version 2022.2)
    • Android Studio at /home/felipe/Applications/android-studio
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)

[✓] Connected device (2 available)
    • Linux (desktop) • linux  • linux-x64      • Arch Linux 6.7.6-arch1-2
    • Chrome (web)    • chrome • web-javascript • Chromium 122.0.6261.94 Arch Linux

[✓] Network resources
    • All expected network resources are available.

• No issues found!

Mr-Pepe avatar Mar 04 '24 08:03 Mr-Pepe