Incorrect code coverage when process isolation is used

[dac514]

Q	A
PHPUnit version	8.5.5
PHP version	7.3.17
Installation Method	Composer

Reported to pcov, answer was "This is an upstream problem: phpunit is failing to set the interpreter for the new process with the configuration from the parent process.":

See: https://github.com/krakjoe/pcov/issues/51

Summary

The problem is, depending on how phpunit.xml is configured, I get two different coverage reports for the same file, one of which is less accurate. See screenshots:

Current behavior

Poor code coverage report about the switch statement.

How to reproduce

1. Download attached code: pcov-bug-report.zip
2. composer install
3. ./vendor/bin/phpunit --configuration tests/phpunit.xml --coverage-clover clover.xml

Zip file contains:

.
├── clover-processIsolation-false.xml
├── clover-processIsolation-true.xml
├── composer.json
├── composer.lock
├── src
│   ├── One
│   │   ├── SupportedCountriesISO2.php
│   │   └── SupportedCountriesISO3.php
│   └── Two
│       └── Example.php
└── tests
    ├── Unit
    │   └── ExampleTest.php
    └── phpunit.xml

Expected behavior

More accurate code coverage reports about the switch statement.

Workarounds

1. If I open tests/phpunit.xml and change processIsolation="false" to processIsolation="true" the coverage is more accurate.

2. Or, if I open tests/phpunit.xml and change: processUncoveredFilesFromWhitelist="true" to processUncoveredFilesFromWhitelist="false" the coverage is more accurate.

Unfortunately, these workarounds improve this specific case, but worsen the overall metrics in my real project so I cannot use them.

[yeroon]

Same problem here with php 7.4.3, phpunit 8.5.24. using pcov

[JoaoDias-223]

Same problem here with php 7.1.33 and PHPUnit 7.5.20 using XDebug

[sebastianbergmann]

Same problem here with php 7.1.33 and PHPUnit 7.5.20 using XDebug

PHPUnit 7.5 is no longer maintained, neither is PHP 7.1.

[mxsxs2]

I think I encountered this same issue with version 10.0.7 of phpunit. When I use --process-isolation I get 0.0% coverage. This used to work in phpunit 9.x

How to reproduce:

1. Create a new Laravel project with composer composer create-project laravel/laravel example-app
2. Run XDEBUG_MODE=coverage ./vendor/phpunit/phpunit/phpunit --coverage-text --process-isolation

Outcome:

Expected:

Both tests were run on the out of the box Laravel project. The only difference is the use of --process-isolation flag.

I tried with two separate PHP setups:

PHP 8.1.2 (cli) (built: Apr 7 2022 17:46:26) (NTS) Copyright (c) The PHP Group Zend Engine v4.1.2, Copyright (c) Zend Technologies with Zend OPcache v8.1.2, Copyright (c), by Zend Technologies with Xdebug v3.1.2, Copyright (c) 2002-2021, by Derick Rethans

PHP 8.2.3 (cli) (built: Feb 14 2023 20:52:32) (NTS) Copyright (c) The PHP Group Zend Engine v4.2.3, Copyright (c) Zend Technologies with Zend OPcache v8.2.3, Copyright (c), by Zend Technologies with Xdebug v3.2.0, Copyright (c) 2002-2022, by Derick Rethans

It would be nice to have process isolation working again as I have over 300 tests and my containers get killed with OOM during tests.

@sebastianbergmann If you'd like I can open this as a new issue, given that this issue is almost a year old and originally reported for different environment.

[sebastianbergmann]

If you'd like I can open this as a new issue, given that this issue is almost a year old and originally reported for different environment.

Yes, please. Please provide a minimal, self-contained, reproducing test case that shows the problem you are reporting. And please compare PHPUnit 10 without process isolation versus PHPUnit 10 with process isolation, not PHPUnit 9 versus PHPUnit 10.

[mxsxs2]

@sebastianbergmann My issue was the same as https://github.com/sebastianbergmann/phpunit/issues/5218 and resolved by https://github.com/sebastianbergmann/phpunit/pull/5221

Thanks for looking into it and get it resolved!