[Bug]: Mutation testing appears to not be correctly running the test suite against mutants

[samlev]

What Happened

I am attempting to run mutation tests against my application, and it is frequently reporting untested methods which are definitely explicitly tested. If I attempt to make the same mutations to the original classes, my test suites will correctly catch the changes and fail, but pest simply reports the classes as untested:

   UNTESTED  src/Data/Frame.php  > Line 21: EmptyStringToNotEmpty - ID: 202027099d87aea4

    */
   readonly class Frame extends BaseData
   {
  -    final public function __construct(public string $file = '', public string $class = '', public string $function = '', public int $line = 0, public object|null $object = null)
  +    final public function __construct(public string $file = 'PEST Mutator was here!', public string $class = '', public string $function = '', public int $line = 0, public object|null $object = null)
       {
           //
       }

This should be caught with this test which explicitly tests the "defaults" of a new class:

use RecursionGuard\Data\Frame;

covers(Frame::class);

it('makes a new frame that is empty', function () {
    $frame = new Frame();

    expect($frame->file)->toBe('')
        ->and($frame->class)->toBe('')
        ->and($frame->function)->toBe('')
        ->and($frame->line)->toBe(0)
        ->and($frame->object)->toBeNull()
        ->and($frame->empty())->toBeTrue();
});

And when I make the same modification to the Frame class, I (correctly) get a test failure:

readonly class Frame extends BaseData
{
    final public function __construct(
        public string $file = 'PEST Mutator was here!',
        public string $class = '',
        public string $function = '',
        public int $line = 0,
        public object|null $object = null,
    ) {
        //
    }
    
    // ...
}

With test results:

 FAILED  Tests\Unit\Data\FrameTest > it makes a new frame that is empty                                                                                                                                                                    
  Failed asserting that two strings are identical.
  -''
  +'PEST Mutator was here!'
  

  at tests/Unit/Data/FrameTest.php:14
     10▕ it('makes a new frame that is empty', function () {
     11▕     $frame = new Frame();
     12▕ 
     13▕     expect($frame->empty())->toBeTrue()
  ➜  14▕         ->and($frame->file)->toBe('')
     15▕         ->and($frame['file'])->toBe('')
     16▕         ->and($frame->class)->toBe('')
     17▕         ->and($frame['class'])->toBe('')
     18▕         ->and($frame->function)->toBe('')

  1   tests/Unit/Data/FrameTest.php:14

How to Reproduce

I'm not entirely sure how to force a reproduction here. It seems to happen on the same files each time, but not on others. There's no common theme that I can discover - some classes are abstract, some are readonly, some methods are final, but then there's another class right next to each one with the same properties that doesn't have the issue.

Code coverage doesn't seem to be an issue, either:

 ./vendor/bin/pest --coverage --parallel
 
  Tests:    409 passed (2247 assertions)
  Duration: 0.91s
  Parallel: 32 processes

  Data/BaseData ............................................. 100.0%  
  Data/Frame ................................................ 100.0%  
  Data/RecursionContext ..................................... 100.0%  
  Data/Trace ................................................ 100.0%  
  Exception/InvalidContextException ......................... 100.0%  
  Exception/InvalidTraceException ........................... 100.0%  
  Exception/RecursionException .............................. 100.0%  
  Factory ................................................... 100.0%  
  Recursable ................................................ 100.0%  
  Recurser .................................................. 100.0%  
  Support/ArrayWritesForbidden .............................. 100.0%  
  Support/WithRecursable .................................... 100.0%  
  ────────────────────────────────────────────────────────────────── 
                                                      Total: 100.0 % 

Sample Repository

No response

Pest Version

3.5.1

PHP Version

8.3.13

Operation System

Linux

Notes

I noticed this in an earlier version of Pest, and mentioned it to @nunomaduro on twitter. At that time it was only reporting a handful of failures, but the latest version of pest seems to be giving me significantly higher numbers of failures.

[riabiy]

Can confirm and thought it's me.

[riabiy]

tl;dr

User error in my case. Check, if every test you have has ...Test in the end. Because PHPStorm will run it no matter what.

broader version

Some findings.

• untested case
• run via id
• add dd() to the test
• still green the whole pre-run, which should not happen

That means my real test case is not run at all means caching. But --clear-cache was used.

Assumed some internal caching, that is failing due to reasons I don't understand. Tried to remove everything, that is possible in .temp folders. No help.

Then I tried to run just a regular test-suite with sail pest --group=__pest_mutate_only. And, bingo, the test file is not present despite having mutates.

Eventually I found out, that it won't get to the __pest_mutate_only group, if ...Test is missing in the name. Though PHPStorm runs it just fine, leading to the behaviour described above in the ticket.

I'm not saying this is the issue, I'm just giving how I've solved it in my case and it was a user error.