[Bug]: Mutation Testing states that $guarded and $hidden properties of model are uncovered even when tests are added

[danielh-official]

What Happened

I run ./vendor/bin/pest --mutate on a test. Everything passes, but then it tells me that the $guarded and $hidden` properties are uncovered.

I have made tests for them.

Example

   UNCOVERED  app/Models/YourModel.php  > Line 49: RemoveArrayItem - ID: 051d3c895907a8cc

   class YourModel extends Model
   {
       use HasFactory;
  -    protected $guarded = ['id', 'created_at', 'updated_at'];
  +    protected $guarded = ['created_at', 'updated_at'];
       protected $hidden = ['id', 'created_at', 'updated_at'];
       protected static function booted(): void
       {

How to Reproduce

In your model test, write the following tests.

test('hidden', function () {
    expect((new YourModel)->getHidden())->toEqual(['id', 'created_at', 'updated_at']);
});

test('guarded', function () {
    expect((new YourModel)->getGuarded())->toEqual(['id', 'created_at', 'updated_at']);
});

Ensure that it covers the model class as specified in Pest's documentation.

Then run ./vendor/bin/pest --mutate.

Sample Repository

No response

Pest Version

3.0.1

PHP Version

8.2

Operation System

macOS

Notes

No response

[nunomaduro]

Can you share the full test? Are you using covers?

[danielh-official]

Can you share the full test? Are you using covers?

It is using covers. I will recreate one in a dummy repository later. If there is no problem with the recreation, I'll mark this closed.

[danielh-official]

Can you share the full test? Are you using covers?

I've created a quick repo for demonstrating the issue: https://github.com/danielh-official/demonstrating-laravel-pest-mutation-testing-issue-with-models

[gehrisandro]

Hi @danielh-official

Thank you very much for the repo.

The problem is, that code coverage by design never reports these lines as covered because the lines are not considered as executable. Therefore, no tests are executed, and the mutations are marked as UNCOVERED.

I know this is not ideal. But for performance reasons, we only want to run the tests, covering the lines mutated.

@nunomaduro In the coverage report, we can see, which lines are considered executable. So we have the following options:

1. Leave it like it is, and document the behavior. Executing with --covered-onlywould remove them entirely. I think this is not ideal.

2. Do not perform mutations on lines not considered as executable. This is less confusing, but we lose a lot of possible mutations.

3. In case of non-executable lines, we could run all tests, or at least all tests marked with covers for the mutated file. I think this is the best option, even when it comes with a little performance drawback. We would need to decide if we consider such mutations as UNCOVERED or UNTESTED if the mutation is not detected. Personally I would prefer UNCOVERED as it then matches the code coverage report. And maybe we could add a hint like Mutation was performed on non-executable code.

What are your opinions?

[danielh-official]

The band-aid I set for this issue is the following:

    /**
     * @pest-mutate-ignore
     */
    protected $guarded = [
        'id',
        'created_at',
        'updated_at',
    ];

    /**
     * @pest-mutate-ignore
     */
    protected $hidden = [
        'id',
        'created_at',
        'updated_at',
    ];

If you want to do the bare minimum, I would suggest including something akin to this in the documentation. It's fairly straightforward.

The main concern is ensuring the usability of the Score that shows at the end of the run. If the Score is always going to be less than 100% because it counts non-executable lines, then the Score becomes useless.

The CLI should show mutations being performed, but maybe it can show something extra for the non-executable lines (which you mention in option 3). And any failed mutation on them would not be counted towards the Score.

If the performance drawback becomes a big deal, then we could add a new CLI option and/or a pest.php configuration.

CLI Example

./vendor/bin/pint --mutate --do-not-count-nonexecutable

Still shows the mutations but does not count non-executable lines in the Score.

Config Example

return [
// ...
    'do-not-count' => [
        'class' => [
             YourModel::class => [
                 '$guarded',
                 '$fillable',
                 'someMethod',
             ],
        ]
    ],
// ...
];

[riabiy]

Upvote for adding --do-not-count-nonexecutable, a lot of non-executable code is important and it's better to test it.

[nunomaduro]

cc @gehrisandro