[Bug]: Ambiguous retry mechanism in playwright 1.42.1

[Smrtnyk]

Version

1.42.1

Steps to reproduce

1. Clone my repo https://github.com/Smrtnyk/PW-repros
2. npm install
3. npm run test

Expected behavior

Test retry mechanism should be more clear

Actual behavior

2 Issues here.

1. In file retries-successful-test.spec the test that succeeded is also retried, not just the failed one. I wasn't expecting to see successful test being retried, just the failing one.

2. In file expecting-retries.spec starting with playwright 1.42 the failing test is not retrying anymore, due to exception being thrown in fixture after run statement. In 1.41 test would still retry. Now it is not clear if this new behavior is expected, but if it is, it should be at least clear why test is not retrying anymore. We have quite a sophisticated test cleanup scenarios, and it was hard to figure out why some of our tests were not being retried anymore due to exception not being shown anywhere.

Additional context

No response

Environment

System:
    OS: Windows 11 10.0.22631
    CPU: (24) x64 12th Gen Intel(R) Core(TM) i9-12950HX
    Memory: 43.18 GB / 63.69 GB
  Binaries:
    Node: 20.10.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.19 - ~\AppData\Roaming\npm\yarn.CMD
    npm: 10.2.3 - C:\Program Files\nodejs\npm.CMD
    pnpm: 8.12.0 - C:\Program Files\nodejs\pnpm.CMD
  IDEs:
    VSCode: 1.86.0 - C:\workspaces\devtools\Microsoft VS Code\bin\code.CMD
  Languages:
    Bash: 5.2.21 - C:\Program Files\Git\usr\bin\bash.EXE
  npmPackages:
    @playwright/test: 1.42.1 => 1.42.1

[mxschmitt]

1. Seems expected because you are using describe.serial. If the second test fails, we will retry the whole group. If you don't want that, remove the "serial".
2. Seems already fixed in Canary. It will now retry again, but not show the "foo" exception from fixtures.ts where I'm pretty sure that we can make an improvement there as well. Assigning it to the next release. It also didn't show it pre 1.42.