test state is in "pending" when `beforeAll` failed

[hi-ogawa]

Describe the bug

I'm creating a dedicated issue based on the discussion in https://github.com/vitest-dev/vitest/pull/4799#discussion_r1437586318

Current behavior seems to be that the test state is kept in "pending" when beforeAll failed, but the desired behavior might be to mark those tests as "skipped".

Here is a repro and a screenshot of Vitest UI:

https://stackblitz.com/edit/vitest-dev-vitest-yr2mrp?file=test%2Frepro.test.ts

Show screenshot

Reproduction

https://stackblitz.com/edit/vitest-dev-vitest-yr2mrp?file=test%2Frepro.test.ts

System Info

System:
    OS: Linux 5.0 undefined
    CPU: (8) x64 Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
    Memory: 0 Bytes / 0 Bytes
    Shell: 1.0 - /bin/jsh
  Binaries:
    Node: 18.18.0 - /usr/local/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 9.4.2 - /usr/local/bin/npm
    pnpm: 8.10.5 - /usr/local/bin/pnpm
  npmPackages:
    @vitest/ui: latest => 1.1.0 
    vite: latest => 5.0.10 
    vitest: latest => 1.1.0

Used Package Manager

npm

Validations

• [X] Follow our Code of Conduct
• [X] Read the Contributing Guidelines.
• [X] Read the docs.
• [X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• [X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• [X] The provided reproduction is a minimal reproducible example of the bug.

[sheremet-va]

That is an interesting dilemma. I decided to look into what other runners do (https://github.com/sheremet-va/test-before):

1. @playwright/test fails the first test and marks all subsequent tests as "skipped"
2. jest fails every test that doesn't have "skip" modifier - every test has errors populated (even skipped ones), but the test status stays "pending"
3. mocha doesn't even register any tests, only failures (looking at json output) - it does show "before all" hook for "<name of the first test>" in the terminal
4. tap is similar to mocha - only the first test (named assert) is marked as failed, all others are not even collected
5. node:test marks every test as a failed one if an error is thrown inside before (it has a special "cancelled" state that is shown as a failing test) - output is similar to jest (every test shows an error "test did not finish before its parent and was cancelled" even if it has { skip: true }, but it's counted as skipped

[AlexPaven]

Was there ever a decision on this? I'm guessing no, since I still see tests in pending state when beforeAll fails. I'm thinking of switching to fixtures which seem to have better behavior and feel, but it's not exactly equivalent to beforeAll, since they get reinitialized for every test - am I right in saying that?

[sheremet-va]

since they get reinitialized for every test - am I right in saying that?

Yes

---

We decided to mark tests in the suite as skipped if beforeAll fails