angular
`unit-test` builder with `vitest` is slow for testing and for code coverage
Command
test
Is this a regression?
- [ ] Yes, this behavior used to work in the previous version
The previous version in which this bug was not present was
No response
Description
I migrated a very small existing project using the jasmine-to-vitest schematic and code coverage is fairly slow when using vitest (roughly 80 tests).
With the karma builder:
ng test --no-watch 9,99s user 1,71s system 214% cpu 5,459 total
ng test --no-watch --code-coverage 17,82s user 2,42s system 232% cpu 8,719 total
When migrates to the unit-test builder (and a few tests disabled):
ng test --no-watch 18,49s user 3,33s system 259% cpu 8,396 total
ng test --no-watch --coverage 38,46s user 4,66s system 210% cpu 20,535 total
The tests themselves takes 18s (10s with Karma), and the code coverage takes 20s instead of 8s. This is very noticeable when using vitest.
Minimal Reproduction
This is probably reproducible in a new CLI project, by generating a bunch of components and comparing test times.
Exception or Error
Your Environment
Angular CLI : 21.0.0-rc.1
Angular : 21.0.0-rc.1
Node.js : 22.18.0
Package Manager : npm 10.9.0
Operating System : darwin arm64
┌───────────────────────────┬───────────────────┬───────────────────┐
│ Package │ Installed Version │ Requested Version │
├───────────────────────────┼───────────────────┼───────────────────┤
│ @angular/build │ 21.0.0-rc.1 │ 21.0.0-rc.1 │
│ @angular/cli │ 21.0.0-rc.1 │ 21.0.0-rc.1 │
│ @angular/common │ 21.0.0-rc.1 │ 21.0.0-rc.1 │
│ @angular/compiler │ 21.0.0-rc.1 │ 21.0.0-rc.1 │
│ @angular/compiler-cli │ 21.0.0-rc.1 │ 21.0.0-rc.1 │
│ @angular/core │ 21.0.0-rc.1 │ 21.0.0-rc.1 │
│ @angular/forms │ 21.0.0-rc.1 │ 21.0.0-rc.1 │
│ @angular/localize │ 21.0.0-rc.1 │ 21.0.0-rc.1 │
│ @angular/platform-browser │ 21.0.0-rc.1 │ 21.0.0-rc.1 │
│ @angular/router │ 21.0.0-rc.1 │ 21.0.0-rc.1 │
│ rxjs │ 7.8.2 │ 7.8.2 │
│ typescript │ 5.9.3 │ 5.9.3 │
│ vitest │ 4.0.8 │ 4.0.8 │
Anything else relevant?
Of course this is maybe an upstream problem, but I thought it was worth mentioning it.
Vitest's test isolation improves reliability of tests but does have a cost. Also the file parallelism allows tests suites to be run in parallel but has a startup time cost per file.
Additionally, for code coverage, Vitest uses an AST based remapping system that provides excellent results but which can add to coverage analysis time.
Can you provide the breakdown of build vs test times for each scenario? Also relevant, builds will cache intermediate information by default which can greatly affect the overall process time. If not disabled, it might be useful to do so for profiling purposes.
Also, DOM emulation (jsdom/etc.) does also have a startup time cost. There are tradeoffs there especially when considering size of each test file and amount of test files.
You should also be able to experiment with some of these options via a custom runner config (runnerConfig option) similar to the following (vitest-base.config.ts):
import { defineConfig } from 'vitest/config';
export default defineConfig({
test: {
isolate: false,
fileParallelism: false,
environment: 'happy-dom',
}
});
I forgot to mention that I used ChromiumHeadless for the vitest runner.
Here are the build/test/coverage time. Build took ~3.5s in all cases.
Karma:
ng test --no-watch 10,04s user 1,78s system 216% cpu 5,452 total
ng test --no-watch --code-coverage 13,16s user 2,02s system 256% cpu 5,921 total
Vitest (ChromiumHeadless):
Duration 3.17s (transform 0ms, setup 5.33s, collect 3.53s, tests 679ms, environment 0ms, prepare 40.31s)
ng test --no-watch 18,85s user 3,24s system 293% cpu 7,518 total
Duration 15.69s (transform 0ms, setup 6.35s, collect 4.07s, tests 805ms, environment 0ms, prepare 61.69s)
ng test --no-watch --coverage 38,67s user 4,71s system 214% cpu 20,193 total
Vitest (ChromiumHeadless with isolate: false) throws with NG0400 as @jnizet mentioned below, I can retry when the issue is fixed.
Vitest (ChromiumHeadless with fileParallelism: false):
Duration 3.37s (transform 0ms, setup 886ms, collect 419ms, tests 355ms, environment 0ms, prepare 48.71s)
ng test --no-watch 14,45s user 1,95s system 219% cpu 7,467 total
Duration 19.25s (transform 0ms, setup 894ms, collect 421ms, tests 431ms, environment 0ms, prepare 116.37s)
ng test --no-watch --coverage 32,47s user 2,83s system 150% cpu 23,456 total
I don't see a difference with this option.
Is there something else I can try/provide?
Some additional numbers and information that might be interesting.
I ran a test on an app where I made 1000 copies of the same component (a form) (without removing the other ones). That makes 9139 tests in total.
This app was originally tested with Karma / Jasmine.
I rewrote the tests to use Vitest browser mode (playwright), with the Vitest browser API to interact with locators, retried assertions ans so on. So of course those tests do more than the original tests: they query by role and text, interact with the fields in a more realistic way, etc. I expect them to be a bit slower. But I consider these to be the best practices.
All tests (Karma and Vitest) are run in headless mode (ChromeHeadless for Karma, ChromiumHeadless for Vitest).
When I ran the original Jasmine tests with Karma, it took 17 min. and 18 seconds. But something was odd: the firsts tests go very fast (1000 tests in 23 seconds), but in the end, everything was very slow. The culprit was the default reporter configured by the CLI, which, I guess, clutters the DOM of the testing page with more and more elements.
Running the same tests with the progress reporter takes 3 minutes and 24 seconds (testing time only, reported by Karma).
Running the tests migrated to Vitest browser mode, on my laptop (MacBook Pro, 64 GB RAM, Intel 8 cores), takes 7 min. 18 seconds. That's really really slower, considering that Vitest runs the tests in parallel, and my machine has 8 cores.
This shows when running the tests with ``fileParallelism: false`: it takes 30 minutes and 17 seconds.
Trying to run them with isolate: false crashes the execution after 1000 tests (or so) with the following error:
Error: Failed to import test file /Users/jb/projects/books2/frontend/init-testbed.js
Caused by: Error: NG0400: A platform with a different configuration has been created. Please destroy it first.
Trying to run them with isolate: false and fileParallelism: false crashes the execution after 1 test with the same error:
Error: Failed to import test file /Users/jb/projects/books2/frontend/init-testbed.js
Caused by: Error: NG0400: A platform with a different configuration has been created. Please destroy it first.
Given that tests run fine with Karma without any isolation, I guess they should run without isolation in Vitest too, but that doesn't seem to be the case (see the error above).
Ideally, Vitest could run the tests in parallel (in N different iframes), but without isolation in each of these iframes. Then I guess we could have tests that take about the same time as with Karma, while still benefitting from the advantages and more realistic behavior of Vitest.
Addendum:
I migrated the duplicated test to Vitest, without switching to locators, i.e. keeping basically the same code as before. Running the 9009 tests with Vitest with fileParallelism: false leads to an execution time of 5 min. 27 seconds.
This shows that isolation, in browser mode, is costly (around 60% of overhead).
If Angular could make it possible to run tests without isolation (as in Karma), it would have a really nice effect on the test execution time.
It is indeed way more performant without isolation. Iframe reload costs around 100ms per test file.
@clydin, I don't remember where I mentioned this but the init should be ran once in the virtual init file with something like:
const testBed = getTestBed();
if (testBed.platform) {
return;
}
testBed.initTestEnvironment(...)
Workaround
In the meantime, it works with the analog plugin and we get really outstanding performance.
https://analogjs.org/docs/features/testing/vitest
It's fixed here: https://github.com/angular/angular-cli/pull/31731 In the mean time you could also add a test setup file with the following workaround:
import { getTestBed } from '@angular/core/testing';
import { afterAll } from 'vitest';
afterAll(() => getTestBed().resetTestEnvironment());
Note that this workaround will be a tiny bit slower than the fix as this recreates a new test bed env for each test file
There is an issue where setup files end up using dependencies duplicate modules when using Vitest with the CLI. This means two instances of TestBed. https://github.com/angular/angular-cli/issues/31732
This only happens in browser mode and the workaround is to mark such dependencies as external dependencies:
- create a custom build configuration in
angular.jsonand add the following option"externalDependencies": ["@angular/core", "@angular/core/testing"] - update
unit-testconfiguration to use"buildTarget": "build:test"
"build": {
"builder": "@angular/build:application",
"options": {...},
"configurations": {
...,
"test": {
"optimization": false,
"extractLicenses": false,
"sourceMap": true,
"externalDependencies": ["@angular/core", "@angular/core/testing"]
}
}
},
"test": {
"builder": "@angular/build:unit-test",
"options": {
"buildTarget": ":build:test",
"runner": "vitest",
...
}
}
Even if rc.3 fixed a bunch of issues, it's still not possible to test isolate: false as it throws
Cannot configure the test module when the test module has already been instantiated. Make sure you are not using `inject` before `TestBed.configureTestingModule`.
Oh! This PR should have fixed that: https://github.com/angular/angular-cli/pull/31792 but it wasn't shipped in rc3. @clydin is there a specific reason or the idea is to only support isolated mode for 21.0.0?
Just tested with the rc4, and isolate: false works flawlessly now. Good job guys.
Thank you @clydin and @yjaaidi
Oh! This PR should have fixed that: #31792 but it wasn't shipped in rc3.
Sorry, that was my bad yesterday. I forgot to check for merge-ready RC PRs before running the release scripts. Glad to hear it's working now! 🎉
After updating to rc.4 which solved various issues and uses isolate: true by default, the unit-test builder with vitest gives the following results:
Duration 3.52s (transform 0ms, setup 7.86s, collect 4.94s, tests 747ms, environment 0ms, prepare 42.26s)
ng test --no-watch 21,55s user 3,68s system 331% cpu 7,614 total
Duration 5.81s (transform 0ms, setup 8.11s, collect 5.76s, tests 849ms, environment 0ms, prepare 57.29s)
ng test --no-watch --coverage 32,36s user 4,92s system 376% cpu 9,893 total
For comparison, the karma builder gives:
ng test --no-watch 10,04s user 1,78s system 216% cpu 5,452 total
ng test --no-watch --code-coverage 13,16s user 2,02s system 256% cpu 5,921 total
It's still slowish compared to Karma but now usable for code coverage.
Using isolate: false (default in next rc):
Duration 2.89s (transform 0ms, setup 6.63s, collect 4.06s, tests 726ms, environment 0ms, prepare 2.02s)
ng test --no-watch 17,75s user 2,70s system 303% cpu 6,739 total
Duration 4.33s (transform 0ms, setup 7.23s, collect 5.88s, tests 847ms, environment 0ms, prepare 2.21s)
ng test --no-watch --coverage 25,01s user 3,30s system 333% cpu 8,481 total
makes it a bit faster.
I tried running my test cases with Vitest with ng-mocks, but I had previously run them successfully using Karma. I was using official release 21.0.0
Test Files 302 failed | 1942 passed (2244)
Tests 494 failed | 7757 passed (8251)
Errors 92 errors
Start at 14:46:27
Duration 1640.51s (transform 392.43s, setup 94.65s, collect 3782.75s, tests 980.44s, environment 10146.52s, prepare 908ms)
export default defineConfig({
test: {
isolate: false,
server: {
deps: {
inline: ['ng-mocks']
}
}
}
});
Angular: 21.0.0
CDK/Material: 21.0.0
Node: 22.14.0
ng-mocks: 14.14.0
Operating System: Windows 10
Log was filled with weird error like.
Failed to construct 'PointerEvent': member view is not of type Window
Error: Component '_XXXComponent' has unresolved metadata. Please call `await TestBed.compileComponents()` before running this test.
Vitest seems quite slow—it took around 27 minutes, whereas the same tests ran in about 6-7 minutes with Karma.
Note that I'm not sure if problems are ng-mocks / vitest / jsdom or test related. Although failing test cases contribute to the longer runtime, the overall performance is still slow.
Hi @GipHub123,
- Are you using
@angular/[email protected]or an RC? - the
Error: Component '_XXXComponent' has unresolved metadata.seems like an AOT issue. Did you try turning onaotoption in Karma? It should produce the same error. AFAIK, there is currently no way of using full JIT with the CLI runner. The current workaround is to use the Analog plugin https://analogjs.org/docs/features/testing/vitest - Did you try Browser Mode? It is usually faster than an emulated environment such as JSDOM. You could also try using
happy-dominstead ofjsdomwhich should be a bit faster too. - Also from the result output given the
environmentduration, it seems that isolation is enabled and that can drastically slow down your tests. - The
collectphase is also a bit slow. Do you have lots of code running outside tests (i.e.it&test) or hooks (e.g.beforeEachetc...`) ?
@yjaaidi Thanks for replying 👍
Are you using @angular/[email protected] or an RC?
- Official release 21.0.0
Did you try turning on aot option in Karma? It should produce the same error.
- I just spent a few hours testing how it works with default settings + with isolated = false -flag.
Did you try Browser Mode? It is usually faster than an emulated environment such as JSDOM. You could also try using happy-dom instead of jsdom which should be a bit faster too
-
I didn't try browser mode. I've not used happy-dom or jsdom before but I get a feeling from various discussions that jsdom is more stable and a bit slower than happy-dom. I guessed that some of the errors might be related to jsdom. Others have reported about similar errors
"test": { "builder": "@angular/build:unit-test", "options": { "tsConfig": "tsconfig.spec.json" } },
Also from the result output given the environment duration, it seems that isolation is enabled and that can drastically slow down your tests.
- I had following configuration that is mentioned in Angular's documentation.
"test": {
"builder": "@angular/build:unit-test",
"options": {
"tsConfig": "tsconfig.spec.json"
}
}
export default defineConfig({
test: {
isolate: false,
server: {
deps: {
inline: ['ng-mocks']
}
}
}
});
The collect phase is also a bit slow. Do you have lots of code running outside tests (i.e. it & test) or hooks (e.g. beforeEach etc...`)
- max 1 - 2 beforeEach -blocks per test file depending how I set up the test case. Example below with Jasmine. I converted test cases with
ng g @schematics/angular:refactor-jasmine-vitest+ manually fixed a few remaining errors when I tried vitest.
describe('AlertComponent', () => {
let fixture: ComponentFixture<AlertComponent>;
let component: AlertComponent;
let loader: HarnessLoader;
beforeEach(async () => {
return MockBuilder(AlertComponent)
.keep(MatIcon);
});
beforeEach(async () => {
fixture = TestBed.createComponent(AlertComponent);
fixture.componentRef.setInput('type', 'primary');
fixture.componentRef.setInput('icon', 'campaign');
loader = TestbedHarnessEnvironment.loader(fixture);
component = fixture.componentInstance;
});
describe('template', () => {
it('components', async () => {
await fixture.whenStable();
const element: HTMLElement = fixture.nativeElement.querySelector('h3');
expect(element.className).toContain('text');
const matIconHarness: MatIconHarness = await loader.getHarnessOrNull<MatIconHarness>(MatIconHarness);
expect(matIconHarness).toBeDefined();
expect(await matIconHarness.getName()).toEqual(component?.icon());
});
});
});
And yes I noticed that some of the errors could be fixed with a temporary hack. For example
export default defineConfig({ test: { pool: 'vmThreads', } });
Using these suggests that Vitest still has some rough edges that need polishing before it’s stable enough for production use with Angular.