feat(browser): introduce `toMatchScreenshot` for Visual Regression Testing

[macarie]

Description

This PR introduces initial support for Visual Regression Testing for Vitest via a new toMatchScreenshot assertion.

Related issue: #6265

In this initial iteration:

• The feature supports PNG screenshots and uses pixelmatch as the comparator.
• The architecture is extensible: new comparators or codecs can be added easily, as long as they implement the expected interface.
• Comparators and codecs can be asynchronous.
• Screenshot comparison is executed in Node as a browser command, which allows for future adoption of native codecs or comparators.

The logic to get a stable screenshot follows Playwright's approach (with some differences):

1. Uses as baseline an optional reference screenshot or captures a new screenshot.
2. Takes a screenshot and compares it to the baseline.
3. If they match, the page is considered stable and the function returns.
4. If not, it continues with the latest screenshot as the baseline.
5. Repeats until stability is reached or a timeout is hit.

The command has 6 possible outcomes:

Outcome	Description	Result
#01	couldn't get a stable screenshot within timeout	fail
#02	stable screenshot taken, but no reference exists and update is not allowed	fail
#03	stable screenshot taken, no reference exists, but updates are allowed	fail
#04	stable screenshot taken on first try and matches reference	pass
#05	stable screenshot taken matches reference after retries	pass
#06	stable screenshot taken doesn't match reference (fallback case)	fail

The client matcher expects an Element or Locator along with:

• An optional name for the screenshot
• An optional options object that includes:
  • Screenshot options, excluding conflicting ones
  • Comparator name and options
  • Timeout value

To-do

• [x] Fix screenshot paths and naming
• [x] Add global config for matcher, including resolveScreenshotPath and resolveDiffPath functions
• [x] Handle default config in one place
• [x] Use annotation API (#7953) to show screenshot paths
• [x] Support thresholds in pixelmatch comparator
• [x] Add documentation
• [ ] Write test cases

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

• [x] It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
• [ ] Ideally, include a test that fails without this PR but passes with it.
• [ ] Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.

Tests

• [ ] Run the tests with pnpm test:ci.

Documentation

• [ ] If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

• [ ] Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

[netlify[bot]]

✅ Deploy Preview for vitest-dev ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	50047212f2a2b7242809f155b869a2f7aaef0457
🔍 Latest deploy log	https://app.netlify.com/projects/vitest-dev/deploys/687f88baa7d0470008c5b5f5
😎 Deploy Preview	https://deploy-preview-8041--vitest-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[macarie]

I moved the types around to what starts to hopefully make some sense. I created a shared folder for the types used in both browser and Node environments.

I've also rebased and resolved the conflicts from main.

[macarie]

I changed the testing approach: instead of relying on pre-generated screenshots, the tests now create them dynamically during execution.

This makes the tests more stable and avoids failures across different OSes, browser versions, or rendering environments. All artifacts are cleaned up automatically after the tests run.

~~One thing still missing is a test for watch mode. I ran into some issues with it and will take another shot at getting it to work tomorrow.~~ ➡️ Done.

[pkg-pr-new[bot]]

@vitest/browser

npm i https://pkg.pr.new/@vitest/browser@8041

@vitest/coverage-istanbul

npm i https://pkg.pr.new/@vitest/coverage-istanbul@8041

@vitest/coverage-v8

npm i https://pkg.pr.new/@vitest/coverage-v8@8041

@vitest/expect

npm i https://pkg.pr.new/@vitest/expect@8041

@vitest/mocker

npm i https://pkg.pr.new/@vitest/mocker@8041

@vitest/pretty-format

npm i https://pkg.pr.new/@vitest/pretty-format@8041

@vitest/runner

npm i https://pkg.pr.new/@vitest/runner@8041

@vitest/snapshot

npm i https://pkg.pr.new/@vitest/snapshot@8041

@vitest/spy

npm i https://pkg.pr.new/@vitest/spy@8041

@vitest/ui

npm i https://pkg.pr.new/@vitest/ui@8041

@vitest/utils

npm i https://pkg.pr.new/@vitest/utils@8041

vite-node

npm i https://pkg.pr.new/vite-node@8041

vitest

npm i https://pkg.pr.new/vitest@8041

@vitest/web-worker

npm i https://pkg.pr.new/@vitest/web-worker@8041

@vitest/ws-client

npm i https://pkg.pr.new/@vitest/ws-client@8041

commit: 5004721

[macarie]

I will have to debug the failing Webdriverio test on my Windows PC, looks like arg is not getting sanitized for some reason. I also don't understand why it fails with ENOENT 🙃

[macarie]

Just tested this PR by replacing Playwright's component testing with Vitest in one of our visual regression test suites at work. Pretty impressed with how it went!

The speed difference is huge:

• Vitest: 4m46s

• Playwright: 34m41s

Not sure why Playwright is taking so long here. This was with just 1 browser instance, we usually run 20 in parallel. Our runs take less than 2 minutes to complete, but the consumed browser-time amounts to ~35m each time, so this result aligns well with what we've always observed. Worth noting: Playwright runs a few more tests, those are non-visual tests and only add 1m40s.

About the failing tests in Vitest:

• half failed because this PR handles some special characters in filenames differently
• the other half looks like an actual bug, I'm gonna dig into this one a bit more. Seems like the tests are running too fast and not waiting for the screenshots to complete. It's happening with our button tests where we capture default and hover states, but the first screenshot is always showing the hover state tho.

Overall tho, really minimal changes needed to get this working!

[macarie]

Let me know if you don't want the merge commit in the PR's history, I will rebase and remove it once we're done with the review.

Force-pushing after a rebase moves comments up in the "load more" hidden section every time I update from main 😄