Support visual regression testing in the browser

[sheremet-va]

Clear and concise description of the problem

There is no built-in way to compare images in the browser mode.

Suggested solution

Implement a snapshot-style assertion to compare previously stored images. I am open to API ideas.

This issue is opened to start a discussion around the topic.

Alternative

Related: https://github.com/vitest-dev/vitest/discussions/690

Additional context

Consider using https://github.com/mapbox/pixelmatch as it is small and fast

Validations

• [X] Follow our Code of Conduct
• [X] Read the Contributing Guidelines.
• [X] Read the docs.
• [X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

[AriPerkkio]

I'm not sure if this should be part of the core. Or at least it should be marked as very-very experimental all the time.

Comparing images is very hard. There are companies building products that especially focus on this, and using those can be expensive. Open source tools like jest-image-snapshot and cypress-image-snapshot (that use pixelmatch) are very flaky, especially when comparing images that are generated on different operating systems. For example pixel-by-pixel comparison fails in as simple cases as rendering fonts.

If we can find a way to implement this confidently then I'm all in. 👍

@JessicaSachs any thoughts? Do you have experience on this area?

[herrlegno]

I tried to implement using jest-image-snapshot but no luck so far, it throws an error:

Here is the repo: https://github.com/herrlegno/vitest-browser-snapshot

[m-lyon]

@herrlegno if you use node polyfills, e.g. via the vite-plugin-node-polyfills plugin, then you should bypass that error. Though, the current outstanding issue with using jest-image-snapshot here is the use of __dirname as can be seen here, which gives the ReferenceError: __dirname is not defined error.

[tsirlucas]

I think it would be a good idea to also have support for https://github.com/chromaui/chromatic-e2e

[wallynm]

ror: __dirname is not defined error.

Trying to configure vitest with snapshot but seens to not be a option right now

[pawelkrystkiewicz]

ror: __dirname is not defined error.

Trying to configure vitest with snapshot but seens to not be a option right now

yup I got the same error. It seems like there aren't any ways to do visual regression tests using vitest@browser + playwright. Any solutions?

[ursulean]

This post gives a code snippet that works as an expect.extend, essentially extracting the core functionality of jest-image-snapshot to work in the @vitest/browser environment. Having also had to implement something like this myself in the past, I support this use-case being added to @vitest/browser, to avoid the code duplication involved when everyone who wants visual testing support needs to implement a custom matcher extension. See my recent suggestion for a more detailed line of reasoning.

@AriPerkkio In my experience, browser rendering differences across different OSes have been the bane of my existence, and are the only unavoidable source of flakiness with jest-image-snapshot/pixelmatch-based image comparison, at least with Playwright which does a pretty good job of pinning browser versions to versions of itself. I've personally gotten around this by using macOS machines in CI as visual test runners for screenshots written on macOS dev machines. We wouldn't be able to eliminate that source of flakiness, but I wonder if we couldn't store the OS the screenshot was written on in the PNG metadata, and print a helpful error message when encountering a comparison across different OS-es.

I think it would be useful as an experimental feature in @vitest/browser, with all the caveats documented. @vitest/browser is experimental itself currently, so it fits pretty well IMO.

I would like to raise a PR with a proof-of-concept for this, but would first like to make sure that the feature is not completely off the table for Vitest. Could someone advise?

[sheremet-va]

I would like to raise a PR with a proof-of-concept for this, but would first like to make sure that the feature is not completely off the table for Vitest. Could someone advise?

The feature is not off the table. As a possible implementation, we were considering investigating playwright's toMatchSnapshot. For now, no one on the team is working on this feature.

[ursulean]

Cool, will find some time to sink into it in the next 1-2 weeks 🙂

[macarie]

@ursulean, have you had a chance to make progress on this? If not, I’d like to pick it up and give it a shot.

[ursulean]

Yes, I've made some progress. I can publish the branch for feedback tomorrow.

[dnotes]

Ah shoot wish I'd seen this...I just did this last night and this morning! I don't want to steal your feature but if you'd prefer not to spend more time I'm happy to PR. If you want to check it out my branch is https://github.com/dnotes/vitest/tree/add-browser-toMatchImage.

[dnotes]

Sorry, I'd just done this for quickpickle/browser but it wasn't working, and I thought better to contribute...well anyhow.

[ursulean]

Hi @dnotes I do not see any changes on your branch compared to main I raised a draft PR with my approach for some initial feedback, but happy to collaborate as I am quite new to this project 😅

[dnotes]

Oops forgot to commit before pushing the branch. Happy to collaborate as well, I'm looking at your branch.

[dnotes]

Well, it looks like it may be a good spot for some collaboration. Your approach of creating a new node command and using the node internals may solve a problem that I was having with the saved images not being visible (maybe because I used commands.saveFile) and gets rid of one extra library. Meanwhile, I've incorporated options for comparing by pixels and percentage, as well as all the pixelmatch options, and saving .diff and .actual images like Playwright does. I could merge the two, but I must admit I don't know how to collaborate on pull requests like this...would I make a PR against the branch in your fork?

[macarie]

Hey @ursulean, I spoke with @sheremet-va about this feature and shared some ideas on how it could evolve. We seem to be on the same page, and I'd love to pick up the work from here, with a focus on making it more reliable, stable, and extensible, and continuing to support it in the long term.

If it's okay with you, I'd like to incorporate parts of your PR where they fit and credit you as co-author for your contribution.

Here's a rough idea of where I see this heading:

• Support for PNG and WebP formats out of the box (and potentially more in the future, as browser support grows; for instance, Playwright doesn't support WebP comparison).
• A set of built-in comparators that cover common use cases, with the flexibility for users to define custom comparators when needed.
• Improved documentation and a dedicated guide to make visual regression testing with Vitest easy to set up, understand, and use effectively.

[dnotes]

Some intricacies I've discovered in testing this, mostly in my own branch instead of on this one:

• screenshots for elements as simple as even a button are different sizes in different browsers, making image comparisons fail completely cross-browser
• it looks like they may also be different sizes based on whether the vitest UI is active, or maybe if it's in headless mode
• for pixelmatch, percentage difference is probably more important than number of pixels, but it's best to provide options for both at the same time, for images of components vs. full pages respectively
• standard practice seems to be to fail the test when the baseline image is first created

@macarie I'm not sure of the utility of webp support, since these files are only for testing and not for public consumption in most cases, and as far as I can tell almost all visual regression testing uses PNG by default since it's a ubiquitous lossless format and any lossy format (like webp can be I think?) makes it possible that two renderings of the same input will come out different.

One thing you might want to consider is supporting the other major image comparison algorithm, SSIM I think, or maybe that's the one Pixelmatch uses and there's another. There was one testing library that has a second option, I forget which one. However, to be honest it's more than I would expect from Vitest at the beginning; Playwright only uses Pixelmatch, and really it's good enough for most use cases. (Note, I just found this issue looking for playwright ssim).

[ursulean]

Hi @macarie Awesome to see more support in favor of this feature!

Generally I'm on board with your direction, but I could see the features you outlined developing as iterative improvements over time, whereas the aim for my PR was to arrive at a minimum viable product that is very stable and robust. I see e.g. webp support and SSIM comparison as nice-to-haves down the line, but not crucial for the initial implementation, as they do not constitute the majority use-case of visual testing today. Therefore I would prioritize the following for an initial PR:

• Determinism - there are several sources of browser rendering differences which I have identified (Host OS, different CPU architectures across same OS, even the current screen resolution of the host machine when using Chromium (yes, even in headless mode)). This can be a bit of a rabbit-hole, but these rendering differences will be very frustrating for end users of Vitest if we don't make an effort to detect when these causal factors differ between the baseline and new screenshot. This is a very common problem to encounter, e.g. when writing screenshots locally that are then checked against in CI.
• Feature parity with jest-image-snapshot. Simple stuff like percentage thresholds, allowing size mismatch, etc.
• Extensibility first - support custom image comparators before providing more built-in ones.
• Documentation

@sheremet-va do you have a preference for smaller iterative PRs, vs. collecting all the changes in a bigger PR as suggested by @macarie ? I'm happy to contribute in whatever capacity fits best with the existing practices of this repo 🙂

[macarie]

• screenshots for elements as simple as even a button are different sizes in different browsers, making image comparisons fail completely cross-browser
• it looks like they may also be different sizes based on whether the Vitest UI is active, or maybe if it's in headless mode

Yes, this is expected and one of the challenges with visual regression testing. These kinds of inconsistencies are exactly what make these tests flaky. This will be covered by documentation. Browsers aren’t as deterministic as we’d like: output can vary based on CPU, OS, screen resolution, browser version, and so on. And once fonts are involved... it's even worse 😄

I'm not sure of the utility of webp support

WebP supports lossless mode. Right now, Chrome appears to use lossy mode for screenshots, but that may change. The main reason to support WebP (or other formats) isn’t public consumption, but rather size. When you have thousands of screenshots, size, reliability, stability, and performance start to matter.

One thing you might want to consider is supporting the other major image comparison algorithm

Yep, this is what I meant by "a set of built-in comparators that cover common use cases". Currently Playwright supports two comparators, afaik, and each has its trade-offs. No single approach works perfectly for all use cases, so giving users options or a way to plug in their own is important.

@ursulean, sorry if I wasn't clear in my previous message. The plan is still to start with something minimal and iterate. Baseline will be PNG + pixelmatch, with an extensible architecture so we can add new features over time without major refactors. I’ll open a PR either today or tomorrow with the core functionality in place.

I also plan to maintain and continue improving this feature long-term, as we actively use visual regression testing in production and want to make the most of it in Vitest. Feedback, issues, or ideas are more than welcome.

[dnotes]

This all sounds great. One other request I'd have is to support all options of pixelmatch, as well as the ones from Locator.screenshot() that make sense, in a pass-thru manner, so you could do something like the following and have it work. I know this isn't the approach of the PR currently but it seems more intuitive than having a separate set of options.

await expect(locator).toMatchScreenshot({

  // options of Locator.screenshot()
  path: 'visual/example.png',

  // pixelmatch options
  threshold: .05,
  alpha: 0.4,

  // comparison options (however you do this)
  maxDiffPercentage: 20,
  maxDiffPixels: 200,

})

[TheYoxy]

Maybe an idea would be to match with what is done with the current providers ? As I found:

• Playwright
• Webdriver.io

For webdriver, I'm not used to it, so another method could exists that I didn't find when quickly looking to the docs

---

I had a futher look for the playwright implementation and I found this snippet which seems to be interesting for an implementation with the playwright provider.

From this snippet, I was able to run this snippet locally to have screenshot working locally.

import { readFile, writeFile, mkdir } from 'node:fs/promises';
import path from 'node:path';
import { existsSync } from 'node:fs';

const toHaveScreenshot: BrowserCommand<[]> = async ctx => {
  const provider = ctx.provider;
  if (isPlaywright(provider)) {
    ctx.project.vitest.snapshot.extension = '.png';

    const result = await ctx.page._expectScreenshot({ timeout: 10000 });

    const filename = ctx.project.vitest.snapshot.resolvePath(ctx.testPath);
    if (!existsSync(filename) && ctx.project.vitest.snapshot.options.updateSnapshot === 'new') {
      await mkdir(path.dirname(filename), { recursive: true });
      await writeFile(filename, result.actual);
    }

    const expected = await readFile(filename);

    const comparison = await ctx.page._expectScreenshot({
      timeout: 10000,
      expected,
    });
    if (comparison.errorMessage) {
      const getFileName = (suffix: string) => {
        const f = ctx.project.vitest.snapshot.resolvePath(ctx.testPath);
        const dir = path.join(
          path.dirname(f),
          `${path.basename(f, path.extname(f))}${suffix}${ctx.project.vitest.snapshot.extension}`
        );
        return dir;
      };

      writeFile(getFileName('-actual'), comparison.actual);
      writeFile(getFileName('-diff'), comparison.diff);
    }
    return comparison;
  }
};

the implementation may not be safe since it is just a POC to integrate screenshot in a vitest-browser project

[macarie]

The implementation proposed in #8041 generally follows Playwright's approach, with some changes where it makes sense. It supports all screenshot options from the provider and all pixelmatch options.

A few things still need to be done (annotation API, global config support, and docs), but I think we're on track for a stable, extensible feature.

[dnotes]

Since there's some activity here: I was thinking about how to handle images of different sizes, which is something that's been mentioned a few times. There's a good discussion of some options at https://github.com/mapbox/pixelmatch/issues/25, and based on that I've created a tiny crop/pad utility with a playground for some visual testing scenarios at https://image-crop-or-pad.pages.dev. I'm eventually planning to include some support for this in quickpickle (my Vitest plugin for behavioral testing). To be clear I think it's too early to support this for Vitest, since I don't think there's a standard way this is done, but it would be nice in the future. It's really not ideal that most visual tests fail opaquely when the images are 1px different length or width.

[BotellaA]

So pleased to see this feature moving forward! Are there any rc version available yet to be tested? If not, how to try it?

[unional]

Hi, we have vitest-addon-vis that support this. Feel free to give it a try.

[tv42]

For the peanut gallery like me, this works in 4.0.0-beta.5:

test("click to check box", async () => {
  const screen = render(MyComponent);
  const button = screen.getByRole("checkbox");
  await button.click();
  await expect.element(button).toBeChecked();
  await expect(screen.container).toMatchScreenshot();
});

https://main.vitest.dev/guide/browser/visual-regression-testing

https://main.vitest.dev/guide/browser/assertion-api#tomatchscreenshot