playwright [Feature] a toMatchScreenshot() for comparing two different screenshots

We currently have expect(screenshot).toMatchSnapshot(name[, options]) which is nice when comparing against at snapshot but it doesn't work when comparing a screenshot against a reference website which is useful for doing visual regression to see what has changed. I propose that we add a toMatchScreenshot(screenshot[, options]) that could be used for something like this:


page.goto("production.example.com")
const ref = await page.screenshot({
  animations: "disabled",
  fullPage: true,
  scale: "css"
});

page.goto("localhost:8080")
const test = await page.screenshot({
  animations: "disabled",
  fullPage: true,
  scale: "css"
});

expect(test, "should  match reference").toMatchScreenshot(ref, {
  maxDiffPixelRatio: 0,
  maxDiffPixels: 0,
  threshold: 0
});

Nov 19 '22 22:11 NVolcz

@NVolcz I think we can probably do something like expect(image1).toMatchImage(image2) for this usecase. See also #18897.

In your particular case, I can recommend to produce screenshots once in production environment, commit them to your version control, and then compare localhost screenshots to these expectations. This way, you'll have a stable set of screenshots to test against. I hope this helps.

Nov 21 '22 21:11 dgozman

expect(image1).toMatchImage(image2) would be awesome! I implemented something like what you are suggesting but handling snapshots is troublesome and doesn't seem worth it in my case.

Nov 22 '22 06:11 NVolcz

@dgozman

Would you consider implementing this in a way that would support the following usecase?

test('MyButton has no side effects', async ({ page }) => {
	await page.goto('/');

	const before = await page.getByTestId('my-element').screenshot();

	await page.getByRole('button', { name: 'MyButton' }).click();

	const after = await page.getByTestId('my-element').screenshot();

        // ensure my-element still looks the same
	await expect(before).toMatchImage(after);
});

Specifically:

Compare a screenshot taken at the beginning of the test with one taken at the end.
Without saving them or having to handle paths, etc.
While benefiting from the 'visual image diff widget' in the test report which makes it easy to find out why a particular test failed. Like the videos, the images can be retained-on-failure, and discarded on success.