[Spike] Add interactions for visual regression testing

[siddharthkp]

Problem

Our visual regression testing isn't helpful for testing interactions on components like open menus, focus styles, tooltips on hover, etc.

Snapshot for menu with single selection, we can't see much because the menu isn't open:

Snapshot for IconButton that has a tooltip but only on hover:

Solution

Storybook introduced interaction testing in v6.4 (beta) which is supported in chromatic as well. We can start adopting these stories for better visual regression tests!

[lesliecdubs]

@rezrah is going to spike on this in Primer Brand and will share findings that can be applied to PRC.

[github-actions[bot]]

Hi! This issue has been marked as stale because it has been open with no activity for 180 days. You can comment on the issue or remove the stale label to keep it open. If you do nothing, this issue will be closed in 7 days.

[lesliecdubs]

@rezrah @joshblack do you think the Playwright tests we've added help solve for the problem laid out in this issue, or is there more to do?

[joshblack]

@lesliecdubs I think Playwright gives us the capability to capture these kinds of interaction tests but unfortunately we would still need to add/write these tests. As an idea, I think there are some that we could capture in an automated way and I bet some of the more common ones (like open/close) could also be included in that

[lesliecdubs]

@joshblack makes sense! Do you think it's useful to keep this issue open for tracking purposes, or is it better to fold in expectations for devs to write these tests as they work on new areas of the codebase and backfill tests any time folks touch older components?

If the latter, I wonder if some of the testing docs @broccolinisoup has been working on could potentially include this guidance 🤔

[rezrah]

👋 Coming in a bit late here, sorry 😅 ...

Just wanted to mention that I have visual regression testing on Storybook interactions working in Primer Brand. Both done through Playwright.

There's a demo of it how it works here: https://www.loom.com/share/ca43c56587774835b02cd45ce89de002

TL;DW

We have stories in Storybook like this:

• SubDomainNavBar in mobile, showing search results
• Button focus states

Both of these get visual snapshots generated automatically, simply by having its story exist. We use interaction tests to give users a quick deep-linkable way to share component behaviour, but more importantly we also run accessibility testing in CI against each story. So we get 3 benefits in one!

Here are the corresponding snapshots for the examples above, which we can generate either locally through Docker or on the server.

• Snapshot for SubdomainNavBar in mobile, showing search results
• Snapshot for Button with focus state

[broccolinisoup]

Hi all 👋🏼

@lesliecdubs we can definitely add guidance around VRT for interaction tests! Currently, we have Interaction tests as a section but they are more about unit tests and Storybook interaction test and not so much about Playwright or from a VRT standpoint. Also we don't have anything documented about Chromatic on that doc either. The main reason of it, I guess we still haven't land on a decision on whether we are going to stop using Chromatic and completely onboard Playwright or use a bit of both. I am not sure if we have discussed this somewhere else and apologised if I missed it but I personally find confusing to use both 😞 I am leaning towards Playwright if we are going to choose one 😅

I'll chime in a little bit about my reasoning by sharing my humble experience writing both Playwright and Storybook interaction tests for UnderlineNav🙇🏼

• Storybook interaction tests were harder for me to write than Playwright and I feel it was certainly because of Playwright giving us the capability to capture some of the interaction, as @joshblack mentioned above. In my experience Playwright set up a nice baseline, I did some clean up and adjustments on the tests structure but writing the actual interaction was a smooth experience.
• As far as I see, we don't write stories to test responsiveness unless explicitly setting chromatic: { viewports: [] } ? Maybe Chromatic generate snapshots for multiple viewports? I am not sure but with Playwright I liked how we write nested structure for tests to test various viewports for UnderlineNav.
• I also, see us writing both Playwright interaction test for VRT and Storybook interaction tests for user behaviour demonstration, as @rezrah mentioned above and I really liked that for UnderlineNav.
• Going with Playwright, will bring consistency with the Primer Brand ✨
• Last but not least Chromatic costs $$ and owns the snapshots, Playwright is free and we get to host our snapshots 😄

Leaving some links that I referenced:

• Snapshot of overflow menu open for UnderlineNav - Playwright
• Snapshot of keeping selected item visible for UnderlineNav - Playwright
• Storybook Interaction tests for UnderlineNav

Looking forward to hearing your thoughts 👀

[lesliecdubs]

Thanks all!

@primer/react-reviewers is there more you all want to do to validate Playwright over Chromatic, or are we pretty aligned on Playwright at this point?

[github-actions[bot]]

Hi! This issue has been marked as stale because it has been open with no activity for 180 days. You can comment on the issue or remove the stale label to keep it open. If you do nothing, this issue will be closed in 7 days.