feedback-assistant
FB13292311: First-class support for snapshot testing
- Date: 2023-10-20
- Resolution: Open
- Area: XCTest
- OS: N/A
- Type: Suggestion
- Keywords:
Description
For a little background, snapshot tests are tests where an image is generated from a view and that image is stored in the codebase. Subsequent runs of the test will check that the newly generated image matches the image in the codebase and if they do not match it will fail the test. This style of test has been around for years and in my opinion has gotten more popular recently with the introduction of Point-Free's Swift Snapshot Testing library (https://github.com/pointfreeco/swift-snapshot-testing).
Snapshot testing gives developers a tool to handle the large number of different UI permutations that their users will interact with because they allow us to create a snapshot of each permutation which is great during code reviews. We can see if the accessibility Dynamic Type sizes cause the layout to break, or text to truncate in ways we don't want it to. We can see that we're using semantic colors and that everything looks right in Dark Mode. With Accessibility Snapshot Testing (https://github.com/cashapp/AccessibilitySnapshot) we can even see what a VoiceOver user would experience.
Without these tests our alternative is to either run the app locally with a variety of different environments or to try to use previews to see those different permutations. But neither of those works great for VoiceOver accessibility where we'd need to run on a device to fully test the VoiceOver UI.
To summarize, I think that snapshot tests are valuable to the development process and to ensuring that our apps serve our customers as well as possible. But the world of snapshot testing is not a utopia and I think that is where y'all can help.
The Swift Snapshot Testing library relies upon one of two different Apple API, depending on the settings provided to it and the environment in which the tests are running. The two API are:
- drawHierarchy(in:afterScreenUpdates:) https://developer.apple.com/documentation/uikit/uiview/1622589-drawhierarchy
- render(in:) https://developer.apple.com/documentation/quartzcore/calayer/1410909-render
The first, drawHierarchy, requires that tests are run inside of a host application. The second does not have that same requirement but also does not provide as accurate of output as drawHierarchy. In addition, each of these methods provide slightly different output on different processor combinations.
I think these problems are partially caused by the fact that these API were not designed to support snapshot testing (I'm sure that they are also partially caused by the fact that this is a hard problem). I'm hoping that I can convince y'all that these API should have first-class support for snapshot testing or that different API should be added for that purpose. What I mean by first-class support is that the API would acknowledge snapshot testing as a valid use case for the API and have that use case in mind when making changes to the API or to the systems that could influence the output of the API. For one small example, tests could be setup around the API to ensure that the same output is generated on both Apple Silicon and Intel processors.
