FRSOURCE
feat: support confirmation of initial image baselines
Is your feature request related to a problem? Please describe. Currently, the first run of a new image match defines the baseline and passes by default. I would like a behavior, where a test fails, if no baseline was explicitly defined yet. This enables developers to double check their initial test setup and not commit faulty tests. It also enables a nice workflow, where I as a developer can write a test for my feature or change and confirm it with another stakeholder.
Describe the solution you'd like I'd like to introduce a configuration e.g 'autoCreateBaseline'. On true, the current behavior should apply. But if set to false, a test should fail, if no baseline exists for matchImage.
Describe alternatives you've considered Right now, the best alternative is to check your version control for newly created baselines, but it doesn't force developer to check it.
Additional context Personally, I'd suggest to disable this behavior by default in the future, because this ensures a cleaner test development flow and follows a more test driven methodology where tests are supposed to fail initially until implemented. I do understand that visual regression tests are most often added in the end, when the ui is mostly fixed, which might be a reason why acceptance is the desired default for most. Although, even in that case, I think a explicit verification is good practice, as it helps to spot bugs.
If you like this option, I'd be happy to provide a PR for it. Really loving the plugin ❤
Hey @goatfryed 👋
Thank you for filing this issue - the feature you have described sounds indeed like a reasonable addition!
Name of the option you've proposed autoCreateBaseline also sounds good and descriptive, I think we can go with it. You're more than welcome to provide a PR if you're able - I'm currently sick, so it's hard for me to spend as much time as I would like on development of this library.
But I would not recommend to change the default behaviour for now. We might think about it for the future though. From my perspective - I've worked with multiple libs/platforms for visual regression testing and none of those had baseline images generation disabled by default. Also your ticket is the first one that references this issue, so I'm not sure if that is a sensible default for whole community.
In the future I'm thinking about providing few more features to this library - for example ready-to-be-used github action that would enable more automated workflows and encourage good practices for our users. Then we might also a breaking change like this and just change the default behaviour 🙂
Hey @goatfryed - I have just created a PR with createMissingImages option which I believe resolves your use case. Would you mind having a look at this PR and giving me feedback if that's what you have had in mind?
Hey @FRSgit, First of all sorry for being unresponsive on this one. Thank you a lot for your time and effort you put into this plugin and for following up on my request.
Yes, that looks great. That's exactly what I had in mind. 👍
It seems I forgot to close this ticket, as it's done. Thanks @goatfryed for the filed issue once again and I hope my addition to the plugin works for you!