garris
high false positive rate in static site?
Hello! Thanks for developing this tool. I'm trying to use it to automate visual regression tests for my static site, since it has lots of pages and I can't always keep an eye on everything. However, I'm unable to get anything working. I've set up backstop as per the documentation in the README, cross-checked with a "static site" example, and made adjustments that I thought would help -- all to no avail: I am consistently getting false positives.
Specifically, it seems like about 25% of all my pages do not load properly when either making references or comparing. One clear example of this is a page with default styling (evidently missing CSS), but there are other, more subtle problems (e.g., a page in which all text is italicised, for some reason?). There seems to be a fixed probability of this happening for any given page. As a result, after 15 runs, I haven't once had a clean report while creating a reference of a page, then comparing that same page, unchanged, to the reference.
I'm running backstop on a python -m http.server local server over HTTP. I have configured the load wait time to be 5 seconds (to give the chromium instance time to load all the assets etc). My site is less than 200kb total per page, so I can't imagine why it would even need that long. When I manually open the site in Chrome, and spam "Refresh without cache", there is visually no load time at all.
What am I doing wrong? Are the steps I can take to mitigate the weird loading failures? Is there something else I'm missing?
Thank you!
Example of a failure:
I'm having a similar issue. While all the assets are loading as expected, I'm just finding small differences between reference and test images that I get a fail. I'm also yet to generate reference images that match test images once in running backstop test about 15-20 times. It's a shame because I know this should work, I'm just not sure how to handle these mismatches because it's clear they're not going to go away.
@garris any advice would be much appreciated! I wondered if perhaps trying to run it through another browser engine (Firefox) is an option or even possible?
Thanks!
Sorry, just seeing the original post. This looks like css not loading on the test run.
There are lots of things that can cause this. Issue is probably environmental. First rule out app issues by troubleshooting locally if possible.
Best wishes!
Sorry, just seeing the original post. This looks like css not loading on the test run.
There are lots of things that can cause this. Issue is probably environmental. First rule out app issues by troubleshooting locally if possible.
Best wishes!
It seems to have resolved itself for me, messing around with some of the config (delay, misMatchThreshold etc.) seemed to get me through the test step.