Test failing in Smoke-test, fonts not loaded?

[klodoma]

I have this situation that some tests are sometimes randomly failing. I noticed the same situation in the smoke-tests.

I cannot figure out why this test is constantly failing, I've tried different setups, but I don't understand why.

The code is in this branch: https://github.com/klodoma/BackstopJS/tree/debug/failing-tests

You can download the test results and artifacts here: https://gitlab.com/klodoma/BackstopJS/-/pipelines/353734894

I want to mention that I have this situation quite often in my tests, that for some reason the "rendered text size" is rendered differently. Maybe it's a bug in pupeteer and it's quite common.

One suspect is that the screenshot is made before the font-files are loaded.

Any thoughts are welcomed.

{
      label: 'keyPressSelector',
      url: `${URL}/examples/featureTests/index.html`,
      keyPressSelectors: [
        {
          selector: 'input[placeholder="Email"]',
          keyPress: '[email protected]'
        },
        {
          selector: 'input[placeholder="Password"]',
          keyPress: '1234'
        }
      ],
      selectors: ['div[id=navbar]'],
      postInteractionWait: 2000,
      viewports: [
        {
          label: 'Desktop',
          width: 800,
          height: 300
        }
      ]
    }

[edvardsniedre]

Likely font loading issue. If it takes too long for your fonts to load BacksotpJS will not wait for it. I am not sure if there is a way how to listen for font to be loaded/rendered in BackstopJS, but the way I solved it for myself is storing font files locally and intercepting fonts server request during a test same as done here for imageStub example. I use this approach for both images and fonts as I had same issue when test fails because server takes longer to respond and since then it was stable and not flaky.

[garris]

It could also be related to this https://github.com/garris/BackstopJS#using-docker-for-testing-across-different-environments -- you can try running the docker version to see if the same thing happens.

[klodoma]

It could also be related to this https://github.com/garris/BackstopJS#using-docker-for-testing-across-different-environments -- you can try running the docker version to see if the same thing happens.

What is the quickest way to mount the local code in docker?

[garris]

This is built-in -- You can just use the --docker argument to your test command. Backstopjs will make a call to docker and render in the container. Details here

e.g. backstop test --docker

[klodoma]

This is built-in -- You can just use the --docker argument to your test command. Backstopjs will make a call to docker and render in the container. Details here

e.g. backstop test --docker

@garris I am aware of the --docker flag but that uses the docker version source code, which is not what I want on local execution. I haven't checked how the docker container is built, Is there an easy way to mount the local sources in the docker container?

So practically, execute everything in the docker container, BUT, the sources should be taken from the local machine.

[garris]

Oh, I think I understand.

If you want to build a docker image into your local docker repo from the current BackstopJS source...

Locally from backstop root

docker build docker -t backstopjs/backstopjs:x.x.x

Make sure that x.x.x is the current version value set in package.json . When BackstopJS runs in --docker mode, it will look for the docker image with the same version.

[Jayarams99]

I also face similar issues when running the test .The font size is showing as minor variance for all the elements on the header .Please let me know if docker setup is the work around

[klodoma]

@Jayarams99 yes, running it through docker is a better option to avoid differences between different OS's and you'll have to preload fonts in order to avoid issues.

Though in the test case that I posted, which is part of the backstopjs testing, I have the feeling it's something else.

Didn't had enough time though to dig in more deeply.