manager icon indicating copy to clipboard operation
manager copied to clipboard

Fix flaky Cypress volume tests, `cy.visitWithLogin(...)` improvements

Open jdamore-linode opened this issue 3 years ago • 3 comments

Description

What does this PR do?

  • Adds the ability to override local storage, individual user preferences, and automatically mock common API requests via cy.visitWithLogin(), facilitating these improvements:
    • Volume tests now force the landing page to show up to 100 results, mitigating the issue we're having with stuck volumes and fixing all of the flakiness I'm currently aware of with regards to volumes
    • Some Linode and Volume smoke tests had an issue where the Akamai/Linode billing banner still appears even though it should be dismissed; allowing user preferences to be overridden individually resolves this issue, as mocks were causing the dismissed banner preference to be dropped
    • The /account API request is now mocked for all tests except some billing tests, dramatically improving test speeds for accounts where that endpoint is particularly slow

How to test

What are the steps to reproduce the issue or verify the changes? You can run Cypress to confirm that these changes do not break anything, but testing that they fix the issues as described is a little bit trickier if your test account doesn't already have a bunch of stuck volumes. Feel free to reach out to me on Slack, and I can help you set up your .env file with a test account that will enable proper testing.

jdamore-linode avatar Jul 08 '22 21:07 jdamore-linode

Odd, I'll look into that one! Thanks @bnussman !

jdamore-linode avatar Jul 11 '22 15:07 jdamore-linode

I've updated the PR to enable the automatic /account mocking for all tests, excluding a few billing tests.

Using one of the test accounts that had timeouts when requesting /account, running yarn cy:run took me 57 minutes. Running the same tests with /account mocking enabled took only 27 minutes.

jdamore-linode avatar Jul 11 '22 16:07 jdamore-linode

@jdamore-linode tests ran and passed for me locally. Soft approval, as i haven't looked deeply at the code. ☑️

patthiel avatar Jul 28 '22 16:07 patthiel

Hey @DevDW, been spending some time on this today and haven't been able to reproduce either of those errors! I've never seen the timeout on cy.screenshot, but I did find a GitHub Issues thread suggesting it may be a longstanding (and still unfixed) Cypress bug.

Do you think you'd be able to give this another try next time you get a chance?

jdamore-linode avatar Aug 16 '22 14:08 jdamore-linode

Thanks @DevDW! If it brings any peace of mind, that failure is one we've been seeing for the past week or so (not related to this PR), and I've written a little bit about it in the frontend notification channel if you're curious.

jdamore-linode avatar Aug 17 '22 13:08 jdamore-linode