Automattic
Canary tests fail on wpcalypso.wordpress.com but pass on production
This is to keep track of the canary test failures running against wpcalypso.wordpress.com even though they are passing on production
cc: @hoverduck
Another possibility is back-to-back wpcalypso deploys cause existing running tests to fail
Could it be that wpcalypso.wordpress.com is a slower environment?
This is true, so I tried bumping up the explicitWaitMS from 20 to 60s, but that didn't seem to make a difference (I've since returned it to the default).
What I'm seeing now that we have screenshots to Slack enabled (also via VNC) is that frequently in the signup process we're getting dumped on the login screen instead of the checkout page. The redirect in the URL is pointing back to checkout.
We're not seeing it on production, so I'm not sure yet what the problem is. The investigation continues.
I wasn't able to reproduce this problem locally, either manually or with the scripts. Most of my time was spent futzing with CircleCI VNC problems, and had to keep restarting the session.
The only error I was able to track down was one time I saw a post-message failure from a wordpress.com iframe to wpcalypso, which I'm guessing may have had something to do with the oauth token. Then once that's gone screwy it thinks we're not logged in, and dumps us back on the login page instead of checkout.
But like I said, I wasn't able to reproduce it locally to confirm any of that. I'll dig in again and ping the Calypso folks on Monday, but we could also add a workaround to the signup flow to detect the login page and just re-authenticate. If it happens in production that's what the end user would do anyway.
I've added a workaround in https://github.com/Automattic/wp-e2e-tests/pull/444, and asked the Calypso devs on p4TIVU-6cx-p2
I'm going to leave this issue open and re-label it as workaround
This bug is rearing its head more frequently, and the workaround from #444 isn't working consistently anymore. I assume it's due to the work on the new login screen, but this screenshot (pulled from the Slack alert) looks like the wp-login.php page, not the new /log-in.
Swapped out the paid sign up tests for free sign up test (temporarily) as this issue is causing us to lose confidence in the canaries. See #649
Is there another environment we could use besides wpcalypso? Horizon? Or will that give us much the same results?
Is there another environment we could use besides wpcalypso? Horizon? Or will that give us much the same results?
We could try Horizon to see if it suffers the same problem...it should have the same Calypso baseline as wpcalypso, barring any feature flag differences.
Since the refresh bug (https://github.com/Automattic/wp-calypso/pull/16784) was fixed I am reenabling the paid sign up tests to see if we're still having issues with wpcalypso
See: https://github.com/Automattic/wp-e2e-tests/pull/653
Updated in https://github.com/Automattic/wp-e2e-tests/pull/740 and all fine so closing this issue now
Seeing this for free signups now :/
https://circleci.com/gh/Automattic/wp-e2e-tests-canary/9447#artifacts/containers/0
Would it be worth turning on the video recordings for the canaries to get more detail about what's happening, or is that too much overhead for these tests?
@hoverduck I've got it recording a PR here now and reproduced it in video: https://circleci.com/gh/Automattic/wp-e2e-tests/22412#artifacts/containers/0
I'm still trying to work out why