metamask-mobile icon indicating copy to clipboard operation
metamask-mobile copied to clipboard

test: Address App launch times failure test

Open cortisiko opened this issue 1 year ago • 7 comments

…ive waits.

Description

This PR aims to put the app launch times tests back in the green. The tests started failing because of a couple of reasons:

  • A new onboarding success screen was added to the app. The launch times test did not get updated correctly
  • The website we used for testing had UI changes that broke our tests. We will now use the mm test dapp for this test
  • The selectors used to identify elements were incorrect.

Note: The warm start tests are failing for a legit reason. The launch times are below the expected bench mark of 8 seconds hovering around ~11 seconds

Related issues

Fixes:

Manual testing steps

  1. Go to this page...

Screenshots/Recordings

Before

After

Pre-merge author checklist

  • [ ] I’ve followed MetaMask Coding Standards.
  • [ ] I've completed the PR template to the best of my ability
  • [ ] I’ve included tests if applicable
  • [ ] I’ve documented my code using JSDoc format if applicable
  • [ ] I’ve applied the right labels on the PR (see labeling guidelines). Not required for external contributors.

Pre-merge reviewer checklist

  • [ ] I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
  • [ ] I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

cortisiko avatar May 20 '24 20:05 cortisiko

https://bitrise.io/ Bitrise

❌❌❌ pr_smoke_e2e_pipeline failed on Bitrise! ❌❌❌

Commit hash: 006e563450a860c4b0e013a30d8b9e70e72dc910 Build link: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/79ff495c-229d-41a7-a2f6-b5702e8da0ed

[!NOTE]

  • You can kick off another pr_smoke_e2e_pipeline on Bitrise by removing and re-applying the Run Smoke E2E label on the pull request

github-actions[bot] avatar May 21 '24 16:05 github-actions[bot]

One smoke test is failing

There seems to be a latency issue with the app? This test does not touch anything swaps related. I will investigate on main and see.

Let's we also post a launch time test run in the comments to confirm the fix

Ah, so this is what I was discussing during the stand-up. The point of failure in the tests has been fixed, but the tests still do not pass the 8-second benchmark. App code changes are needed to address the performance issue.

. See Tomas's message here: https://consensys.slack.com/archives/C04GLB755TK/p1716311340155619

cortisiko avatar May 21 '24 18:05 cortisiko

@cortisiko Could you post a link of the successful app start time test in Bitrise?

Cal-L avatar May 21 '24 22:05 Cal-L

@Cal-L Because this test relies on changes to the app code to improve performance I went ahead and updated the bench mark from 8 seconds to 12 seconds.

Here is a screenshot of passing warm start tests Screenshot 2024-05-21 at 6 37 25 PM

I will kick off a bitrise build right now

cortisiko avatar May 21 '24 22:05 cortisiko

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

sonarqubecloud[bot] avatar May 21 '24 22:05 sonarqubecloud[bot]

https://bitrise.io/ Bitrise

✅✅✅ pr_smoke_e2e_pipeline passed on Bitrise! ✅✅✅

Commit hash: 2695475b81c6f4008faea63774bdda36c0833ca8 Build link: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/fd14331a-f2b6-4f68-a66e-f537f2368ad8

[!NOTE]

  • You can kick off another pr_smoke_e2e_pipeline on Bitrise by removing and re-applying the Run Smoke E2E label on the pull request

github-actions[bot] avatar May 21 '24 22:05 github-actions[bot]

@Cal-L Passing App launch times test on bitrise with adjusted benchmark: https://app.bitrise.io/build/4bb0f921-edf8-4685-a30d-0e2843819686

cortisiko avatar May 22 '24 00:05 cortisiko