studio icon indicating copy to clipboard operation
studio copied to clipboard

Fix navigation e2e tests

Open gcsecsey opened this issue 5 months ago • 2 comments

Related issues

I noticed that after merging #2054, a couple of tests were still flaky on trunk. This PR fixes the issues reported on these runs: 39bbd-pb

Proposed Changes

  • Increases the timeout for the "View Post" button after post creation
  • Attempts to close the welcome guide modal more times
  • Adds a wait for networkidle after media upload

Testing Instructions

  • CI should be green
  • Visual review

Pre-merge Checklist

  • [ ] Have you checked for TypeScript, React or other console errors?

gcsecsey avatar Nov 27 '25 16:11 gcsecsey

📊 Performance Test Results

Comparing a8a479f36ae5235ac9b8d28b85acd83c6bd7436c vs trunk

site-editor

Metric trunk a8a479f36ae5235ac9b8d28b85acd83c6bd7436c Diff Change
load 8212.00 ms 9479.00 ms +1267.00 ms 🔴 15.4%

site-startup

Metric trunk a8a479f36ae5235ac9b8d28b85acd83c6bd7436c Diff Change
siteCreation 18223.00 ms 15063.00 ms -3160.00 ms 🟢 -17.3%
siteStartup 5950.00 ms 5956.00 ms +6.00 ms 🔴 0.1%

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change

github-actions[bot] avatar Nov 27 '25 16:11 github-actions[bot]

I tested it locally and for me the test creates a post fails. It seems that it correctly clicks the Publish button, but then it automatically clicks the "View post" button even before the post has been saved, so when the posts load, the new post doesn't appear yet.

e2e-view-post-error.mp4 I tested with these two commands: npx playwright test --grep "Site Navigation" --ui

and

❯ npm run e2e -- site-navigation.test.ts

> [email protected] e2e
> npx playwright install && npx playwright test site-navigation.test.ts


Running 9 tests using 1 worker

  ✓  1 e2e/site-navigation.test.ts:77:6 › Site Navigation › opens site at homepage (626ms)
  ✓  2 e2e/site-navigation.test.ts:94:6 › Site Navigation › opens and automatically logs in to WP Admin (4.9s)
  ✘  3 e2e/site-navigation.test.ts:110:6 › Site Navigation › creates a post (21.6s)
  ✓  4 e2e/site-navigation.test.ts:156:6 › Site Navigation › uploads media (9.3s)
  ✓  5 e2e/site-navigation.test.ts:192:6 › Site Navigation › activates themes (2.4s)
  ✓  6 e2e/site-navigation.test.ts:218:6 › Site Navigation › adds new themes (17.6s)
  ✓  7 e2e/site-navigation.test.ts:247:6 › Site Navigation › activates plugin (3.0s)
  ✓  8 e2e/site-navigation.test.ts:269:6 › Site Navigation › adds new plugin (18.3s)
  ✓  9 e2e/site-navigation.test.ts:299:6 › Site Navigation › "Post name" permalink structure works (13.6s)


  1) e2e/site-navigation.test.ts:110:6 › Site Navigation › creates a post ──────────────────────────

    Error: Timed out 10000ms waiting for expect(locator).toBeVisible()

    Locator: locator('a.row-title:has-text("E2E Test Post")')
    Expected: visible
    Received: <element(s) not found>
    Call log:
      - expect.toBeVisible with timeout 10000ms
      - waiting for locator('a.row-title:has-text("E2E Test Post")')


      151 |             // Verify post was created by visiting posts list
      152 |             await page.goto( getUrlWithAutoLogin( `${ wpAdminUrl }/edit.php` ) );
    > 153 |             await expect( page.locator( 'a.row-title:has-text("E2E Test Post")' ) ).toBeVisible();
          |                                                                                     ^
      154 |     } );
      155 |
      156 |     test( 'uploads media', async ( { page } ) => {
        at /Users/macbookpro/Documents/projects-m3.nosync/studio/e2e/site-navigation.test.ts:153:75

    Error Context: test-results/site-navigation-Site-Navigation-creates-a-post/error-context.md

    attachment #2: trace (application/zip) ─────────────────────────────────────────────────────────
    test-results/site-navigation-Site-Navigation-creates-a-post/trace.zip
    Usage:

        npx playwright show-trace test-results/site-navigation-Site-Navigation-creates-a-post/trace.zip

    ────────────────────────────────────────────────────────────────────────────────────────────────

  1 failed
    e2e/site-navigation.test.ts:110:6 › Site Navigation › creates a post ───────────────────────────
  8 passed (2.3m)

Thanks @sejas for the sharing the detailed logs, I was occasionally getting this error but thought that the most recent changes fixed it. 🤔 The CI also failed on a different issue again where the Welcome modal obstructs the view:


2) e2e/site-navigation.test.ts:301:6 › Site Navigation › "Post name" permalink structure works ───
--
 
Test timeout of 180000ms exceeded.
 
Error: locator.click: Test timeout of 180000ms exceeded.
Call log:
- waiting for locator('.editor-post-publish-panel__header-publish-button button.editor-post-publish-button__button')
- locator resolved to <button type="button" aria-disabled="false" class="components-button editor-post-publish-button editor-post-publish-button__button is-primary is-compact">Publish</button>
- attempting click action
2 × waiting for element to be visible, enabled and stable
- element is not stable
- retrying click action
- waiting 20ms
2 × waiting for element to be visible, enabled and stable
- element is visible, enabled and stable
- scrolling into view if needed
- done scrolling
- <div class="components-modal__screen-overlay">…</div> intercepts pointer events
- retrying click action
- waiting 100ms
108 × waiting for element to be visible, enabled and stable
- element is visible, enabled and stable
- scrolling into view if needed
- done scrolling
- <div class="components-modal__screen-overlay">…</div> intercepts pointer events
- retrying click action
- waiting 500ms
- waiting for element to be visible, enabled and stable
- element is not enabled
211 × retrying click action
- waiting 500ms
- waiting for element to be visible, enabled and stable
- element is visible, enabled and stable
- scrolling into view if needed
- done scrolling
- <div class="components-modal__screen-overlay">…</div> intercepts pointer events
- retrying click action
- waiting 500ms
 
 
351 \|
352 \| 		await confirmPublishButton.waitFor( { state: 'visible', timeout: 10_000 } );
> 353 \| 		await confirmPublishButton.click();
\| 		                           ^
354 \|
355 \| 		// Wait for the "View Post" link to appear
356 \| 		await page
at /opt/ci/builds/builder/automattic/studio/e2e/site-navigation.test.ts:353:30
 
Error Context: test-results/site-navigation-Site-Navig-ddb91-e-permalink-structure-works/error-context.md


I'll iterate on handling these more robustly and ping you for another review soon.

gcsecsey avatar Nov 28 '25 11:11 gcsecsey

Thanks for reviewing this a couple times @epeicher! I also tested on Mac and Windows using Parallels, and the suite is consistently passing for me as well. I think we're good to merge this and iterate on these tests if needed.

Mac Windows
Image 02-12-2025 at 11 50 CleanShot 2025-12-02 at 12 30 18@2x

@epeicher @sejas could you please give this another test and approve this if it passes?

gcsecsey avatar Dec 02 '25 12:12 gcsecsey