[Regression]: Artifacts created during test execution get random string in the name

[aoj2]

Last Good Version

1.42.1

First Bad Version

1.43.0

Steps to reproduce

This problem only seems to happen when a test case is having a "long" title. Started to happen in Playwright 1.43.0. This issue breaks an update-snapshot use case for us, where we compare actual files with the current existing ones.

import { expect, test } from "@playwright/test"

test("Here is a test with a very long title to reproduce the problem", async ({ page }) => {
    await page.goto("https://playwright.dev");
    await expect(page.locator("body")).toHaveScreenshot();
});

Expected behavior

Expects to get the full test case name:

Actual behavior

Actual result, the folder and file created contains random strings.

Additional context

No response

Environment

System:
    OS: Linux 6.6 Ubuntu 22.04.4 LTS 22.04.4 LTS (Jammy Jellyfish)
    CPU: (16) x64 Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
    Memory: 5.86 GB / 7.75 GB
    Container: Yes
  Binaries:
    Node: 20.12.2 - /usr/bin/node
    Yarn: 1.22.22 - /usr/bin/yarn
    npm: 10.5.0 - /usr/bin/npm
    pnpm: 8.6.10 - /usr/bin/pnpm
  IDEs:
    VSCode: 1.89.0 - /vscode/vscode-server/bin/linux-x64/b58957e67ee1e712cebf466b995adf4c5307b2bd/bin/remote-cli/code
  Languages:
    Bash: 5.1.16 - /usr/bin/bash
  npmPackages:
    @playwright/test: 1.44.0 => 1.44.0

[mxschmitt]

where we compare actual files with the current existing ones.

In which directories are you comparing them? We did a change but this should only have affected the test-results directory, not the actual golden test expectations on disk. I tried to reproduce it on my machine, and there it was working as expectd (only test-results have changed).

// 1.42 /Users/maxschmitt/Developer/tmp/1510/test-results/project1-Here-is-a-test-with-a-very-long-title-5bd13-long-title-tlong-title-to-reproduce-the-problem-test
// 1.43 /Users/maxschmitt/Developer/tmp/1510/test-results/project1-Here-is-a-test-wi-5bd13-le-to-reproduce-the-problem-test

Do you consume them via test.info().outputDir or via the reporting api?

[aoj2]

Yes, we use the files outputted in the test-results folder. We have an manual approval step on failed test runs on CI, that if approved, runs a script that compare file names and update the golden snapshots using the actual ones from test-results.

Can you please confirm that this change is only affecting file names?

[mxschmitt]

It only affects the test-results directory which is not intended to be consumed directly. Its intended to be consumed via our APIs, either inside a test via test.info().outputDir or via the reporter API.

[yury-s]

Yes, we use the files outputted in the test-results folder. We have an manual approval step on failed test runs on CI, that if approved, runs a script that compare file names and update the golden snapshots using the actual ones from test-results.

As Max mentioned you can implement a custom reporter and access all the images including their paths via TestResult.attachments. The exact naming of the paths in toHaveScreenshot is an implementation detail and it can change between versions.

Can you please confirm that this change is only affecting file names?

The change affected file names of the diff and actual images. Actual snapshot name stays the same. It can