RFC: support global `playwright.config.js|ts` for browser checks

[tnolet]

Playwright recommends delegating a lot of settings to the per project global playwright.config.js file. We currently do not support this for legacy reasons. We should support this and enable users to leverage the options it gives them.

Core use cases

Based on feedback from users we found the following two use cases to be core:

1. Using the global and per project setup / teardown options to establish shared authentication state per PWT test run.
2. Using the global and per project use options to set defaults for viewports, headers etc.

User journey

1. User has a PWT config file in their repo. This file can include local settings that work for non-Checkly use cases, i.e. local testing.
2. User adds a browser check
3. User run npx checkly test and the CLI prints out that it has detected and is using the playwright.config.ts

% npx checkly test --use-pwt-config
Parsing your project... done
Using playwright.config.ts found at the root of this project.
Running 1 check in eu-west-1.

__checks__/example.spec.ts
  ✔ example.spec.ts (5s)

5. If the playwright.config.ts can't be parsed (syntax errors, unsupported dependencies etc.) we print an error before running the test command.
6. A user deploys their project to Checkly and the CLI prints again it is using the config file

% npx checkly deploy --use-pwt-config
Parsing your project... done
Using playwright.config.ts found at the root of this project.
Successfully deployed project "My Project" to account "my Checkly account".

Commands

This addition impacts the test and deploy commands. For both commands we add two flags:

1. --pwt-config-file : an optional path to define the config file. Defaults to playwright.config.ts and if not found playwright.config.js
2. --use-pwt-config : a boolean value whether to use the config at all. Default false. Later we can default it to true if stable.

Supported options

global config

This is the stock global config file created on initialising a PWT repo with all the extra advanced options added.

export default defineConfig({
  testDir: './tests',
  /* Run tests in files in parallel */
  fullyParallel: true,
  /* Fail the build on CI if you accidentally left test.only in the source code. */
  forbidOnly: !!process.env.CI,
  /* Retry on CI only */
  retries: process.env.CI ? 2 : 0,
  /* Opt out of parallel tests on CI. */
  workers: process.env.CI ? 1 : undefined,
  /* Reporter to use. See https://playwright.dev/docs/test-reporters */
  reporter: 'html',
  // Glob patterns or regular expressions to ignore test files.
  testIgnore: '*test-assets',
  // Glob patterns or regular expressions that match test files.
  testMatch: '*todo-tests/*.spec.ts',
  // Folder for test artifacts such as screenshots, videos, traces, etc.
  outputDir: 'test-results',
  // path to the global setup files.
  globalSetup: require.resolve('./global-setup'),
  // path to the global teardown files.
  globalTeardown: require.resolve('./global-teardown'),
  // Each test is given 30 seconds.
  timeout: 30000,
  expect: {},
  use: {},
  /* Configure projects for major browsers */
  projects: [],
  /* Run your local dev server before starting the tests */
  // webServer: {
  //   command: 'npm run start',
  //   url: 'http://127.0.0.1:3000',
  //   reuseExistingServer: !process.env.CI,
  // },
});

option	supported	description
testDir	❌	files are referenced thru the Checkly config.
fullyParallel	❌	currently not supported due to runtime model.
forbidOnly	✅	can be supported. Has no direct impact.
retries	✅	allow retry defaults.
workers	❌	currently not supported due to runtime mode.l
reporter	❌	has no effect as we parse the output specifically for Checkly.
testMatch	❌	files are referenced thru the Checkly config.
testIgnore	❌	files are referenced thru the Checkly config.
outputDir	❌	has no effect as we parse the output specifically for Checkly.
globalSetup	✅	Should be supported as it enables key use cases like auth.
globalTeardown	✅	Should be supported as it enables key use cases like auth.
timeout	✅	should be be supported.
use	✅	See use config
expect	✅	see expect config
projects	✅	see projects config
webServer	❌	has no effect as a user has no interactive access to the runtime

global expect config

See https://playwright.dev/docs/test-configuration#expect-options

  expect: {
    // Maximum time expect() should wait for the condition to be met.
    timeout: 5000,
    toHaveScreenshot: {
      // An acceptable amount of pixels that could be different, unset by default.
      maxDiffPixels: 10,
    },
    toMatchSnapshot: {
      // An acceptable ratio of pixels that are different to the total amount of pixels, between 0 and 1.
      maxDiffPixelRatio: 0.1,
    },
  },

option	supported	description
timeout	✅	Can be supported.
toHaveScreenshot	❌	currently not supported due to runtime model.
toMatchSnapshot	❌	currently not supported due to runtime model.

use config

See https://playwright.dev/docs/test-use-options

  use: {
    // Base URL to use in actions like `await page.goto('/')`.
    baseURL: 'http://127.0.0.1:3000',
    // Populates context with given storage state.
    storageState: 'state.json',
    colorScheme: 'dark',
    // Context geolocation.
    geolocation: { longitude: 12.492507, latitude: 41.889938 },
    // Emulates the user locale.
    locale: 'en-GB',
    // Grants specified permissions to the browser context.
    permissions: ['geolocation'],
    // Emulates the user timezone.
    timezoneId: 'Europe/Paris',
    // Viewport used for all pages in the context.
    viewport: { width: 1280, height: 720 },
    deviceScaleFactor: 2,
    hasTouch: true,
    isMobile: false,
   javaScriptEnabled: true,
    acceptDownloads: false,
    // An object containing additional HTTP headers to be sent with every request.
    extraHTTPHeaders: {
      'X-My-Header': 'value',
    },
    // Credentials for HTTP authentication.
    httpCredentials: {
      username: 'user',
      password: 'pass',
    },
    // Whether to ignore HTTPS errors during navigation.
    ignoreHTTPSErrors: true,
    // Whether to emulate network being offline.
    offline: true,
    // Proxy settings used for all pages in the test.
    proxy: {
      server: 'http://myproxy.com:3128',
      bypass: 'localhost',
    },
    // Capture screenshot after each test failure.
    screenshot: 'only-on-failure',
    // Record trace only when retrying a test for the first time.
    trace: 'on-first-retry',
    // Record video only when retrying a test for the first time.
    video: 'on-first-retry',
    // Maximum time each action such as `click()` can take. Defaults to 0 (no limit).
    actionTimeout: 0,
    navigationTimeout: 3000,
    // Name of the browser that runs tests. For example `chromium`, `firefox`, `webkit`.
    browserName: 'chromium',
    // Toggles bypassing Content-Security-Policy.
    bypassCSP: true,
    // Channel to use, for example "chrome", "chrome-beta", "msedge", "msedge-beta".
    channel: 'chrome',
    // Run browser in headless mode.
    headless: false,
    // Change the default data-testid attribute.
    testIdAttribute: 'pw-test-id',
   launchOptions: {
      slowMo: 50,
    },
    connectOptions: {
      wsEndpoint: 'ws://localhost:5678',
    },
    contextOptions: {
      reducedMotion: 'reduce',
    },
  },

option	supported	description
baseURL	✅	should be supported
storageState	✅	should be supported. See implementation notes
colorScheme	✅	should be supported
geolocation	✅	should be supported
locale	✅	should be supported
permissions	✅	should be supported
timezoneId	✅	should be supported
viewport	✅	should be supported
deviceScaleFactor	✅	should be supported
hasTouch	✅	should be supported
isMobile	✅	should be supported
javaScriptEnabled	✅	should be supported
extraHTTPHeaders	✅	should be supported
httpCredentials	✅	should be supported
ignoreHTTPSErrors	✅	should be supported
offline	✅	should be supported
proxy	❌	currently not supported due to runtime model.
screenshot	❌	currently not supported as Checkly applies defaults
trace	❌	currently not supported as Checkly applies defaults
video	❌	currently not supported as Checkly applies defaults
actionTimeout	✅	should be supported
navigationTimeout	✅	should be supported
browserName	❌	currently not supported as Checkly only supports Chrome and Chromium
channel	❌	currently not supported as Checkly only supports Chrome and Chromium
headless	❌	currently not supported due to runtime model.
testIdAttribute	✅	should be supported
launchOptions	✅	should be supported
connectOptions	❌	currently not supported due to runtime model.
contextOptions	✅	should be supported

projects config

Supporting the projects option is important as it is the recommended way to set up any sufficiently complex PWT codebase.

See https://playwright.dev/docs/test-projects

The trickiest one here is the testMatch option, as we do want to support matching on setup files, but not other .spec.ts files as due to our runtime we do not have access to all spec.ts files on each check run.

projects: [
    {
      name: 'setup',
      testMatch: /global.setup\.ts/,
      teardown: 'teardown',
    },
    {
      name: 'teardown',
      testMatch: /global.teardown\.ts/,
    },
    {
      name: 'chromium',
      use: devices['Desktop Chrome'],
      dependencies: ['setup','teardown'],
    },
  ],

option	supported	description
name	✅	should be supported
use	✅	see the use config
teardown	✅	should be supported
storageState	✅	should be supported. See implementation notes
dependencies	✅	should be supported
timeout	✅	should be be supported.
expect	✅	see expect config
retries	✅	allow retry defaults.
metadata	✅	should be be supported.
testMatch	✅	should be supported to enable setup and auth storage state scenarios.
fullyParallel	❌	currently not supported due to runtime model.
grep	❌	files are referenced thru the Checkly config.
grepInvert	❌	files are referenced thru the Checkly config.
testIgnore	❌	files are referenced thru the Checkly config.
testDir	❌	currently not supported due to runtime model.
outputDir	❌	currently not supported due to runtime model.
snapshotDir	❌	currently not supported due to runtime model.
snapshotPathTemplate	❌	currently not supported due to runtime model.

Implementation details

• globalSetup references a file. We need to parse that file, its dependency tree and check for unsupported dependencies. We need to then rollup and bundle these files as we do for all other dependencies.
• testMatch inside a project, typically a setup project, can reference a file. We need to also rollup that file and its dependencies.
• storageState can be defined by the user anywhere on the filesystem. We need to handle any errors there and allow the Checkly backend to (re)create that file tree at runtime.

Shortcut link

No response

Additional resources

No response

[malinskibeniamin]

Thank you for the RFC! Hope we can keep the ball rolling here.

I have posted on the community Slack and wanted to share my use case:

As part of the authentication strategy, in order to avoid having to go to an email provider to retrieve the link for accepting an invite link, I copy the URL by pressing on a button. In order for that to work on Chromium-based browsers, I need to be able to pass in the permissions flag to allow the navigator clipboard events for read/write

        contextOptions: {
          // chromium-specific permissions
          permissions: ['clipboard-read', 'clipboard-write'],
        }

if (browserName === 'chromium' || browserName === 'webkit') {
      link = await page.evaluate(() => navigator.clipboard.readText());
    }

Not providing the additional flags causes the following error:

Error: page.evaluate: DOMException: Read permission denied.

I agree that supporting projects config might solve a lot of problems for people, since I imagine others might encounter similar issues.

[tnolet]

@malinskibeniamin sorry for the late reply. I'm not sure how your clipboard example relates to projects, but I'm ok with keeping it here.

[malinskibeniamin]

Thanks! I meant that having the ability to customise projects property for checkly will allow me to pass a flag that enables asserting against clipboard events, making certain tests easier to write.

[tnolet]

Hi all, we just release a beta for this. It does not support all configuration options yet, but is already very usable. Check the full changelog here

[thareekanvarm]

Hi @tnolet, I checked your new version. But we need auth.setup.ts|js. Because for my use case my every page is accessible only if the user is logged in other wise not .so i cant run auth test using checkly. so can u guys please provide that. Its will be a better improvement.

[tnolet]

@thareekanvarm we are aware of this and as we mentioned in the changelog we will work on this for the GA release.

[MrRio]

Hey – any workarounds for storageState?

[tnolet]

@MrRio Hey James, sadly not really. The solution we want to have has some nasty intersections with security and sandboxing related issues. Think reading and writing anything to disk and Checkly persisting that. We had hoped to ship support earlier, but we had to shift platforms (Lambda to managed EKS) which brought some extra complexity.

Long story short: We are still working on it and we want to make it happen ASAP.

[tnolet]

This RFC is stale. We are currently brainstorming this. Upvotes and further requests are tracked in our feedback portal https://feedback.checklyhq.com/p/global-playwrightconfigts-support-viewport-setup-teardown-storagestate