[Feature] Run project against other one (i.e. staging tests with all browsers on given environment)

[jaktestowac]

In some cases we need to run given project with different browsers or environments. I wish to have i.e. in UI mode:

Currently when we having many projects, what need to be tested against multiple devices, we need to make a lot of copy pasting:

   {
          name: 'Smoke-ff',
          grep: /@smoke/,
          use: {
            ...devices['Desktop Firefox'],
          }          
        },
        {
          name: 'Smoke-ch',
          grep: /@smoke/,
          use: {
            ...devices['Desktop Chrome'],
          }          
        },
        {
          name: 'Smoke-sf',
          grep: /@smoke/,
          use: {
            ...devices['Desktop Safari'],
          }          
        },
   {
          name: 'Integration-ff',
          grep: /@integration/,
          use: {
            ...devices['Desktop Firefox'],
          }          
        },
        {
          name: 'Integration-ch',
          grep: /@integration/,
          use: {
            ...devices['Desktop Chrome'],
          }          
        },
        {
          name: 'Integration-sf',
          grep: /@integration/,
          use: {
            ...devices['Desktop Safari'],
          }          
        },

Splitting projects by i.e. types and adding possibility to merging (running together) can give good possibility for flexible test running:

    {
      name: "Smoke",
      type: category,
      grep: /@smoke/,
    },
    {
      name: "Integration",
      type: category,
      grep: /@integration/,
    },
    {
      name: "Chrome",
      type: browser,
      use: { ...devices["Desktop Chrome"] },
    },
    {
      name: "Firefox",
      type: browser,
      use: { ...devices["Desktop Firefox"] },
    },
    {
      name: "Safari",
      type: browser,
      use: { ...devices["Desktop Safari"] },
    },
    {
      name: "Staging",
      type: environment,
      use: { baseURL: "stage.app.com" },
    },
    {
      name: "Production",
      type: environment,
      use: { baseURL: "production.app.com" },
    },

We can use it as before: npx playwright test --project Smoke or with type (i.e. run Smoke across all type browser) npx playwright test --project Smoke -with-type browser or specify type for given environment npx playwright test --project Smoke -with-type browser -with-type environment:Staging

At the end suggested configuration with types can be translated to current projects syntax (in above example we end with 18 projects), so user can check final result how it will be merged.

Finally this change can introduce possibilities to keep configuration of projects much more organized and maintainable. For backward compatibility nothing need to be changed until type is used. Types can be narrowed to suggested to avoid over complicated matrix.

This feature was inspired by https://github.com/microsoft/playwright/issues/13046

[gskierk]

I would say the best place for custom strategies is the user's land.

As was said in the referenced issue, you can pass multiple --project options to run multiple projects.

But to handle tons of possibilities and avoid copy-pasting, you can define your parameters and generate all their combinations based on your custom project template, e.g., smoke-chrome-staging, etc.

Then, all you need is npx playwright test --project smoke-chrome-staging --project smoke-safari-staging.

[jaktestowac]

@gskierk Playwright itself is full of custom strategies (how you explain famous dependencies: in project while you can do this by fixtures?) why do not add another useful one.

If generating tons of project for config file is quite easy (and not efficient) why not add it as simplified types to the Playwright, specially if we think about using tons of projects in UI mode - it will make its UI mode ui (sic!) burning🔥

I don't want to end with npx playwright test --project smoke-chrome-staging --project smoke-safari-staging --project smoke-firefox-staging --project smoke-safari-nonprod --project smoke-safari-nonprod --project integration-safari-nonprod --project integration-chrome-staging --project integration-safari-staging --project integration-firefox-staging --project integration-safari-nonprod --project integration-safari-nonprod --project integration-safari-nonprod while I wish to do just i.e. :
--npx playwright test --project [smoke, integration] -with-type browser -with-type environment:[staging, nonprod]

[dgozman]

@jaktestowac I'd still recommend doing this in your own config. For example,

TESTS="smoke,integration" ENVIRONMENT="staging,nonprod" npx playwright test

And then in your config generate projects based on environment variables:

for (const tests of process.env.TESTS.split(',')) {
  for (const env of process.env.ENVIRONMENT.split(',')) {
    config.projects.push({
      name: `${tests}-${env}`,
      grep: new RegExp('@' + tests),
      use: { baseURL: `https://${env}.example.com` },
    });
  }
}

This gives you the most flexibility and allows you to quickly iterate on your preferred way of organizing tests.

[jaktestowac]

@dgozman thanks for nice code snipped. At least can we add this as future feature and listen community for feedback?

[zola-25]

@jaktestowac I'd still recommend doing this in your own config. For example,

TESTS="smoke,integration" ENVIRONMENT="staging,nonprod" npx playwright test

And then in your config generate projects based on environment variables:

for (const tests of process.env.TESTS.split(',')) {
  for (const env of process.env.ENVIRONMENT.split(',')) {
    config.projects.push({
      name: `${tests}-${env}`,
      grep: new RegExp('@' + tests),
      use: { baseURL: `https://${env}.example.com` },
    });
  }
}

This gives you the most flexibility and allows you to quickly iterate on your preferred way of organizing tests.

@dgozman I appreciate this, and it makes sense I think for more simple project configurations.

I still +1 this feature, over dynamically generating config, for two reasons

1. I feel that when you dynamically generate config, you're pushing back your different project configurations, from the config file, into environment variables - This is fine when project configurations only vary by one or two parameters, but what if, for example, you want to add different number of retries depending on the browser? (Something I had to do recently) - Then you've got another environment variable to add or more logic in your config generation code. So it can get a bit messy.

2. Dynamic configuration seems to not integrate well with other developer tools?

Like VS Code for example - Say for some projects I want to use different tests, set in a different test folder - setting different hardcoded testDir: folders - at the project level and/or root level - is picked up by the IDE and all the tests, for all projects and folders, show up in the Testing tab. But when I set it dynamically at the root based on an environment variable, they're not picked up automatically.

So I'm more inclined to having as much configuration set in one place as possible, but also with as little repetition as possible as your sample snippet achieves. So the ability to specify multiple browsers at the project level would really help achieve the right balance here.

Ideally then environment variable are only needed for secrets, or maybe for certain situations where you can't override a setting with the CLI.

[olenahuk]

This all is good but for example we have 11 sites to test x3 environments to run tests on (prod, pre-prod and test) and after that if we want more than one device we would nee to do x3? It would be much much easier just set devices [device1, device2, device3] than triple already existing config.