[Feature]: Add the default console spy

[kettanaito]

🚀 Feature Request

Spying on the browser's console is something I do almost in every project. It's a great way to test side-effects and ensure no errors were printed (if we exclude the page errors). It's also critical for asserting on logic that prints to console.

I propose to add a built-in consoleSpy API to Playwright. The purpose of that API is to take page.on('console') and give a develoer-friends API over it. I'll talk more about this in the Motivation section of this proposal.

Example

Here's what a console spy API looks like:

test('my playwright test', async ({ consoleSpy }) => {
  await doAction()
  expect(consoleSpy.get('log')).toEqual(['hello world'])
})

The exact implementation of the console spy is debatable but I find it convenient to use Map to group messages by type. You rarely need to assert the entire console output, and when you do, you can always have a getter that returns you all the messages of all the types.

Internally, what I usually do for this kind of API is this:

const messages = new Map()

page.on('console', (message) => {
  messages.set(message.type(), (messages.get(message.type()) || []).concat(message.text())
})

return messages

Motivation

The default page.on('console') API is great but it's low-level. Using it on a regular basis becomes verbose and inefficient.

test('asserts on console warnings', ({ page }) => {
  const warnings = []

  page.on('console', (message) => {
    if (message.type() === 'warning') {
      warnings.push(message.text())
    }
  })

  // doAction

  expect(warnings).toContain('foo')
})

Now imagine this in a test suite that asserts on warnings in every test. This quickly gets out of hand.

The console API in Playwright is also event-based, which forces me to introduce an intermediary data structure, like the warnings array in the test above, just to accumulate values. That's not nice and it's awkward to use.

A built-in consoleSpy can breach the developer experience gap and allow Playwright users to access and assert on console messages with more convenience. Note that the proposed API is in no way suggested to replace the existing page.on('console') but rather complement it.

[vladkrasn]

Looking at trace files, it seems like Page automatically collects console errors even when not specifically asked to do so through page.on().

Would indeed be nice to just access the list of errors from Page object in code

[kettanaito]

My main pain point is accessing those logs (I've illustrated it in the example above). I want the list of console messages at a given point in time, not a listener to give me any messages at every point in time. Event-based APIs generally work poorly for assertions as you need to dance your way around their asynchronous nature.

[mxschmitt]

While the feature request sounds great, it can be solved in the user-land already:

// baseTest.ts
import { test as baseTest } from '@playwright/test';
export const test = baseTest.extend<{ consoleSpy: Map<string, string[]> }>({
  consoleSpy: async ({ context }, use) => {
    const messages = new Map<string, string[]>();
    context.on('console', message => {
      messages.set(message.type(), (messages.get(message.type()) ?? []).concat(message.text()));
    });
    await use(messages);
  },
});
export { expect } from '@playwright/test';

Your actual test (not the import which is different):

// example.spec.ts
import { test, expect } from './baseTest';

test('has title', async ({ page, consoleSpy }) => {
  await page.goto('https://example.com');
  await page.evaluate(() => console.log('hello', 'world'));
  expect(consoleSpy.get('log')).toEqual(['hello world']);
});

This will define a test fixture which listens to the console events, stores them in a map and makes them accessible to the tests.

[vladkrasn]

@mxschmitt What if I have a test that utilizes two different page fixtures? This thing will probably gather all console messages from both of them with no way to easily determine the source

[mxschmitt]

You could do e.g. this:

// baseTest.ts
import { BrowserContext, ConsoleMessage, Page, test as baseTest } from '@playwright/test';

class ConsoleSpy {
  private _messages: ConsoleMessage[] = [];
  constructor(context: BrowserContext) {
    context.on('console', message => this._messages.push(message));
  }
  forPage(page: Page) {
    return this._messages.filter(message => message.page() === page);
  }
}
export const test = baseTest.extend<{ consoleSpy: ConsoleSpy }>({
  consoleSpy: async ({ context }, use) => {
    await use(new ConsoleSpy(context))
  },
});
export { expect } from '@playwright/test';

// exampe.spec.ts
import { test, expect } from './baseTest';

test('has title', async ({ page, consoleSpy }) => {
  await page.goto('https://example.com');
  await page.evaluate(() => console.log('hello', 'world'));
  expect(consoleSpy.forPage(page).map(m => m.text())).toEqual(['hello world']);
});

The Fixture API makes it really flexible depending on your use case and requirements of your test.

[vladkrasn]

I can't figure out how to make it work with pattern described here https://playwright.dev/docs/auth#testing-multiple-roles-with-pom-fixtures (without the POM part, just bare page with preassigned storage state)

[mxschmitt]

You could do it like that. (There a lot of different ways of accomplishing it)

[yury-s]

Closing per the response above, feel free to open a new issue if it doesn't work.