playwright-java icon indicating copy to clipboard operation
playwright-java copied to clipboard
verified publisher icon microsoft

[Bug]: Playwright-Junit lifecycle incompatible with maven retries 'surefire.rerunFailingTestsCount'

Open rtretyak opened this issue 11 months ago • 9 comments

Version

1.50.0

Steps to reproduce

  1. Have a basic Playwright test that fails
  2. Try running it with maven retries: ./mvnw test -Dtest=MyTest -Dsurefire.rerunFailingTestsCount=2

Expected behavior

System is able to retry the failing test, system can work with Playwright context (browser, page etc.)

Actual behavior

First attempt is a legit failure, subsequent rerun attempts have broken Playwright context

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 s <<< FAILURE! -- in fm.seller.onboardingweb.test.MyTest
[ERROR] fm.seller.onboardingweb.test.MyTest.test(Page) -- Time elapsed: 0.001 s <<< ERROR!
org.junit.jupiter.api.extension.ParameterResolutionException: Failed to resolve parameter [com.microsoft.playwright.Page page] in method [public void fm.seller.onboardingweb.test.MyTest.test(com.microsoft.playwright.Page)]: Playwright connection closed
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: com.microsoft.playwright.PlaywrightException: Playwright connection closed
        at com.microsoft.playwright.impl.PipeTransport.send(PipeTransport.java:50)
        at com.microsoft.playwright.impl.Connection.internalSendMessage(Connection.java:168)
        at com.microsoft.playwright.impl.Connection.sendMessageAsync(Connection.java:134)
        at com.microsoft.playwright.impl.ChannelOwner.sendMessageAsync(ChannelOwner.java:109)
        at com.microsoft.playwright.impl.SelectorsImpl.setTestIdAttributeName(SelectorsImpl.java:42)
        at com.microsoft.playwright.impl.SharedSelectors.lambda$setTestIdAttribute$2(SharedSelectors.java:72)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at com.microsoft.playwright.impl.SharedSelectors.setTestIdAttribute(SharedSelectors.java:72)
        at com.microsoft.playwright.impl.junit.ExtensionUtils.setTestIdAttribute(ExtensionUtils.java:53)
        at com.microsoft.playwright.impl.junit.BrowserContextExtension.getOrCreateBrowserContext(BrowserContextExtension.java:60)
        at com.microsoft.playwright.impl.junit.PageExtension.getOrCreatePage(PageExtension.java:51)
        at com.microsoft.playwright.impl.junit.PageExtension.resolveParameter(PageExtension.java:35)
        ... 2 more

[ERROR] Errors: 
[ERROR] fm.seller.onboardingweb.test.MyTest.test(Page)
[ERROR]   Run 1: MyTest.test:13
[ERROR]   Run 2: MyTest.test(Page) » ParameterResolution Failed to resolve parameter [com.microsoft.playwright.Page page] in method [public void fm.seller.onboardingweb.test.MyTest.test(com.microsoft.playwright.Page)]: Playwright connection closed
[ERROR]   Run 3: MyTest.test(Page) » ParameterResolution Failed to resolve parameter [com.microsoft.playwright.Page page] in method [public void fm.seller.onboardingweb.test.MyTest.test(com.microsoft.playwright.Page)]: Playwright connection closed

Additional context

Minimal test:

@UsePlaywright(OptionsFactory.class)
public class MyTest {
    @Test
    public void test(Page page) {
        page.navigate("https://google.com/");
        assert false;
    }
}

Environment

  • OS name: "mac os x", version: "15.3", arch: "aarch64", family: "mac"
  • Java version: 17.0.2, vendor: Oracle Corporation
  • Apache Maven 3.8.5 (3599d3414f046de2324203b78ddcf9b5e4388aa0)
  • playwright 1.50.0
  • Junit 5.11.4
  • surefire 3.5.2

rtretyak avatar Feb 20 '25 09:02 rtretyak

  • Have a basic Playwright test that fails

Please share a complete example. I was not able to reproduce it with a simple test.

@uchagani can you have a look?

yury-s avatar Feb 21 '25 18:02 yury-s

I cannot reproduce this. Please provide a complete example. Here is mine that works as expected:

import com.microsoft.playwright.Page;
import com.microsoft.playwright.assertions.PageAssertions;
import com.microsoft.playwright.junit.Options;
import com.microsoft.playwright.junit.OptionsFactory;
import com.microsoft.playwright.junit.UsePlaywright;
import org.junit.jupiter.api.Test;

import static com.microsoft.playwright.assertions.PlaywrightAssertions.assertThat;

@UsePlaywright(SampleTests.PlaywrightOptions.class)
public class SampleTests {
    public static class PlaywrightOptions implements OptionsFactory {
        @Override
        public Options getOptions() {
            return new Options().setBaseUrl("https://www.playwright.dev");
        }
    }

    @Test
    void testPageTitle(Page page) {
        page.navigate("/");
        assertThat(page).hasTitle("foo", new PageAssertions.HasTitleOptions().setTimeout(1000));
    }
}

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>playwright-java-sample</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.microsoft.playwright</groupId>
            <artifactId>playwright</artifactId>
            <version>1.50.0</version>
        </dependency>

        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>5.11.4</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <version>3.5.2</version>
                </plugin>
            </plugins>
        </pluginManagement>
    </build>
</project>

uchagani avatar Feb 21 '25 20:02 uchagani

pom:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>playwrightRetries</artifactId>
    <version>1.0-SNAPSHOT</version>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.4.2</version>
        <relativePath/>
    </parent>

    <properties>
        <maven.compiler.source>17</maven.compiler.source>
        <maven.compiler.target>17</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.5.2</version>
            </plugin>
        </plugins>
    </build>
    <dependencies>
        <dependency>
            <groupId>com.microsoft.playwright</groupId>
            <artifactId>playwright</artifactId>
            <version>1.50.0</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>
</project>

test:

import com.microsoft.playwright.Page;
import com.microsoft.playwright.junit.UsePlaywright;
import org.junit.jupiter.api.Test;

@UsePlaywright
public class MyTest {
    @Test
    public void test(Page page) {
        page.navigate("https://google.com/");
        assert false;
    }
}

command: mvn clean test -Dsurefire.rerunFailingTestsCount=1

output:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running MyTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.671 s <<< FAILURE! -- in MyTest
[ERROR] MyTest.test(Page) -- Time elapsed: 3.640 s <<< FAILURE!
java.lang.AssertionError
        at MyTest.test(MyTest.java:10)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)

[INFO] Running MyTest
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.004 s <<< FAILURE! -- in MyTest
[ERROR] MyTest.test(Page) -- Time elapsed: 0.002 s <<< ERROR!
org.junit.jupiter.api.extension.ParameterResolutionException: Failed to resolve parameter [com.microsoft.playwright.Page page] in method [public void MyTest.test(com.microsoft.playwright.Page)]: Playwright connection closed
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
Caused by: com.microsoft.playwright.PlaywrightException: Playwright connection closed
        at com.microsoft.playwright.impl.PipeTransport.send(PipeTransport.java:50)
        at com.microsoft.playwright.impl.Connection.internalSendMessage(Connection.java:168)
        at com.microsoft.playwright.impl.Connection.sendMessageAsync(Connection.java:134)
        at com.microsoft.playwright.impl.ChannelOwner.sendMessageAsync(ChannelOwner.java:109)
        at com.microsoft.playwright.impl.SelectorsImpl.setTestIdAttributeName(SelectorsImpl.java:42)
        at com.microsoft.playwright.impl.SharedSelectors.lambda$setTestIdAttribute$2(SharedSelectors.java:72)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
        at com.microsoft.playwright.impl.SharedSelectors.setTestIdAttribute(SharedSelectors.java:72)
        at com.microsoft.playwright.impl.junit.ExtensionUtils.setTestIdAttribute(ExtensionUtils.java:53)
        at com.microsoft.playwright.impl.junit.BrowserContextExtension.getOrCreateBrowserContext(BrowserContextExtension.java:60)
        at com.microsoft.playwright.impl.junit.PageExtension.getOrCreatePage(PageExtension.java:51)
        at com.microsoft.playwright.impl.junit.PageExtension.resolveParameter(PageExtension.java:35)
        ... 2 more

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR] MyTest.test(Page)
[ERROR]   Run 1: MyTest.test:10
[ERROR]   Run 2: MyTest.test(Page) » ParameterResolution Failed to resolve parameter [com.microsoft.playwright.Page page] in method [public void MyTest.test(com.microsoft.playwright.Page)]: Playwright connection closed
[INFO] 
[INFO] 
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE

rtretyak avatar Feb 21 '25 22:02 rtretyak

@uchagani I tried your project and I'm getting the same problem. The retry is not able to start properly:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running MyTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.565 s <<< FAILURE! -- in MyTest
[ERROR] MyTest.testPageTitle(Page) -- Time elapsed: 5.528 s <<< FAILURE!
org.opentest4j.AssertionFailedError: 
Page title expected to be: foo
Received: Fast and reliable end-to-end testing for modern web apps | Playwright

Call log:
  - Locator.expect with timeout 1000ms
  - waiting for locator(":root")
    5 × locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-hydrated="true" class="plugin-pages plugin-id-default" data-rh="lang,dir,class,data-has-hydrated">…</html>
      - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
        at com.microsoft.playwright.impl.AssertionsBase.expectImpl(AssertionsBase.java:74)
        at com.microsoft.playwright.impl.AssertionsBase.expectImpl(AssertionsBase.java:50)
        at com.microsoft.playwright.impl.AssertionsBase.expectImpl(AssertionsBase.java:42)
        at com.microsoft.playwright.impl.PageAssertionsImpl.hasTitle(PageAssertionsImpl.java:45)
        at MyTest.testPageTitle(MyTest.java:22)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)

[INFO] Running MyTest
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 s <<< FAILURE! -- in MyTest
[ERROR] MyTest.testPageTitle(Page) -- Time elapsed: 0.002 s <<< ERROR!
org.junit.jupiter.api.extension.ParameterResolutionException: Failed to resolve parameter [com.microsoft.playwright.Page arg0] in method [void MyTest.testPageTitle(com.microsoft.playwright.Page)]: Playwright connection closed
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
Caused by: com.microsoft.playwright.PlaywrightException: Playwright connection closed
        at com.microsoft.playwright.impl.PipeTransport.send(PipeTransport.java:50)
        at com.microsoft.playwright.impl.Connection.internalSendMessage(Connection.java:168)
        at com.microsoft.playwright.impl.Connection.sendMessageAsync(Connection.java:134)
        at com.microsoft.playwright.impl.ChannelOwner.sendMessageAsync(ChannelOwner.java:109)
        at com.microsoft.playwright.impl.SelectorsImpl.setTestIdAttributeName(SelectorsImpl.java:42)
        at com.microsoft.playwright.impl.SharedSelectors.lambda$setTestIdAttribute$2(SharedSelectors.java:72)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
        at com.microsoft.playwright.impl.SharedSelectors.setTestIdAttribute(SharedSelectors.java:72)
        at com.microsoft.playwright.impl.junit.ExtensionUtils.setTestIdAttribute(ExtensionUtils.java:53)
        at com.microsoft.playwright.impl.junit.BrowserContextExtension.getOrCreateBrowserContext(BrowserContextExtension.java:60)
        at com.microsoft.playwright.impl.junit.PageExtension.getOrCreatePage(PageExtension.java:51)
        at com.microsoft.playwright.impl.junit.PageExtension.resolveParameter(PageExtension.java:35)
        ... 2 more

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR] MyTest.testPageTitle(Page)
[ERROR]   Run 1: MyTest.testPageTitle:22 Page title expected to be: foo
Received: Fast and reliable end-to-end testing for modern web apps | Playwright

Call log:
  - Locator.expect with timeout 1000ms
  - waiting for locator(":root")
    5 × locator resolved to <html lang="en" dir="ltr" data-theme="light" data-has-hydrated="true" class="plugin-pages plugin-id-default" data-rh="lang,dir,class,data-has-hydrated">…</html>
      - unexpected value "Fast and reliable end-to-end testing for modern web apps | Playwright"
[ERROR]   Run 2: MyTest.testPageTitle(Page) » ParameterResolution Failed to resolve parameter [com.microsoft.playwright.Page arg0] in method [void MyTest.testPageTitle(com.microsoft.playwright.Page)]: Playwright connection closed
[INFO] 
[INFO] 
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE

Here is also my mvn version output:

 % mvn -version
Apache Maven 3.8.5 (3599d3414f046de2324203b78ddcf9b5e4388aa0)
Maven home: /opt/homebrew/Cellar/maven/3.8.5/libexec
Java version: 23, vendor: Homebrew, runtime: /opt/homebrew/Cellar/openjdk/23/libexec/openjdk.jdk/Contents/Home
Default locale: en_UA, platform encoding: UTF-8
OS name: "mac os x", version: "15.3", arch: "aarch64", family: "mac"

And I also tried with this with the same result:

 % ./mvnw -version
Apache Maven 3.8.5 (3599d3414f046de2324203b78ddcf9b5e4388aa0)
Maven home: /Users/user/.m2/wrapper/dists/apache-maven-3.8.5-bin/67203e94/apache-maven-3.8.5
Java version: 17.0.2, vendor: Oracle Corporation, runtime: /Users/user/Library/Java/JavaVirtualMachines/openjdk-17.0.2/Contents/Home
Default locale: en_UA, platform encoding: UTF-8
OS name: "mac os x", version: "15.3", arch: "aarch64", family: "mac"

rtretyak avatar Feb 22 '25 11:02 rtretyak

I'm not sure what could be causing this and I can't seem to reproduce it.

uchagani avatar Feb 23 '25 03:02 uchagani

I have also tried it on my home Windows PC which is completely independent from the previous environment and was able to reproduce it with your project.

mvn -version
Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
Maven home: E:\workspace\apache-maven-3.8.4
Java version: 23.0.2, vendor: Oracle Corporation, runtime: C:\Users\38097\.jdks\openjdk-23.0.2
Default locale: en_US, platform encoding: UTF-8
OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"

Let me know if there is any other debug information I can provide you with...

rtretyak avatar Feb 23 '25 16:02 rtretyak

Are you configuring your Java or maven in any specific way?

Can you check and post your settings.xml file (without any private info).

Are you running these tests from inside of an IDE or just the command line/terminal?

uchagani avatar Feb 23 '25 16:02 uchagani

Are you configuring your Java or maven in any specific way?

No, nothing special. I tried with multiple different Javas as you can see above. Not doing anyhting special to them. 1 was installed with homebrew, other 2 just a default installation by IntelliJ IDEA

settings.xml file

default one, just empty tags, no additional configurations

Are you running these tests from inside of an IDE or just the command line/terminal?

I'm running from terminal. Tried both in terminal inside IntelliJ and native terminal apps on windows/mac, same result.

rtretyak avatar Feb 23 '25 17:02 rtretyak

You can also reproduce it in the official playwright-java image.. docker run --rm -v "$(pwd):/app" -w /app mcr.microsoft.com/playwright/java:v1.50.0-noble mvn clean test -Dsurefire.rerunFailingTestsCount=1

rtretyak avatar Mar 03 '25 15:03 rtretyak