microsoft
[Feature]: Add Configuration Option to Mark Test Suite as Failed When Containing Flaky Tests
🚀 Feature Request
Feature Description
Currently, Playwright supports a "retries" configuration where tests that initially fail but subsequently pass are marked as "flaky". When such flaky tests exist, the overall test suite is considered successful. I propose adding a configuration option that allows marking the entire test suite as failed if it contains any flaky tests.
Proposed Solution
Introduce a new configuration parameter, such as failOnFlakyTests, which when set to true, will cause the entire test suite to fail if any tests are marked as flaky. This will help teams differentiate between suites with consistent test failures and those with flaky tests, promoting more robust testing practices.
Impact
This enhancement will allow developers and QA teams to enforce stricter quality controls by treating flaky tests as critical errors, ensuring that they receive immediate attention.
Thank you for considering this feature.
Example
{
"failOnFlakyTests": true,
"retries": 2
}
Motivation
This feature would be beneficial for maintaining high code quality in continuous integration (CI) environments. By enabling this option, teams can ensure that flaky tests are highlighted and addressed promptly, rather than allowing them to degrade the reliability of the test suite over time.
Hi there, would it be possible for myself and https://github.com/BJSS-russell-pollock to be assigned to this? We're working on it at the moment and should have a PR for review next week.
