flow icon indicating copy to clipboard operation
flow copied to clipboard

Flaky tests across Flow modules

Open lycoris106 opened this issue 5 months ago • 1 comments

Description of the bug

The test failures in several modules were found with NonDex, which explores non-determinism in tests. These tests can cause test failure under different JVMs, etc.

An example of failing test is: In ./flow-plugins/flow-maven-plugin, the test com.vaadin.flow.plugin.maven.CleanFrontendMojoTest#should_notRemoveNpmPackageLockFile_hilla can fail when the order of nondeterministic data is shuffled.

I have fixed some of the flaky tests and will soon submit multiple PRs for each module. The detailed root cause for each test and fix will be explained in each PR message.

Expected behavior

The test results should be independent of the order of non-deterministic methods, including:

  • HashMap iteration
  • Reflection methods like getDeclaredFields()
  • and more

Minimal reproducible example

Build the module and run tests with NonDex, for example:

mvn edu.illinois:nondex-maven-plugin:2.2.1:nondex -pl flow-plugins/flow-maven-plugin \
    -Dtest=com.vaadin.flow.plugin.maven.CleanFrontendMojoTest#should_notRemoveNpmPackageLockFile_hilla \
    -Djacoco.skip -Drat.skip -Dpmd.skip -Denforcer.skip

Versions

  • Vaadin / Flow version: 25.0 (main branch)
  • Java version:
openjdk 21.0.8 2025-07-15
OpenJDK Runtime Environment (build 21.0.8+9-Ubuntu-0ubuntu124.04.1)
OpenJDK 64-Bit Server VM (build 21.0.8+9-Ubuntu-0ubuntu124.04.1, mixed mode, sharing)
  • OS version: Ubuntu 24.04.3 LTS
  • Browser version (if applicable):
  • Application Server (if applicable):
  • IDE (if applicable):

lycoris106 avatar Nov 18 '25 10:11 lycoris106

@lycoris106 thanks for the issue and contributions to codebase 👍

Good to know about this tool as well!

mshabarov avatar Nov 18 '25 15:11 mshabarov