realm
JUnit Reporter Missing Test Count
New Issue Checklist
- [x] I've Updated SwiftLint to the latest version.
- [x] I've searched for existing GitHub issues.
Bug Description
Coming out of PR #4725 updates were made to enhance the JUnit reporting with additional details that are necessary for some pipeline engines; specifically Bitbucket.
Unfortunately, one aspect of that PR was seemingly dropped, namely the "total tests count" which we're finding appears to be a requirement for Bitbucket ingestion in particular. For example, we're getting this message as Bitbucket's JUnit parsing logic:
Merged test suites, total number tests is 0, with 0 failures and 2 errors.
While it properly detects the presence of failures/errors per that referenced PR's changes; it's detecting 0 tests which we believe it causing them to disable all their UI integrations for reporting from this file.
The JUnit Complete Reference Example shows this as simply adding a "tests" attribute to both "testsuites" and "testsuite" XML nodes.
Additionally, I suspect the "testsuite" XML nodes need a "name" property for effective rendering. Most other tools I've seen (e.g. eslint in the Javascript world) would populate this simply as the filename though depending on how Swiftlint marshals this file maybe the linting rule is more appropriate or even just a single "SwiftLint" name if everything is marshalled under a single "testsuite".
Environment
- SwiftLint version: 0.59.1
- Xcode version: 16.4 Build 16F6
- Installation method used: Homebrew
- Configuration file:
disabled_rules:
- file_length
- function_body_length
- line_length
- type_body_length
opt_in_rules:
- conditional_returns_on_newline
- contains_over_filter_count
- contains_over_filter_is_empty
- contains_over_first_not_nil
- contains_over_range_nil_comparison
- empty_collection_literal
- empty_string
- explicit_init
- fatal_error_message
- first_where
- flatmap_over_map_reduce
- identical_operands
- implicit_return
- joined_default_parameter
- last_where
- legacy_multiple
- let_var_whitespace
- multiline_function_chains
- multiline_literal_brackets
- object_literal
- optional_enum_case_matching
- prefer_self_in_static_references
- prefer_self_type_over_type_of_self
- prefer_zero_over_explicit_init
- reduce_into
- redundant_nil_coalescing
- redundant_type_annotation
- return_value_from_void_function
- sorted_first_last
- static_operator
- toggle_bool
- trailing_closure
- unavailable_function
- untyped_error_in_catch
- xct_specific_matcher
conditional_returns_on_newline:
if_only: true
opening_brace:
ignore_multiline_statement_conditions: true
identifier_name:
min_length:
warning: 0
error: 0
max_length:
warning: 9999
error: 9999
unallowed_symbols_severity: warning
validates_start_with_lowercase: warning
type_name:
max_length:
warning: 60
Are you using nested configurations? If so, paste their relative paths and respective contents.
Not applicable
The problem is that there is no "total number of test". So what would be your proposal to put there?
However, there are few other details that could be added, like the name you suggested or the line number that's currently missing.
So I suspect there could be two ways.
The way ESLint in the JavaScript world appears to do it; they create a "testsuite" for each "testcase" failure reported, total being set to "1" then. Then the aggregated total on "testsuites" is simply the sum.
Or you could just have your current single testsuite be the sum of "testcase" failures.
The net result is the same, it appears as if the number of tests run is dynamic and effectively it's like the linter only ran the exact tests that it needs to find the exact failures it found. From a pipeline standpoint, this is typically fine since realistically the number of tests the linter is actually running = lines of code * linting rules and that metric isn't terribly useful.