ReferenceTests.jl
ReferenceTests.jl copied to clipboard
rework `@test_reference` messages - test msgs only on ci - drop lena
- rework
@infoin@test_reference: now includesrender(...)(more structured output for the end user, see below, works well with images), this helps CI in ensuring correct message order (io flush); - rework tests not to show spurious messages when working with
@test_referencemacro unlessCIenv var is set totrue: this was is very confusing when running] test ReferenceTests: only the tests summary should be shown to the end user, detailed are activated in CI or by the env varCI; -
drop
lenainruntests.jl: replaced withmonarchwhich is very similar (same format and size); - stop writing / deleting temporary test files in
test/references: if testing is interrupted, these temporaries are leftovers, and thus can land in a commit (usemktempdirinstead).
As an example, this job output (PR) is more readable than this one (master): most likely flush of stdout and stderr is de-synchronized.
PR
julia> ENV["CI"] = true # which is the default in `github` workflows / CI
pkg> test ReferenceTests
[...]
Test Summary: | Pass Total
world age issues | 1 1
┌ Info: Reference Test for "string1.txt" failed:
│ - DIFF ------------------------
│ "fintentionally wrobng to check that this message prints"
│ -------------------------------
│ reference = "[...]/test/references/string1.txt"
└ actual = "/tmp/jl_h48D23/string1.txt"
┌ Info: Reference Test for "string5.txt" failed:
│ - DIFF ------------------------
│ """
│ - This is a
│ + This is an incorrect
│ multiline string that does not end with a new line."""
│ -------------------------------
│ reference = "[...]/test/references/string5.txt"
└ actual = "/tmp/jl_79MGXW/string5.txt"
[...]
Testing ReferenceTests tests passed
julia> ENV["CI"] = false # which is the default on a local pc
pkg> test ReferenceTests
[...]
Test Summary: | Pass Total
world age issues | 1 1
Test Summary: | Pass Total
ReferenceTests | 49 49
Testing ReferenceTests tests passed
master
pkg> test ReferenceTests
[...]
┌ Info: Reference Test for "string1.txt" failed.
│ reference = "[...]/test/references/string1.txt"
└ actual = "/tmp/jl_egX948/string1.txt"
- DIFF ------------------------
"fintentionally wrobng to check that this message prints"
-------------------------------
┌ Info: Reference Test for "string5.txt" failed.
│ reference = "[...]/test/references/string5.txt"
└ actual = "/tmp/jl_qicFon/string5.txt"
- DIFF ------------------------
"""
- This is a
+ This is an incorrect
multiline string that does not end with a new line."""
-------------------------------
[...]
Testing ReferenceTests tests passed