ReferenceTests.jl icon indicating copy to clipboard operation
ReferenceTests.jl copied to clipboard

rework `@test_reference` messages - test msgs only on ci - drop lena

Open t-bltg opened this issue 3 years ago • 0 comments

  • rework @info in @test_reference: now includes render(...) (more structured output for the end user, see below, works well with images), this helps CI in ensuring correct message order (io flush);
  • rework tests not to show spurious messages when working with @test_reference macro unless CI env var is set to true: this was is very confusing when running ] test ReferenceTests: only the tests summary should be shown to the end user, detailed are activated in CI or by the env var CI;
  • drop lena in runtests.jl: replaced with monarch which is very similar (same format and size);
  • stop writing / deleting temporary test files in test/references: if testing is interrupted, these temporaries are leftovers, and thus can land in a commit (use mktempdir instead).

As an example, this job output (PR) is more readable than this one (master): most likely flush of stdout and stderr is de-synchronized.

PR

julia> ENV["CI"] = true  # which is the default in `github` workflows / CI

pkg> test ReferenceTests
[...]
Test Summary:    | Pass  Total
world age issues |    1      1
┌ Info: Reference Test for "string1.txt" failed:
│ - DIFF ------------------------
│ "fintentionally wrobng to check that this message prints"
│ -------------------------------
│   reference = "[...]/test/references/string1.txt"
└   actual = "/tmp/jl_h48D23/string1.txt"
┌ Info: Reference Test for "string5.txt" failed:
│ - DIFF ------------------------
│ """
│ - This is a
│ + This is an incorrect
│   multiline string that does not end with a new line."""
│ -------------------------------
│   reference = "[...]/test/references/string5.txt"
└   actual = "/tmp/jl_79MGXW/string5.txt"
[...]
     Testing ReferenceTests tests passed

julia> ENV["CI"] = false  # which is the default on a local pc

pkg> test ReferenceTests
[...]
Test Summary:    | Pass  Total
world age issues |    1      1
Test Summary:  | Pass  Total
ReferenceTests |   49     49
     Testing ReferenceTests tests passed 

master

pkg> test ReferenceTests
[...]
┌ Info: Reference Test for "string1.txt" failed.
│   reference = "[...]/test/references/string1.txt"
└   actual = "/tmp/jl_egX948/string1.txt"
- DIFF ------------------------
"fintentionally wrobng to check that this message prints"
-------------------------------
┌ Info: Reference Test for "string5.txt" failed.
│   reference = "[...]/test/references/string5.txt"
└   actual = "/tmp/jl_qicFon/string5.txt"
- DIFF ------------------------
"""
- This is a
+ This is an incorrect
  multiline string that does not end with a new line."""
-------------------------------
[...]
     Testing ReferenceTests tests passed 

t-bltg avatar Aug 10 '22 18:08 t-bltg