zig icon indicating copy to clipboard operation
zig copied to clipboard

lib: add panic test runner

Open matu3ba opened this issue 3 years ago • 2 comments

Panic test runner starts itself as child process with the test function index representing a test block. The custom panic handler compares a global previously written string. If the panic message is expected, the success count is increased and the next test block is run. On failure the parent process prints collected results, before it terminates.

This means, that only one @panic is possible within a test block and that no follow-up code after the @panic in the test block can be run.

Depends on #14152. Closes #1356.

matu3ba avatar Jan 17 '23 20:01 matu3ba

TODO:

  • [x] As far as I understand, signal handling code by Kernel does not lead to panic being called. Clarify desired semantics: Do we want to add the complexity to check "all exit code paths" from a program once "expectPanic" has been called? A: Document all global state under our control in test runner.
  • [x] panic testing requires process spawn due to noreturn of custom panic handler or "recovery code" by the user: Clarify, if we should add extra complexity to compiler to provide enable non-spawning ones to use panic checks (requires panic to be returnable) briefly mentioned here https://github.com/ziglang/zig/issues/5917#issuecomment-1385994191:
- 1. use user provided "restoring to safe process state" code + longjmp
- 2. add to all functions under test a quasi-reserved + implicitly added (propagating) error.panic and let user deal with "restoring to safe process state"

A: after thinking about it for some time: No.

  • [x] cleanup of code
  • [x] Document process spawning + IPC as requirement to keep things simple and test behavior = code behavior.
  • [x] Document all global OS state under our control in test runner.
  • [x] Think of and document panic test naming convention
  • [ ] test runner and build runner coordinate over a protocol via IPC, so implement it
  • [ ] Think of + document best practice for code reusage: Do not use @panic, unless your code path is strictly intended for non-embedded, because downstream users can not efficiently test error handling of their code on embedded devices. If you must use @panic or there is no reasonable alternative, write sufficient tests and documentation.

Separate PR[s]:

  • [ ] os.posix for the signal mask(s) to clarify that they are not unifiable
  • [ ] fix default implementation of panic handler
  • [ ] make running panic signal safe

matu3ba avatar Jan 17 '23 20:01 matu3ba

Rust does things abit differently with macro-annotation how the test is run and then (https://github.com/rust-lang/rust/blob/65d2f2a5f9c323c88d1068e8e90d0b47a20d491c/library/test/src/lib.rs#L342):

    let run_strategy = if opts.options.panic_abort && !opts.force_run_in_process {
        RunStrategy::SpawnPrimary
    } else {
        RunStrategy::InProcess
    };

However, I prefer the approach to put things into a different package, so things dont even compile if one does not opt-in into process spawning as dependency.

matu3ba avatar Jan 19 '23 10:01 matu3ba

Superseded by #15991, because the spawned process can not be handled by the server.

matu3ba avatar Jun 09 '23 23:06 matu3ba