Expected failure

[kraigher]

I would like to add an expected failure mechanism to VUnit. An expected failure means the expectation is that the test case fails. Failing the test case will thus be an OK result. if the test case does not fail as expected it shall be a NOT_OK result.

Use case

-The use case is to add tests to the test suite that do not yet work but are planned to be working in the future. -The use case is not to check for a failure that is actually by specification supposed to happen in the DUT. For such use cases one should check for error or failure logs to the logging system using for example the mocking feature.

Proposal

In the Python API:

test_case.expect_failure()

To guard against that an expected failure is caused by an unintended failure I would also to have this in the VHDL code:

-- If failure occurs before it is an unwanted failure
begin_expected_failure;
-- Expected failures can only happen in-between here
end_expected_failure;
-- If failure occurs after it is an unwanted failure

If for some reason the user does not want this extra safety guards it can be disabled in Python via:

test_case.expect_failure(guarded=False)

It might also be convenient to add comment pragmas to the vhdl code to indicate that failure is expected:

if run("Test that does not work yet") then -- vunit: expect_failure
    begin_expected_failure;
    stuff_that_fails;
    end_expected_failure;
    -- The expected failure comment pragma is valid until the next if run("...") 
end if;

Unguarded:

if run("Test that does not work yet") then -- vunit: expect_failure unguarded
    stuff_that_fails;
end if;

In case of test bench without test cases or if all test cases should fail:

test_runner_setup(runner, runner_cfg); -- vunit: expect_failure

Reporting

For the test report the expected failures there are a number of alternatives:

• Alternative 1 - No distinction
  • Write pass in green text for expected failure that happened
  • Write fail in red text for expected failure that did not happen.
• Alternative 2 - With distinction
  • Write fail_ok in green text for an expected failure that happened
  • Write fail in red text for an expected failure that did not happen

Post check

Regarding the post_check feature it is normally not executed on a failure. If an expected failure was not present in the VHDL code the post_check function should still run and can itself trigger an expected failure. The use case would be that the post_check function checks output data and when the test is not working yet this can cause an expected failure.

[svenka3]

I like this. It is similar to Mock frameworks available in Java etc. We have a similar feature named log-predictor in Go2UVM

Just curious - why not name it mock? Or something similar with prefix/suffix

[kraigher]

@svenka3 The use case is different than mocking and I state that in the use case section. The purpose of expected failure is for tests that do not yet pass but eventually should.

When performing negative tests of things that should fail by specification we have added a mocking feature to our logging framework in the development branch intended for VUnit 3.0.

mock(queue_logger, error);
value := pop(queue); -- We want to test that pop:ing from the empty queue causes failure
check_only_log(queue_logger, failure, "Pop from empty queue");
unmock(queue_logger);

So in summary:

• Expected failures are for things that do not yet work but eventually should.
• Mocking is for negative testing of failures that should occur by specification.

[LarsAsplund]

@kraigher Some initial thoughts

• We should use vunit_pragma rather than just vunit to be consistent with previous pragmas.
• I prefer having a distinction between a normal pass and an expected failure. It's easier to forget about the test case not yet supported if the test report is all pass. What should the color be?
• Need to consider how copy and paste mistakes should be handled. For example, one of begin_expected_failure and end_expected_failure are used but not the other. Same call made twice in a row...
• Pros/cons of using a special run call rather than a pragma

[kraigher]

I would like to change from vunit-pragma: to vunit: since it looks better and is shorter. The old would Still be supported.

[LarsAsplund]

I would like to change from vunit-pragma: to vunit: since it looks better and is shorter. The old would Still be supported.

That make sense since "pragma" doesn't add any readability. However, I would prefer expect_to_fail to prevent that someone thinks it must fail with log level failure.

Also, I prefer

if expect_to_fail("Test that does not work yet") then

over using pragmas. It's even shorter and it's as easy to parse but more importantly

• The solution would be more consistent if not using a mix of pragmas and subprogram calls
• By keeping VHDL in the loop we avoid closing the door for what can be done in that area. For example for those that have reasons to use VUnit without the Python test runner

[kraigher]

1. Sure we can have expect_to_fail as a VHDL-function but it would not work for test benches without test cases. The use could always create a test case though.

2. For reporting I think I also prefer fail_ok or xfail in maybe yellow or something else than green or red as such as a test case is a TODO that is intended to be fixed. Currently a skipped test is yellow with skip text. Skipped tests only happen in case of same_sim tests where one of the test fail. Maybe we want to have support for the user to explicitly skip tests in the future.

3. Maybe we should leave the possibility open to use this mechanism for failures that are supposed to happen by specification for cases where the mocking feature of the logging system could not be used. In that case the only difference would be the intention of the user and in such cases it makes sense to write pass in green text. In such cases the user might also want to check that some output message was written from the simulator as a failure reason. Even if this is not added at the same time as the expeced failure mechanism we have to think about a terminology to distinguish between them.

[svenka3]

@kraigher - that sounds interesting- will Explore the mocking feature soon - is that language neutral - meaning available for Verilog/SV as well?

Thanks Srini

[kraigher]

The mocking feature is part of our VHDL logging library. We do not provide any logging library for Verilog.

[LarsAsplund]

@kraigher

Sure we can have expect_to_fail as a VHDL-function but it would not work for test benches without test cases. The use could always create a test case though.

There's also the case where you expect all test cases to fail. Still, I'm ok with having a restriction that you need test cases to use this functionality.

For reporting I think I also prefer fail_ok or xfail in maybe yellow or something else than green or red as such as a test case is a TODO that is intended to be fixed. Currently a skipped test is yellow with skip text. Skipped tests only happen in case of same_sim tests where one of the test fail. Maybe we want to have support for the user to explicitly skip tests in the future.

Yes, I think we need a color. Otherwise it will be easy to miss these special cases when looking quickly at the report. Why not make all exceptions from the normal pass/fail result yellow.

In addition to skip we should also make sure that our approach would work for conditional skip and expect_to_fail

Maybe we should leave the possibility open to use this mechanism for failures that are supposed to happen by specification for cases where the mocking feature of the logging system could not be used. In that case the only difference would be the intention of the user and in such cases it makes sense to write pass in green text. In such cases the user might also want to check that some output message was written from the simulator as a failure reason. Even if this is not added at the same time as the expeced failure mechanism we have to think about a terminology to distinguish between them.

That's a complicating factor but a valid use case. The VUnit test suite used for the VHDL standard packages would be an example. We can't mock assert

Even if "expected to fail" is used out there it's not a very good name if we want to distinguish between the two cases. I think we should so I'm open to introducing new names.

[joshrsmith]

Would an expected use case of this functionality also cover expected failures (by specification) in the case of implementation code that has a failed assertion? For example: you have an assertion in an implementation file that checks that some combination of generics is not used. You want to test that when the "bad" combination of generics is specified that the test fails (as expected).

For the implementation file, having an assertion fail can be helpful to stop synthesis from running in the case that a bad combination of generics is used. This is an additional guard over needing to read log messages.

[LarsAsplund]

@joshrsmith It isn't part of the stated use case but I think it should be. Assert statements can't be mocked (yet) but your use case is one where asserts should be used over VUnit checks. I do exactly the same thing ever since Vivado got this support.

[freyn42]

Is there any news for this feature? In particular I would be interested in the possibility to test an assert statement.

[jobtijhuis]

I would also like this feature. It can be useful to disable tests temporarily in the scripting, especially when certain tests don't work with certain simulators. I don't know of a way to easily do this inside the VHDL.