goblint
Three incremental tests failing with the cfg comparison
Currently three incremental test cases fail when the cfg comparison is activated. The first two throw a Not_found exception, for the third the result is not as expected.
- [ ] 03/02 reluctant-int-annotation
- [ ] 03/03 reluctant-int-annotation-dyn
- [ ] 13/05 race-call-remove
Since the cfg comparison often breaks when new features are introduced and the only the AST comparison is considered when adapting the incremental pre-processing for the solver, I would suggest to run the incremental tests also with the cfg comparison in the regular regression test runs.
Testing 03/02 precision-annotation/reluctant-int-annotation Status: 2 (exception)
- Timings:
- TOTAL 0.007 s
- parse 0.002 s
- convert to CIL 0.003 s
- mergeCIL 0.002 s
- compareCilFiles 0.001 s
- createCFG 0.001 s
- handle 0.001 s
- iter_connect 0.001 s
- computeSCCs 0.001 s
Testing 03/03 precision-annotation/reluctant-int-annotation-dyn Status: 2 (exception)
- Timings:
- TOTAL 0.008 s
- parse 0.003 s
- convert to CIL 0.003 s
- mergeCIL 0.002 s
- compareCilFiles 0.001 s
- createCFG 0.001 s
- handle 0.001 s
- iter_connect 0.001 s
- computeSCCs 0.001 s
03/02 failed!
Expected assert, but registered nothing on reluctant-int-annotation-dyn:10
__goblint_check(in == 17);
Expected fail, but registered nothing on reluctant-int-annotation-dyn:21
__goblint_check(a); // FAIL!
Expected assert, but registered nothing on reluctant-int-annotation-dyn:23
__goblint_check(a == 17);
03/03 failed!
Expected norace, but registered race on race-call-remove:7
g++; // NORACE (unique thread)
Expected norace, but registered race on race-call-remove:12
g++; // NORACE
13/05 failed!
3 test(s) failed: ["03/02 reluctant-int-annotation", "03/03 reluctant-int-annotation-dyn", "13/05 race-call-remove"]
Some of the incremental tests might have been written in a way that assumes AST comparison: for example, a change being made to a function with the purpose of reanalyzing the entire function from scratch.
You could still achieve this by setting the configuration option force-reanalyze. I would find that more transparent because I think that there should not be any hidden assumption regarding the kind of comparison.