exercism
non-exhaustive tests
As explained here #80 there is a way to write tests that is not exhaustive enough. As an example, if we have a simple program like:
add(X,Y,Z) :-
Z >= X+Y.
A test like:
?- add(1,2,3).
true.
succeeds. When is obviously not correct. nevertheless if we do this:
?- add(1,2,X).
ERROR: Arguments are not sufficiently instantiated
ERROR: In:
ERROR: [9] _2658>=1+2
ERROR: [7] <user>
ERROR:
ERROR: Note: some frames are missing due to last-call optimization.
ERROR: Re-run your program in debug mode (:- debug.) to get more detail.
throws an error. In this case, this predicate should be tested like so add(1,2,X), x =:= 3.
This is a particular characteristic of Prolog, and that is why is particularly confusing.
This are all tests that have this flaw:
- [x] anagram
- [x] binary
- [X] complex-numbers
- [x] grains
- [x] hamming
- [x] nucleotide-count
- [x] pascals-triangle
- [x] rna-transcription
- [x] spiral-matrix
- [x] sum-of-multiples
- [ ] triangle
If you submit a PR to fix one of them, please refer to this issue. And if you find another exercise with this problem add a comment here.
I am working on the triangle problem's tests, and at the same time referencing https://github.com/exercism/problem-specifications/blob/master/exercises/triangle/canonical-data.json to make the tests congruent to the canonical tests. In this test I am concern about how the example implementation limits itself to a single answer because the test days proposes the following test case:
{
"description": "equilateral triangles are also isosceles",
"property": "isosceles",
"input": {
"sides": [4, 4, 4]
},
"expected": true
},
which if we test for explicitly triangle(4,4,4, "isosceles") returns true, but under this proposed change with the example solution triangle(4, 4, 4, Result), Result == "isosceles" returns false. If I just remove the ! from the equilateral predicate, then it will succeed with a choicepoint.
Is there a way to accept the test with a choicepoint without raising a warning? Or should the test be changed to findall(R, triangle(4,4,4, R), Rs), member("isosceles", Rs).
I think we should aim to be conformant to the canonical tests, so this requires an interface change somewhere. Thoughts?
I think it all depends on how we want to view the query. triangle(+SideA, +SideB, +SideC, +Type) or triangle(+SideA, +SideB, +SideC, -Type) or triangle(+SideA, +SideB, +SideC, ?Type). If we want a deterministic answer (which may be the purest prolog solution), then maybe we shouldn't fix it.
(edit) Had more time to think/read. Changed my mind.
I think it all depends on how we want to view the query. triangle(+SideA, +SideB, +SideC, +Type) or triangle(+SideA, +SideB, +SideC, -Type) or triangle(+SideA, +SideB, +SideC, ?Type). If we want a deterministic answer (which may be the purest prolog solution), then maybe we shouldn't fix it.
(edit) Had more time to think/read. Changed my mind.
@neenjaw What do you suggest we do with triangle?
I haven't really thought about this issue for a long time, but I lean towards the problem set being written in such a way to promote deterministic solutions. I think those are better supported from the unit test library (if I recall correctly).
https://github.com/exercism/prolog/pull/289 Just linking this here for triangle.
