python
Common test suite for Python type checkers
It would be a good idea to have a common test suite for different Python type checkers like Mypy, PyCharm, PyType. By introducing it, we could ensure that our checkers have a common understanding of the type system.
The existing checkers detect various kinds of errors depending on the depth of code analysis and type inference. The idea is to use the common test suite just for testing basic scenarios every checker should understand according to PEP 484.
As a starting point, we could use a subset of Mypy tests and Mypy test format.
For example, a typical Mypy test looks like that:
[case testTupleAssignmentWithTupleTypes]
from typing import Tuple
t1 = None # type: Tuple[A]
t2 = None # type: Tuple[B]
t3 = None # type: Tuple[A, A]
t4 = None # type: Tuple[A, B]
t5 = None # type: Tuple[B, A]
t1 = t2 # E: Incompatible types in assignment (expression has type "Tuple[B]", variable has type "Tuple[A]")
t1 = t3 # E: Incompatible types in assignment (expression has type "Tuple[A, A]", variable has type "Tuple[A]")
t3 = t1 # E: Incompatible types in assignment (expression has type "Tuple[A]", variable has type "Tuple[A, A]")
t3 = t4 # E: Incompatible types in assignment (expression has type "Tuple[A, B]", variable has type "Tuple[A, A]")
t3 = t5 # E: Incompatible types in assignment (expression has type "Tuple[B, A]", variable has type "Tuple[A, A]")
# Ok
t1 = t1
t2 = t2
t3 = t3
t4 = t4
t5 = t5
class A: pass
class B: pass
[builtins fixtures/tuple.pyi]
There are several directives in square brackets like [case ...] for defining a test case, [builtins ...] for including minimalistic stubs for built-ins, and some others. The expected error messages are written using the # E: ... comments.
I would like to introduce these tests in the typing repo and maybe setup Travis CI for running them.
We wouldn't need to support [builtins ...], which specifies a fairly minimal test-only stub to use for a particular mypy test case. These tests could probably use the real stubs for builtins and other modules, since these would be more like acceptance tests rather than unit tests.
I'd suggest using whatever stubs each tool is shipped with. For mypy this would be typeshed, but other tools might use custom stubs for builtins, for example.
Also I would assume the error messages wouldn't have to be identical for the different tools.
Maybe we'll need a separate library for parsing such test files? Otherwise there would be a lot of hacky duplicate code -- mypy has its version of this parser (and it's evolving, e.g. https://github.com/python/mypy/pull/3347).
Or even better, we could have no parsing of tests at all by using YAML for them. YAML is easy to type and extendable so we could introduce additional attributes easily without breaking existing test runners.
An example using YAML:
- name: testTupleAssignmentWithTupleTypes
files:
- content: |
from typing import Tuple
t1 = None # type: Tuple[A]
t2 = None # type: Tuple[B]
t3 = None # type: Tuple[A, A]
t4 = None # type: Tuple[A, B]
t5 = None # type: Tuple[B, A]
t1 = t2 # E: Incompatible types in assignment (expression has type "Tuple[B]", variable has type "Tuple[A]")
t1 = t3 # E: Incompatible types in assignment (expression has type "Tuple[A, A]", variable has type "Tuple[A]")
t3 = t1 # E: Incompatible types in assignment (expression has type "Tuple[A]", variable has type "Tuple[A, A]")
t3 = t4 # E: Incompatible types in assignment (expression has type "Tuple[A, B]", variable has type "Tuple[A, A]")
t3 = t5 # E: Incompatible types in assignment (expression has type "Tuple[B, A]", variable has type "Tuple[A, A]")
# Ok
t1 = t1
t2 = t2
t3 = t3
t4 = t4
t5 = t5
class A: pass
class B: pass
Having a common parsing library for Mypy-specific format in Python wouldn't help PyCharm since our codebase runs on JVM (and Jython is too slow, unfortunately).
Parsing .test files is actually pretty simple -- most of the related code is about creating files related to test cases, cleaning them up and handling various special cases that wouldn't be needed for these tests.
One nice thing about the existing mypy test file format is that the source code isn't indented so it's easy to copy&paste between a source file and a test file. Not sure if that would be practical with YAML.
Still, it would require a parser and a some sort of a standard for the file format that cares about its extensibility. Having a block de-dented after copy&paste should be fairly easy in most text editors.
+1 for YAML from me, -0.5 for the current Mypy format.
@gvanrossum What do you think about the format for the tests? Should it be Mypy test format, YAML, or something else?
Hm. I worry that yaml indentation and python indentation will make things hard to follow or edit. But we may need to experiment to (dis)prove our concerns.
I'm going to implement a prototype with a simple YAML-based format and test runners for Mypy and PyCharm here at the PyCon sprints. Let's see if the resulting tests will be easy to read and edit.
