typing icon indicating copy to clipboard operation
typing copied to clipboard

Conformance tests: Align automated and manual scoring

Open JelleZijlstra opened this issue 1 year ago • 1 comments

In #1678 I added an automated tool for scoring the conformance test suite, and #1682 and #1683 added some features to support edge cases.

Now I am working on going through the remaining files where the automated conformance judgment on a file differs from the manual one (open PRs: #1684, #1685, #1687, #1688, #1689, #1690, #1691).

I'll use this issue to track the remaining discrepancies (from conformance/src/unexpected_fails.py):

  • [x] mypy/dataclasses_usage.toml: Pass vs. Fail (#1684)
  • [x] mypy/directives_version_platform.toml: Pass vs. Fail (#1685)
  • [ ] mypy/namedtuples_define_functional.toml: Pass vs. Fail
  • [x] mypy/overloads_basic.toml: Pass vs. Fail (#1687)
  • [x] mypy/specialtypes_never.toml: Pass vs. Fail (#1688)
  • [x] mypy/typeddicts_alt_syntax.toml: Pass vs. Fail (#1689)
  • [x] mypy/typeddicts_type_consistency.toml: Pass vs. Fail (#1690)
  • [x] pyre/directives_reveal_type.toml: Unsupported vs. Pass
  • [x] pyre/namedtuples_define_functional.toml: Pass vs. Fail
  • [x] pyright/classes_override.toml: Pass vs. Fail (#1691)
  • [x] pyright/dataclasses_frozen.toml: Pass vs. Fail (#1699)
  • [x] pyright/dataclasses_slots.toml: Pass vs. Fail (#1699)
  • [x] pyright/dataclasses_transform_field.toml: Pass vs. Fail (#1699)
  • [x] pyright/dataclasses_transform_func.toml: Pass vs. Fail (#1699)
  • [x] pyright/generics_defaults.toml: Pass vs. Fail (#1700)
  • [x] pyright/literals_literalstring.toml: Pass vs. Fail (#1701)
  • [x] pyright/overloads_basic.toml: Pass vs. Fail (#1687)
  • [x] pyright/protocols_variance.toml: Pass vs. Fail (#1702)
  • [x] pyright/specialtypes_never.toml: Pass vs. Fail (#1688)
  • [x] pyright/tuples_unpacked.toml: Pass vs. Fail (#1703)
  • [x] pytype/dataclasses_usage.toml: Pass vs. Fail (#1684)
  • [x] pytype/directives_assert_type.toml: Pass vs. Fail (#1706)
  • [x] pytype/directives_version_platform.toml: Pass vs. Fail (#1685)
  • [x] pytype/generics_scoping.toml: Pass vs. Fail (#1704)
  • [x] pytype/typeddicts_inheritance.toml: Pass vs. Fail (#1705)

Any help is appreciated.

JelleZijlstra avatar Apr 06 '24 12:04 JelleZijlstra

The only remaining issue now is mypy/namedtuples_define_functional.toml. There the next mypy release should have full support (python/mypy#17247), which will remove the discrepancy.

JelleZijlstra avatar Jun 07 '24 22:06 JelleZijlstra