Imported TypedDict class with 'total = False' requires all fields in variable construction

[theahura]

If I have two files, like so: test.py

from typing import TypedDict
import test2

class Foo(TypedDict, total=False):
  x: int
  y: int

x: Foo = {'x': 1}
x: test2.Foo = {'x': 1}

test2.py

from typing import TypedDict

class Foo(TypedDict, total=False):
  x: int
  y: int

I get the following error:

(env) soot@soot:~/code/soot/experimental/amol/pytype-test-partial$ pytype test.py 
Computing dependencies
Analyzing 1 sources with 1 local dependencies
ninja: Entering directory `.pytype`
[1/1] check test
FAILED: /home/soot/code/soot/experimental/amol/pytype-test-partial/.pytype/pyi/test.pyi 
/home/soot/code/soot/experimental/amol/pytype-test-partial/env/bin/python3 -m pytype.single --imports_info /home/soot/code/soot/experimental/amol/pytype-test-partial/.pytype/imports/test.imports --module-name test --platform linux -V 3.8 -o /home/soot/code/soot/experimental/amol/pytype-test-partial/.pytype/pyi/test.pyi --analyze-annotated --enable-typed-dicts --nofail --quick /home/soot/code/soot/experimental/amol/pytype-test-partial/test.py
File "/home/soot/code/soot/experimental/amol/pytype-test-partial/test.py", line 11, in <module>: Type annotation for x does not match type of assignment [annotation-type-mismatch]
  Annotation: test2.Foo(TypedDict)
  Assignment: Dict[str, int]
  
  TypedDict missing keys: y

For more details, see https://google.github.io/pytype/errors.html#annotation-type-mismatch

In other words, when I import a typeddict class and use it for a variable annotation, I have to provide all fields (even though total = False). That requirement is not enforced if the typeddict class is located in the same file.

Tested on 2022.04.26

[rchen152]

Well, that's slightly embarrassing! We never resolved this TODO to actually use the total keyword: https://github.com/google/pytype/blob/6afed8efc508c8d5a540efa48c5e435071f3cee8/pytype/pyi/classdef.py#L57.

[rchen152]

This is a little tricky because pytd.Class doesn't have anywhere to store keywords passed to the classdef: https://github.com/google/pytype/blob/0537dc266d09a7fcc7da069238e173addb55cdfe/pytype/pytd/pytd.py#L157. What we probably want to do is change the metaclass attribute to a keywords attribute that can store things like {'metaclass': Foo, 'total': True}, but that requires some fiddly release managing to avoid pickle-related breakages.

[theahura]

hey there! Any progress on this one? I've been able to get around this limitation using casting for now, but just ran into a bug where I was casting incorrectly :sweat_smile: would love to remove those casts

[theahura]

Just checking on this one again, somehow I'd thought this was resolved so was confused when I ran into this in a different place :sweat_smile:

[rchen152]

Sorry, haven't had time to look into this yet. I'm currently dealing with some 3.10 issues, but I should be able to work through a bit of our GitHub backlog after that.

[yixun-alpha]

hihi, ran into a somewhat related with total=False. As per the above issue, the following snippet works correctly:

from typing_extensions import TypedDict

class TestDict(TypedDict, total=False):
    a: int
    b: str

x: TestDict = {"a": 1, "b": "2"}
y: TestDict = {"a": 1}

> Success: no errors found

However, replacing the class definition with the functional definition:

from typing_extensions import TypedDict

TestDict = TypedDict("TestDict", {"a": int, "b": str}, total=False)

x: TestDict = {"a": 1, "b": "2"}
y: TestDict = {"a": 1}

incurs:

$ pytype main.py
Computing dependencies
Analyzing 1 sources with 0 local dependencies
ninja: Entering directory `.pytype'
[1/1] check main
FAILED: /home/yixun/code/py/pyProject1/.pytype/pyi/main.pyi 
/home/yixun/code/py/pyProject1/venv/bin/python -m pytype.single --imports_info /home/yixun/code/py/pyProject1/.pytype/imports/main.imports --module-name main --platform linux -V 3.7 -o /home/yixun/code/py/pyProject1/.pytype/pyi/main.pyi --analyze-annotated --nofail --quick /home/yixun/code/py/pyProject1/main.py
File "/home/yixun/code/py/pyProject1/main.py", line 6, in <module>: Type annotation for y does not match type of assignment [annotation-type-mismatch]
  Annotation: TestDict(TypedDict)
  Assignment: Dict[str, int]
  
  TypedDict missing keys: b

Using pytype 2022.08.30 on python 3.7.6

thanks for taking a look!

[theahura]

Hey hey, just checking in on the status of this one @rchen152 Also, will fixes only be deployed to py 3.10? or is this something that we can expect to be backwards compatible?