pytype
pytype copied to clipboard
google
non deterministic "Can't find module [import-error]"
Hi, I am checking my project with pytype and I am experiencing an error that happens non-deterministically, even resetting the caches happens about 30% of the attempts
project: https://github.com/andrew-ld/LL-mtproto commit: 6d4c0bac3575061710bf4601772f19825e69b9af python version: Python 3.10.6 pytype version: 2022.08.03
(venv-client) [andrew@arch ll_mtproto]$ python3 -m pytype .
Computing dependencies
Analyzing 27 sources with 0 local dependencies
ninja: Entering directory `.pytype'
[5/37] check byteutils
FAILED: /home/andrew/PycharmProjects/llmtproto/ll_mtproto/.pytype/pyi/byteutils.pyi
/home/andrew/venv-client/bin/python3 -m pytype.single --imports_info /home/andrew/PycharmProjects/llmtproto/ll_mtproto/.pytype/imports/byteutils.imports --module-name byteutils --platform linux -V 3.10 -o /home/andrew/PycharmProjects/llmtproto/ll_mtproto/.pytype/pyi/byteutils.pyi --analyze-annotated --nofail --quick /home/andrew/PycharmProjects/llmtproto/ll_mtproto/tl/byteutils.py
File "/home/andrew/PycharmProjects/llmtproto/ll_mtproto/tl/byteutils.py", line 8, in <module>: Can't find module 'typed'. [import-error]
For more details, see https://google.github.io/pytype/errors.html#import-error
ninja: build stopped: subcommand failed.
Leaving directory '.pytype'
I have also been hitting this recently, though I can reproduce it with pytype 2019.12.06 and with importlab 0.2.1 so it does not appear to be a regression. Setting PYTHONHASHSEED will make pytype fail/succeed consistently depending on the value. I can hit it more reliably when running pytype as python -m pytype instead of the script.
Minimal reproducer (needs PyGObject) a.py:
from b import x
b.py:
from gi.repository import GLib
x = 0
Bad output with -v2
$ PYTHONHASHSEED=1234 python -m pytype -v2 a.py b.py
INFO No config file specified, and no [pytype] section in setup.cfg. Using default configuration.
INFO Configuration:
exclude = set()
inputs = {'/home/cebtenzzre/src/bugs/pytype/issue1271/a.py', '/home/cebtenzzre/src/bugs/pytype/issue1271/b.py'}
<snip>
Computing dependencies
INFO Source tree:
source: /home/cebtenzzre/src/bugs/pytype/issue1271/a.py
deps:
/home/cebtenzzre/src/bugs/pytype/issue1271/b.py
source: /home/cebtenzzre/src/bugs/pytype/issue1271/b.py
deps:
/home/cebtenzzre/src/bugs/pytype/issue1271/<gi.repository.GLib>
source: /home/cebtenzzre/src/bugs/pytype/issue1271/<gi.repository.GLib>
INFO ------------- Starting pytype run. -------------
INFO infer command: /usr/bin/python -m pytype.single --imports_info $imports --module-name $module --platform linux -V 3.10 -o $out --no-report-errors --nofail --quick $in
INFO check command: /usr/bin/python -m pytype.single --imports_info $imports --module-name $module --platform linux -V 3.10 -o $out --analyze-annotated --nofail --quick $in
INFO generate default: System module gi.repository
INFO check b
imports: /home/cebtenzzre/src/bugs/pytype/issue1271/.pytype/imports/b.imports
deps: ()
output: /home/cebtenzzre/src/bugs/pytype/issue1271/.pytype/pyi/b.pyi
INFO check a
imports: /home/cebtenzzre/src/bugs/pytype/issue1271/.pytype/imports/a.imports
deps: ('/home/cebtenzzre/src/bugs/pytype/issue1271/.pytype/pyi/b.pyi',)
output: /home/cebtenzzre/src/bugs/pytype/issue1271/.pytype/pyi/a.pyi
Analyzing 2 sources with 0 local dependencies
ninja: Entering directory `.pytype'
[1/2] /usr/bin/python -m pytype.single --imports_info /home/cebtenzzre/src/bugs/pytype/issue1271/.pytype/imports/b.imports --module-name b --platform linux -V 3.10 -o /home/cebtenzzre/src/bugs/pytype/issue1271/.pytype/pyi/b.pyi --analyze-annotated --nofail --quick /home/cebtenzzre/src/bugs/pytype/issue1271/b.py
FAILED: /home/cebtenzzre/src/bugs/pytype/issue1271/.pytype/pyi/b.pyi
/usr/bin/python -m pytype.single --imports_info /home/cebtenzzre/src/bugs/pytype/issue1271/.pytype/imports/b.imports --module-name b --platform linux -V 3.10 -o /home/cebtenzzre/src/bugs/pytype/issue1271/.pytype/pyi/b.pyi --analyze-annotated --nofail --quick /home/cebtenzzre/src/bugs/pytype/issue1271/b.py
File "/home/cebtenzzre/src/bugs/pytype/issue1271/b.py", line 1, in <module>: Can't find module 'gi.repository'. [import-error]
For more details, see https://google.github.io/pytype/errors.html#import-error
ninja: build stopped: subcommand failed.
Leaving directory '.pytype'
Good output with -v2
$ PYTHONHASHSEED=1234 pytype -v2 a.py b.py
INFO No config file specified, and no [pytype] section in setup.cfg. Using default configuration.
INFO Configuration:
exclude = set()
inputs = {'/home/cebtenzzre/src/bugs/pytype/issue1271/a.py', '/home/cebtenzzre/src/bugs/pytype/issue1271/b.py'}
<snip>
Computing dependencies
INFO Source tree:
source: /home/cebtenzzre/src/bugs/pytype/issue1271/a.py
deps:
/home/cebtenzzre/src/bugs/pytype/issue1271/b.py
source: /home/cebtenzzre/src/bugs/pytype/issue1271/b.py
deps:
/usr/lib/python3.10/site-packages/gi/repository/__init__.py
source: /usr/lib/python3.10/site-packages/gi/repository/__init__.py
INFO ------------- Starting pytype run. -------------
INFO infer command: /usr/bin/python -m pytype.single --imports_info $imports --module-name $module --platform linux -V 3.10 -o $out --no-report-errors --nofail --quick $in
INFO check command: /usr/bin/python -m pytype.single --imports_info $imports --module-name $module --platform linux -V 3.10 -o $out --analyze-annotated --nofail --quick $in
INFO generate default: System module gi.repository.__init__
INFO check b
imports: /home/cebtenzzre/src/bugs/pytype/issue1271/.pytype/imports/b.imports
deps: ()
output: /home/cebtenzzre/src/bugs/pytype/issue1271/.pytype/pyi/b.pyi
INFO check a
imports: /home/cebtenzzre/src/bugs/pytype/issue1271/.pytype/imports/a.imports
deps: ('/home/cebtenzzre/src/bugs/pytype/issue1271/.pytype/pyi/b.pyi',)
output: /home/cebtenzzre/src/bugs/pytype/issue1271/.pytype/pyi/a.pyi
Analyzing 2 sources with 0 local dependencies
ninja: Entering directory `.pytype'
[1/2] /usr/bin/python -m pytype.single --imports_info /home/cebtenzzre/src/bugs/pytype/issue1271/.pytype/imports/b.imports --module-name b --platform linux -V 3.10 -o /home/cebtenzzre/src/bugs/pytype/issue1271/.pytype/pyi/b.pyi --analyze-annotated --nofail --quick /home/cebtenzzre/src/bugs/pytype/issue1271/b.py
[2/2] /usr/bin/python -m pytype.single --imports_info /home/cebtenzzre/src/bugs/pytype/issue1271/.pytype/imports/a.imports --module-name a --platform linux -V 3.10 -o /home/cebtenzzre/src/bugs/pytype/issue1271/.pytype/pyi/a.pyi --analyze-annotated --nofail --quick /home/cebtenzzre/src/bugs/pytype/issue1271/a.py
Leaving directory '.pytype'
Success: no errors found
Bad runs generate __pycache__/b.cpython-310.pyc, good runs don't. This is where that happens:
Stack trace
Traceback (most recent call first):
<built-in method open of module object at remote 0x7ffff75c7380>
<built-in method open_code of module object at remote 0x7ffff75c7380>
File "<frozen importlib._bootstrap_external>", line 1073, in get_data
File "<frozen importlib._bootstrap_external>", line 975, in get_code
File "<frozen importlib._bootstrap_external>", line 879, in exec_module
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
<built-in method __import__ of module object at remote 0x7ffff7580590>
File "/usr/lib/python3.10/importlib/util.py", line 94, in find_spec
parent = __import__(parent_name, fromlist=['__path__'])
File "/home/cebtenzzre/.local/lib/python3.10/site-packages/importlab/import_finder.py", line 101, in _resolve_import_versioned
spec = importlib.util.find_spec(name)
File "/home/cebtenzzre/.local/lib/python3.10/site-packages/importlab/import_finder.py", line 114, in _resolve_import
return _resolve_import_versioned(name)
File "/home/cebtenzzre/.local/lib/python3.10/site-packages/importlab/import_finder.py", line 130, in resolve_import
ret = _resolve_import(name)
File "/home/cebtenzzre/.local/lib/python3.10/site-packages/importlab/import_finder.py", line 150, in get_imports
imports.append(i + (resolve_import(name, is_from, is_star),))
File "/home/cebtenzzre/.local/lib/python3.10/site-packages/importlab/parsepy.py", line 80, in get_imports
imports = import_finder.get_imports(filename)
File "/home/cebtenzzre/.local/lib/python3.10/site-packages/importlab/graph.py", line 241, in get_file_deps
for imp in parsepy.get_imports(filename, self.env.python_version):
File "/home/cebtenzzre/.local/lib/python3.10/site-packages/importlab/graph.py", line 105, in add_file_recursive
deps, broken = self.get_file_deps(filename)
File "/home/cebtenzzre/.local/lib/python3.10/site-packages/importlab/graph.py", line 227, in create
import_graph.add_file_recursive(os.path.abspath(filename), trim)
File "/home/cebtenzzre/src/forks/pytype/pytype/tools/analyze_project/main.py", line 60, in main
import_graph = importlab.graph.ImportGraph.create(env, conf.inputs, trim=True)
File "/home/cebtenzzre/src/forks/pytype/pytype/__main__.py", line 10, in <module>
sys.exit(main())
<built-in method exec of module object at remote 0x7ffff7580590>
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
my project always fail with PYTHONHASHEED=0 instead with PYTHONHASHEED=1 no longer gives import-error
Thanks for the bug report, and sorry for the slow response. (I've been out of office.) We've had problems in the past with pickles changing due to set iteration order not being deterministic and things like that, so I wonder if this is an issue along the same lines.
