Incremental parsing issue with namespace packages

[PeterJCLaw]

Given a local file which uses something which is a namespace package and then another local file which uses the first one, there's an intermittent bug where the namespace packaged module's types aren't found. This seems likely similar to https://github.com/python/mypy/issues/7276, however even with https://github.com/python/mypy/pull/10937 in place (i.e: on a more recent mypy release) the issue remains.

To Reproduce

$ mkdir demo
$ echo 'import ruamel.yaml' > demo/wrapper.py 
$ echo 'import wrapper' > demo/consumer.py

$ mypy demo  # first run
Success: no issues found in 2 source files

$ mypy demo  # second run
demo/wrapper.py:1: error: Skipping analyzing "ruamel": module is installed, but missing library stubs or py.typed marker
demo/consumer.py:1: error: Skipping analyzing "ruamel": module is installed, but missing library stubs or py.typed marker
demo/consumer.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 2 errors in 2 files (checked 2 source files)

Subsequent runs will alternate between success and failure. Any additional source files in the project (whether they import the wrapper or not) will also have the error message applied to them, which makes failing runs vastly less useful.

Running with --no-incremental appears to avoid the issue; I can run several times in a row with --no-incremental without encountering the failure mode (and have not seen it with that flag enabled).

Running with --namespace-packages appears to have no effect.

Unfortunately as ruamel.yaml is a typed library, ignoring the imports with type: ignore[import] ends up generating more errors as the ignores appear unused on a "successful" run (and you have warn_unused_ignores enabled).

Setup & versions

python3.9 -m venv venv
. venv/bin/activate
pip install -U pip setuptools wheel
pip install mypy ruamel.yaml

$ pip list
Package           Version
----------------- -------
mypy              0.942
mypy-extensions   0.4.3
pip               22.0.4
ruamel.yaml       0.17.21
ruamel.yaml.clib  0.2.6
setuptools        62.1.0
tomli             2.0.1
typing_extensions 4.2.0
wheel             0.37.1

Your Environment

• Mypy version used: 0.942
• Mypy command-line flags: see above
• Mypy configuration options from mypy.ini (and other config files): none
• Python version used: 3.9 (though similar behaviour observed on other versions; at least on 3.8)
• Operating system and version: Ubuntu 20.04 LTS

[hauntsaninja]

Thanks, I can repro on master, which means the unreleased https://github.com/python/mypy/pull/12250 does not fix either

edit: this still repros as of 2022/10/28, so I guess #13124 didn't fix either (looks unrelated to the all_imported_modules_in_file code path). Also linking this comment: https://github.com/python/mypy/issues/13085#issuecomment-1178036945

[jacleggmsr]

I'm getting the same issue.

I'm running python 3.10.7, mypy 0.982, mypy-extensions 0.4.3, ruamel.yaml 0.17.21 and ruamel.yaml.clib 0.2.7 in a larger project.

I followed the exact steps above to repro with a new virtual environment after pip install ruamel-yaml and pip install mypy and fails in the same way.
Output of pip list:

Package           Version
----------------- -------
mypy              0.991
mypy-extensions   0.4.3
pip               22.2.2
ruamel.yaml       0.17.21
ruamel.yaml.clib  0.2.7
setuptools        63.2.0
tomli             2.0.1
typing_extensions 4.4.0

Python 3.10.7, Windows 11.

[sfc-gh-dschumann]

Hitting the same issue. Do we know what causes this?

[ssbarnea]

The no incremental does not help with runamel.yaml because the final behavior of mypy depends on presence or absence of the optional binary dependency of it, ruamel-yaml-clib.

When ruamel-yaml-clib is installed, it fails to detect the types for some methods, but in its absence it does. This combined with https://github.com/python/mypy/issues/8823 puts us in a difficult situation, especially if using strict mode.

[davnat]

I can confirm this on Ubuntu 20.04.6 LTS, python 3.8.10, mypy 1.3.0.

In my case the problem appears only in a package that depends on ruamel-yaml very indirectly (it's a dep of a dep of a dep), while in the original package that has the direct dependency and all others in the chain all seems OK, for now.

I noted that the problematic package has a lot of from foo import *, maybe this could trigger the error?

Some more details:

• in all packages in the chain we're using mypy in strict mode, but in the package where we face the problem we have strict = False in mypy.ini
• ruamel.yaml.clib is also installed
• in our case, adding incremental = False in mypy.ini solves the issue... but slows down all builds in CI quite a bit, obviously

[florian-cl]

I can confirm this issue on MacOS 14.1.2, python 3.11.6, mypy 1.7.1

[dahlbaek]

I can confirm this issue on MacOS 13.2.1, python 3.10.10, mypy 1.8.0

[DBCerigo]

Similarly still seeing this issue on intermitant mypy runs. --no-incremental fixes it, but also seems to slow down the typechecking significantly (as I think is to be expected), so is far from an ideal solution.

On Ubuntu Debian 22.04 LTS

mypy==1.8.0
ruamel.yaml==0.18.6
ruamel.yaml.clib==0.2.8

[Sh4pe]

We are affected by this issue as well. Using --no-incremental or simply deleting the .mypy_cache directory each time works as a workaround, but this slows down type checking considerably. Which is obviously not preferred.

We are using Poetry and this are the relevant versions from our poetry.lock:

mypy              1.10.0
ruamel-yaml       0.18.6
ruamel-yaml-clib  0.2.8

I want to say that using mypy is very valuable to us, but this is a small annoyance. Is there a workaround that works with the cache?

[petergaultney]

python 3.10.13
mypy 1.10.1
google-auth 2.30.0

Likely not relevant, but this does not ever appear to happen with .pth local editable dependencies that are themselves namespace packages. We have lots of those and this issue never occurs with them.

It's also not deterministic, but I can confirm the behavior where it always succeeds when the cache is empty.

[rworkduo]

after a year, this issue is still haunting me and costing me a significant amount of CI running time is it even being looked into yet?

[martinhoyer]

This is ugly, but avoids usage of --no-incremental, and only skips the cache for problematic modules. In our case it's ruamel.yaml - mypy.ini:

[ruamel.*]
cache_dir = /dev/null

(the ruamel/yaml cache is still created, but not used in subsequent runs, IIUC)

[ohshazbot]

This is ugly, but avoids usage of --no-incremental, and only skips the cache for problematic modules. In our case it's ruamel.yaml - mypy.ini:

[ruamel.*]
cache_dir = /dev/null

(the ruamel/yaml cache is still created, but not used in subsequent runs, IIUC)

unfortunately this solution does not seem to work for me, it does not cause any sort of behavior change

[ego-thales]

Thank god for the OP perfectly describing the issue. I encounter it here.

Using uv 0.6.6, mypy v1.15.0, ruamel-yaml v0.18.10 on Ubuntu 22.04 and python 3.12/3.13.

All the best.

[hauntsaninja]

One hack "fix" is to add this to your pyproject.toml under [tool.mypy]:

[[tool.mypy.overrides]]
module = "ruamel.yaml"
follow_imports = "skip"

Another one is to just # type: ignore (if you use --warn-unused-ignores something like # type: ignore[import, unused-ignore] should work)

[ego-thales]

Thanks @hauntsaninja! This alternative monkeypatch is much more appropriate to me. It allows caching to work properly only at the cost of not following this specific package, which does not really need type checking in my project.

It'll save me a lot of time <3

[hauntsaninja]

This is released now in 1.16