Poetry 1.1.x breaks mypy in a project with source dependencies

[kkozmic]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: MacOS 10.16.5
• Poetry version: 1.1.1
• Link of a Gist with the contents of your pyproject.toml file: link

Issue

This is an issue with mypy under .venv created with latest Poetry (v1.1.x) which works just fine if the .venv was created with older version (v1.0.10 or before)

Our project has two source dependencies on other internal projects. After the upgrade (and re-creating the .venv) running poetry run mypy . produces a whole bunch of errors like:

error: Cannot find implementation or library stub for module named 'our_source_project1.something'
error: Cannot find implementation or library stub for module named 'our_source_project2.something_else'

Digging through the logs, diffs and then mypy source code I narrowed it down to this:

Under Poetry v1.0 a .venv/lib/python3.8/site-packages/easy-install.pth file gets created which contains paths to each of our code dependency projects.

Under Poetry v1.1 this file doesn't get created.

Thing is, mypy reads this specific file to find these source dependencies. (source). Without it it doesn't know how to find them, hence the errors, and a broken build.

[finswimmer]

Hello @kkozmic,

is this related: https://github.com/python-poetry/poetry/issues/2194#issuecomment-703246165 ?

fin swimmer

[kkozmic]

hey @finswimmer

I looked at that workaround, but if I understand it correctly, mypy_path/MYPYPATH adds other paths for mypy to scan. It's saying "my project's code is in **/*.py but also in this other folder and that second other folder as well".

That's not what I want - I want those additional dependencies to be treated as dependencies, not as my project code.

Additionally, running mypy with mypy_path added fails as it ignores these projects' own mypy settings. Trying to make my project's settings accommodate those dependencies (and keeping them up to date over time) is a recipe for a headache.

I want paths to those two projects to end up in package_path, not mypy_path, like they do under Poetry v1.0.x (and sadly there's no equivalent package_path config value as far as I'm aware).

[abn]

@kkozmic I am guessing in this scenario your dependencies are poetry projects installed with develop = true? If so, they should be still written to package-name.pth instead of easy-install.pth

[abn]

~~Just read your pypt, this is actually https://github.com/python-poetry/poetry/issues/3086.~~

[kkozmic]

Just read your pypt, this is actually #3086.

OK? I saw that issue before submitting this one, and it looked related, albeit concerned with race conditions whereas my problem is broken mypy.

I am guessing in this scenario your dependencies are poetry projects installed with develop = true?

We've got tag = something and branch = something. Not sure how that affects things.

[kkozmic]

I tried it with poetry config experimental.new-installer false --local and that doesn't seem to affect this particular problem. Only rolling back to v1.0.10 fixed it

[abn]

@kkozmic I was on my mobile. So looking at the mypy code; it seems they should really be loading all pth files as easy-install.pth is only guaranteed by setuptool installs and as a consequence pip. In a PEP 517 world; this is a broken assumption I feel.

EDIT: The code above the referenced one seems to be processing site packages. I wonder why the paths do not get loaded.

EDIT2: Would be great to get a sample project to test this.

[kkozmic]

Right, so you're basically saying it's more of a mypy bug than Poetry (which is fair enough).

EDIT2: Would be great to get a sample project to test this.

I'd have to put something together from scratch as obviously I can't use our production code. I'll look into it over the coming weekend

[abn]

Right, so you're basically saying it's more of a mypy bug than Poetry (which is fair enough).

Yes. However, if there is anything we can do to keep things compatible, I am happy to look into it. We can, at least for 1.1 play nice and write to easy-install.pth as well if there are no undesired side-effects.

I'd have to put something together from scratch as obviously I can't use our production code. I'll look into it over the coming weekend

Awesome. That would make things easier to triage/decide.

[kkozmic]

Yes. However, if there is anything we can do to keep things compatible, I am happy to look into it

Awesome, and I appreciate it 👏

[kkozmic]

So I got carried away and created a reproduction @abn

git clone [email protected]:bug-repros/poetry-main.git
cd poetry-main
poetry install
make

This works under Poetry 1.0.10 and fails under 1.1.1 with the error from OP.

[bryanforbes]

The next release of mypy (0.800) will include a fix for this, but I'm not sure when it will come out since 0.790 was just released today

[kkozmic]

No worries, I'll be keeping an eye out for it. Thanks @bryanforbes

[abn]

@kkozmic thank you for the reproducer. It is extremely helpful.

Had a quick look into this, and I am leaning towards not making any changes at the moment as I could not find any other cases where the easy-install.pth is expected/required in a similar fashion. And considering we expect a fix in mypy, adding an additional point of failure seems unwise. The easy-install.pth file is intended to be operated on only by setuptools and with that in mind, poetry writing to it and removing lines from it might end up creating more problems than it solves.

If anyone has a compelling case other than mypy that will also beneifit from this, please do list it here.

[bryanforbes]

My only suggestion would be to add a flag to optionally generate easy-install.pth. Right now, I'm using a script to generate easy-install.pth and I have to add it to a few of my projects and remember to run it. I understand, though, if you decide against it.

[abn]

To be completely honest, it seems like too much effort for a temporary workaround. I'll think on it.

If there are other cases that are important relying on the file, it might be a different story. Right now, I just haven't found any.

[kkozmic]

Fair call @abn . Now 🤞 mypy 0.800 gets released soon so I can upgrade :partyparrot:

[edmondop]

Can the problem I have be related to this issue https://stackoverflow.com/questions/65512870/mypy-failing-on-test-source-folders-because-of-absence-of-type-hints-or-library?noredirect=1#comment115833764_65512870 ? And if that's the same cause, is there a known workaround?

[andrewtemme]

I just found this issue today when I started running into the problem. Looks like MyPy v.0.800 was released today, 22 Jan 2021. I'm getting this error with the new 0.800 version.

[dimbleby]

works fine with latest poetry / mypy - dunno which is fixed but this seems good now

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.