Poetry environment breaks when Homebrew upgrades Python on MacOS

[keithvsr]

Issue experienced in Poetry 1.2.1 Python version 3.10.6_2 -> 3.10.7 and 3.10.7 -> 3.10.8 upgrades OS version MacOS 12.6 (not project-specific so no pyproject.toml)

Issue

Poetry commands cease working and throw errors after upgrading the installed Python version using Homebrew. This has occurred the last two times (3.10.6_2 upgrade to 3.10.7 and 3.10.7 upgrade to 3.10.8) using brew upgrade which included the two Python updates. Afterward, attempting to use any Poetry command fails with error output similar to the following:

poetry --version
dyld[42062]: Library not loaded: '/usr/local/Cellar/[email protected]/3.10.7/Frameworks/Python.framework/Versions/3.10/Python'
  Referenced from: '/Users/keith/Library/Application Support/pypoetry/venv/bin/python'
  Reason: tried: '/usr/local/Cellar/[email protected]/3.10.7/Frameworks/Python.framework/Versions/3.10/Python' (no such file), '/Library/Frameworks/Python.framework/Versions/3.10/Python' (no such file), '/System/Library/Frameworks/Python.framework/Versions/3.10/Python' (no such file)
zsh: abort      poetry --version

In both cases, the only Python installations available are system packages (if these even still exist) or those installed by Homebrew. Poetry has only been installed via the recommended curl -sSL https://install.python-poetry.org | python3 - command. The only way I've found to restore Poetry functionality after brew upgrade is to uninstall via the same recommended script i.e. curl -sSL https://install.python-poetry.org | python3 - --uninstall and then to reinstall Poetry

[social-anthrax]

Similar issue here. I ran into this previously when upgrading from 3.10.6 and from 3.10.7. It seems that the old binaries are referenced by /Users/<user>/Library/Application\ Support/pypoetry/venv/bin/python3, reinstalling fixes it. Alternitively an easy fix is to add a symlink in /opt/homebrew/Cellar/[email protected] with the target_file being named the old version. Not sure if this will break versioning

[jeffzyliu]

same thing happened to me from 3.10.8 to 3.10.9, reinstall worked just fine.

wondering why the binary needs to reference a specific python version. might install this with pipx instead if homebrew python gives me more trouble with the official installer.

[SaulAryehKohn]

The source of this same issue for me was conflicting poetry installs. Changing the name of whatever came up from which poetry (which was in .local/bin for my setup) let the system discover it in /opt/homebrew/. Everything worked after that.

[kevinrenskers]

Running into the same issue. I used Python3 installed via Homebrew, and after a minor version update, Poetry was broken. Should we not use Homebrew to install Python? Would using pyenv solve this issue going forward?

[jmichaels]

Same issue just hit me too, fixed with uninstall and reinstall:

curl -sSL https://install.python-poetry.org | python3 - --uninstall
curl -sSL https://install.python-poetry.org | python3

[jmichaels]

I just ran into this again, after doing a brew update. The above uninstall/reinstall failed to fix it this time.

New solution:

1. Remove the homebrew installed python
2. Install pyenv via homebrew
3. Install python via pyenv
4. Reinstall poetry

brew remove python
brew install pyenv

echo 'export PATH="$HOME/.pyenv/shims:$PATH"' >> ~/.profile
source ~/.profile

pyenv install 3.11.4
pyenv global 3.11.4

curl -sSL https://install.python-poetry.org | python3 - --uninstall
curl -sSL https://install.python-poetry.org | python3 -

[abulte]

Ran into the same issue.

Removing homebrew's python does not work for me, I have many dependent formulas.

I also use pyenv for my python dev needs.

I managed to successfully reinstall poetry by using the pipx method (installed via homebrew): https://python-poetry.org/docs#installing-with-pipx

[honzajavorek]

I don't think you need to uninstall the homebrew installed python. Install a separate python using pyenv, don't make it global, then use that python to install poetry:

curl -sSL https://install.python-poetry.org | "$(pyenv root)/versions/3.11.4/bin/python" -

[social-anthrax]

I found a solution in issue #52. The current way to fix this is to change the venv.EnvBuilder symlink parameter to True when reinstalling.

Fix:

1. Uninstall poetry
2. Download the install script
3. Change the venv.EnvBuilder symlink parameter to True in the install script.
4. Run the new script

Steps

mkdir -p /tmp/poetry_installer/ ; cd /tmp/poetry_installer # This can be anywhere
curl -sSL https://install.python-poetry.org > poetry_installer.py
python3 ./poetry_installer.py --uninstall

Change line 317 of poetry_installer.py (at the time of writing) from

-              builder = venv.EnvBuilder(clear=True, with_pip=True, symlinks=False)
+              builder = venv.EnvBuilder(clear=True, with_pip=True, symlinks=True)

python3 ./poetry_installer.py

Poetry should work again!

[jlrobins]

Thanks social-anthrax -- that procedure worked!

[social-anthrax]

I'd be interested in seeing what would break if one set the symlinks to True on a standard install. If it doesn't break anything and doesn't change behaviour it could be worth having that as the standard option if it fixes MacOS installation

[nfantone]

Hit this today after a brew upgrade. --uninstall did not work.

dyld[36086]: Library not loaded: @loader_path/../../../../Python.framework/Versions/3.11/Python
  Referenced from: <16376C03-0DA9-3714-9207-5AF8D546EED4> /Users/nicolas.fantone/Library/Application Support/pypoetry/venv/bin/python3.11
  Reason: tried: '/Users/nicolas.fantone/Library/Application Support/pypoetry/venv/bin/../../../../Python.framework/Versions/3.11/Python' (no such file), '/Library/Frameworks/Python.framework/Versions/3.11/Python' (no such file), '/System/Library/Frameworks/Python.framework/Versions/3.11/Python' (no such file, not in dyld cache)

Traceback:

  File "<stdin>", line 923, in main
  File "<stdin>", line 562, in run

I'm curious as to where that @loader_path/../../../../Python.framework/Versions/3.11/Python path is being derived from.

@social-anthrax solution actually worked — but I'll admit, I'm not aware of the implications.

[keithvsr]

Thank you @social-anthrax for the follow-up!

My setup has evolved to include pyenv use since initial issue reporting so I can't confirm if your solution works specifically for Homebrew updates but I can confirm it does not seem to solve the Python update issue generally. Specifically the symlink created for the Poetry virtual environment after following your installation steps references an exact binary which breaks that linkage on a new pyenv Python installation:

/Users/{user}/Library/Application Support/pypoetry/venv/bin/python3 -> /Users/{user}/.pyenv/versions/3.11.5/bin/python3

Simulating installing a new version of Python (3.11.5 is the latest I see in pyenv currently so instead I downversioned to 3.11.4):

% poetry --version
Poetry (version 1.6.1)
% pyenv install 3.11.4
... installation snipped ...
Installed Python-3.11.4 to /Users/{user}/.pyenv/versions/3.11.4
% pyenv uninstall 3.11.5
pyenv: remove /Users/{user}/.pyenv/versions/3.11.5? [y|N] y
pyenv: 3.11.5 uninstalled
% poetry --version
/Users/{user}/.local/bin/poetry: line 2: /Users/{user}/Library/Application Support/pypoetry/venv/bin/python: No such file or directory
/Users/{user}/.local/bin/poetry: line 2: exec: /Users/{user}/Library/Application Support/pypoetry/venv/bin/python: cannot execute: No such file or directory

More generally, my expectation of the symlinks option being False by default in the Poetry installer was that an isolated Python binary would be copied (per the venv API documentation) for use running Poetry and therefore system/pyenv/homebrew updates would not impact Poetry itself running.

I have to admit to being confused that no matter which option we choose, external updates seem to be interfering with what should otherwise be an isolated virtual environment. Am I the problem here?

[nfantone]

@keithvsr I don't think you are. From Poetry's own docs:

Poetry makes project environment isolation one of its core features.

What this means is that it will always work isolated from your global Python installation.

This doesn't seem to be the case as removing/moving the global python binary, breaks every poetry env for me on macOS.

[sebpretzer]

I was able to convert @social-anthrax's answer into a one liner so you don't have to download anything if you don't want to:

curl -sSL https://install.python-poetry.org | sed -e 's|symlinks=False|symlinks=True|' | python3 -

although this may not work in the future if symlinks= is used in multiple places.

[ChristopherHammond13]

Thank you @sebpretzer! This finally works now after much head-scratching. I upgraded Python via Homebrew as I normally do but for the first time doing an --uninstall followed by a regular install did not work, and the installer kept failing as it tried to point to a (non-existent) system version of Python. That symlink parameter patch appears to make it play nicely with my Homebrew-installed Python 3.11.

[social-anthrax]

@keithvsr My apologies, I should have clarified that my code was only a fix for the reinstallation error rather than a fix for the underlying problem!

[yuriyostapenko]

I've opened #133 that should fix the installation issue with symlinks (#24), but as I also experienced this issue with upgrades previously myself, I am curious what the longer term or better fix should be to address this.

Relative symlinks are clearly an issue, and if goal is to isolate poetry's own environment, would it make sense to resolve symlinks recursively until we find the real binary instead of installing with symlinks=True?