Bug: `poetry install` fails with transitive path dependencies via git

[jpugliesi]

• Poetry version: 1.7.1
• Python version: 3.10.12
• OS version and name: macOS 13.4
• pyproject.toml: https://github.com/jpugliesi/poetry-git-local-dep-bug

• [x] I am on the latest stable Poetry version, installed using a recommended method.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have consulted the FAQ and blog for any relevant entries or release notes.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

Poetry fails to correctly install when it has a transitive path dependency, via a git subdirectory dependency.

For example, see https://github.com/jpugliesi/poetry-git-local-dep-bug where the example package's pyproject.toml contains a git dependency namespace-cli:

[tool.poetry]
name = "example"
version = "0.1.0"
description = ""
authors = ["John Pugliesi <[email protected]>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "~3.10"
namespace-cli = {git = "https://github.com/adriangb/python-monorepo.git", subdirectory = "poetry/workspaces/cli"}

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

The namespace-cli package then has a relative path dependency on another project in the same repo as namespace-cli: (namespace-lib):

[tool.poetry]
name = "namespace-cli"
version = "0.0.0"
description = ""
authors = []
packages = [{include = "namespace", from = "src"}]

[tool.poetry.dependencies]
python = "^3.10"
rich = "^12.6.0"
namespace-lib = { path = "../lib"} # relative path dependency

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Installing the original example package will produce an error:

git clone https://github.com/jpugliesi/poetry-git-local-dep-bug && cd poetry-git-local-dep-bug
pipx install poetry==1.7.1
poetry install

Error:

➜  poetry install
Configuration file exists at /Users/x/Library/Preferences/pypoetry, reusing this directory.

Consider moving TOML configuration files to /Users/x/Library/Application Support/pypoetry, as support for the legacy directory will be removed in an upcoming release.
The currently activated Python version 3.9.16 is not supported by the project (~3.10).
Trying to find and use a compatible version.
Using python3.10 (3.10.12)
Creating virtualenv example in /private/tmp/poetry-git-local-dep-bug/.venv
Installing dependencies from lock file
Path /private/tmp/lib for namespace-lib does not exist
Path /private/tmp/poetry-git-local-dep-bug/.venv/src/python-monorepo/poetry/workspaces/lib for namespace-lib does not exist

Path /private/tmp/lib for namespace-lib does not exist

It seems poetry is resolving the path of the transitive namespace-lib relative to the root directory of the poetry-managed virtualenv, whereas it should be resolving the path relative to the subdirectory of the git dependency.

[dimbleby]

layered path dependencies are pretty much always a bad idea - if you search this issue tracker you will find plenty of examples.

In this case you can see that what poetry has put into its lockfile for namespace-lib is this:

  [package.source]
  type = "directory"
  url = ".venv/src/python-monorepo/poetry/workspaces/lib"

which probably was true at time of locking. But if it's not true at time of install - well, too bad.

Similarly look at the dependency in the lockfile from namespace-cli on namespace-lib:

  [package.dependencies]
  namespace-lib = {path = "../lib"}

well of course, what else should it say? but in the new context this just doesn't mean anything useful.

(This sort of thing is the reason that you cannot upload wheels containing path dependencies to pypi: if you use relative paths then the metadata becomes meaningless, if you use absolute paths then it only works on your machine)

This is doomed. Find some other way!

[JonathanRayner]

layered path dependencies are pretty much always a bad idea - if you search this issue tracker you will find plenty of examples.

Perhaps the point is this is less of a bug and more of a feature request. It's pretty common to work in monorepos with path dependencies, then find you need to install something via git in some other repo.

It's true that someone could find another way, but it doesn't seem that unreasonable to me to feature request that when adding a git dependency poetry could detect that there are path dependencies and resolve them (or raise a helpful error warning that this is impossible).

In my opinion, the expected behavior when adding a git dependency that itself contains path dependencies would be to resolve them as multiple git dependencies with different subdirs from the same repo (which would require solving https://github.com/python-poetry/poetry/issues/6958 ). Maybe there are flaws to this idea, I didn't think too carefully about it yet.

[tbreslein]

I just found this because I ran into this issue. Funnily enough though, running poetry add <URL_WITH_SUBDIR> works fine. So in the example this should work:

poetry add "https://github.com/adriangb/python-monorepo.git#subdirectory=poetry/workspaces/cli"

This does add the entry to the pyproject.toml, but if I were to do a fresh poetry install in this project, it runs into the issue with transitive dependencies again. So there apparently is a way to make it work, and it is anything but "doomed" :smile:

[norbusan]

Same same here, this is a more than common problem I have seen and is a PITB to work around.

The option that works is instead of using path component to use github again, but that brings one into a Goedelian hell of self-referential updates, because poetry.lock needs to refer to the same git commit hash it tries to change. Add a chain of dependency and the ONLY option is hand-editing poetry.lock file.

[abn]

Is this still an issue? I cannot reproduce this.

podman run --rm -i --entrypoint bash python:3.10 <<EOF
set -xe
python -m pip install --disable-pip-version-check --root-user-action ignore -q poetry

install -d foobar
pushd foobar

cat > pyproject.toml <<TOML
[tool.poetry]
name = "example"
version = "0.1.0"
description = ""
authors = ["John Pugliesi <[email protected]>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "~3.10"
namespace-cli = {git = "https://github.com/adriangb/python-monorepo.git", subdirectory = "poetry/workspaces/cli"}

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
TOML

poetry install --no-root
EOF

+ python -m pip install --disable-pip-version-check --root-user-action ignore -q poetry
+ install -d foobar
/foobar /
+ pushd foobar
+ cat
+ poetry install --no-root
Creating virtualenv example-lWDpn5M1-py3.10 in /root/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...

Package operations: 12 installs, 0 updates, 0 removals

  - Installing dnspython (2.7.0)
  - Installing idna (3.10)
  - Installing typing-extensions (4.12.2)
  - Installing annotated-types (0.7.0)
  - Installing email-validator (2.2.0)
  - Installing pydantic-core (2.27.2)
  - Installing commonmark (0.9.1)
  - Installing pydantic (2.10.5)
  - Installing pygments (2.19.1)
  - Installing namespace-lib (0.0.0 /root/.cache/pypoetry/virtualenvs/example-lWDpn5M1-py3.10/src/python-monorepo/poetry/workspaces/lib)
  - Installing rich (12.6.0)
  - Installing namespace-cli (0.0.0 9bb66f4)

Writing lock file

[abn]

Closing this for now as this cannot be reproduced anymore. If this is still an issue, please raise a new issue with a reproducer demonstrating the issue and linking back to this issue.

Thank you @jpugliesi for reporting the issue with great details.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.