python-poetry
Resolution of dependencies in directories fails to respect explicit source
Description
When using a mono-repo with dependencies that are sub-directories, installation works fine, but when adding/removing deps, the resolution process fails because an internal dependency can't be found on PyPi. This happens because the source information isn't propagated correctly from the Provider.
This is admittedly a pretty niche issue for our project structure. As best as I can tell, the explicit nature of the repository isn't pulled into the _explicit_sources in the provider from the DirectoryPackage that actually holds the definition.
Workarounds
I was able to mitigate by modifying the source file directly for Provider, changing from this:
https://github.com/python-poetry/poetry/blob/a74d3a79028535baf48073c969d327978d87d6c9/src/poetry/puzzle/provider.py#L708
To this:
if explicit_source := self._explicit_sources.get(dependency.name) or package.source_reference:
This has the effect (as far as I can tell) that the source pulled from the lock file will always be respected, so I don't know if it's a valid long term fix.
Poetry Installation Method
other
Operating System
Ubuntu 22.04.4
Poetry Version
1.8.3
Poetry Configuration
cache-dir = "/home/Philip.Capel/.cache/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
repositories.internal.url = "https://pypi.dev.internal.tools"
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs" # /home/Philip.Capel/.cache/pypoetry/virtualenvs
virtualenvs.prefer-active-python = true
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true
Python Sysconfig
No response
Example pyproject.toml
You will need to have a mono-repo style project, where there is a top level package defined by the following:
[tool.poetry.dependencies]
python = ">=3.10,<3.11"
some-sub-package = {path = "./components/some-sub-package", develop = true, optional = true}
Within the some-sub-package if you define the internal source and point to it explicitly in the dependency you could have something like:
[tool.poetry.dependencies]
dep_that_breaks = {version = "^1.12.0", source = "internal"}
[[tool.poetry.source]]
name = "internal"
url = "https://pypi.dev.internal.tools"
priority = "explicit"
Poetry Runtime Logs
Loading configuration file /home/Philip.Capel/.config/pypoetry/config.toml
Adding repository evozyne (https://pypi.dev.internal.tools) and setting it as explicit
Using virtualenv: /home/Philip.Capel/projects/mew/.venv
Updating dependencies
Resolving dependencies...
# redacted
Source (PyPI): Getting info for dep_that_breaks (0.4) from PyPI
[urllib3:urllib3.connectionpool] https://pypi.org:443 "GET /pypi/dep_that_breaks/0.4/json HTTP/11" 404 24
Falling back to installed packages to discover metadata for dep_that_breaks
Found 0 compatible packages for dep_that_breaks
1: Version solving took 0.500 seconds.
1: Tried 1 solutions.
there is no way to pass source information through standard metadata, it is expected that a parent project knows nothing about sources defined in a dependency.
