python-poetry
Cannot install Monorepo deps without sourcecode for Dockerfile caching
Description
Hi all,
I'm having difficulty setting up a proper Dockerfile for a service in a monorepo that doesn't require the source code for the services and libraries included in the monorepo.
The monorepo has a shared venv. There is a root pyproject.toml and a few packages/libraries and services nested, each with their own pyproject.toml.
My root pyproject.toml looks like this:
# ./pyproject.toml
[tool.poetry.group.main.dependencies]
# all the shared deps
[tool.poetry.group.my-library.dependencies]
my-library = { path = "packages/my-library", develop = true }
[tool.poetry.group.my-service.dependencies]
my-service = { path = "services/my-service", develop = true }
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
and my service will depend on a library as such:
# services/my-service/pyproject.toml
[tool.poetry.dependencies]
python = "3.11.4"
my-library = { path = "../../packages/my-library" }
# my-service's specific deps
I'm trying to optimize my Dockerbuild so that I don't have to rebuild it everytime I have some source changes. Here's what I'd like to do:
FROM python-base-image
RUN apt-get update && \
apt-get install -y supervisor poppler-utils && \
apt-get install --no-install-suggests --no-install-recommends --yes python3-venv pipx
ENV PATH="/root/.local/bin:${PATH}"
RUN pipx install poetry
# Copy poetry and pyproject files over so deps can be installed
COPY ./poetry.lock /home/app/poetry.lock
COPY ./poetry.toml /home/app/poetry.toml
COPY ./pyproject.toml /home/app/pyproject.toml
COPY ./packages/my-library/pyproject.toml /home/app/packages/my-library/pyproject.toml
COPY ./services/my-service/pyproject.toml /home/app/services/my-service/pyproject.toml
WORKDIR /home/app
RUN poetry lock --no-update
RUN poetry install --no-root --no-directory --only my-service,main
# Copy source code over
COPY ./packages /home/app/packages
COPY ./services/my-service /home/app/services/my-service
My understanding is that --no-root --no-directory will allow me to install my service's deps, including my-library, without needing the source code. This way, I can utilize Docker layers so I don't have to rebuild when I have source code changes.
However, I keep getting this error, and I'm unsure of how to fix it - could I get some guidance?
#0 425.8 • Installing my-library (0.1.0 /home/app/packages/my-library)
#0 432.8
#0 432.8 ChefBuildError
#0 432.8
#0 432.8 Backend subprocess exited when trying to invoke build_editable
#0 432.8
#0 432.8 Traceback (most recent call last):
#0 432.8 File "/root/.local/pipx/venvs/poetry/lib/python3.7/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in <module>
#0 432.8 main()
#0 432.8 File "/root/.local/pipx/venvs/poetry/lib/python3.7/site-packages/pyproject_hooks/_in_process/_in_process.py", line 357, in main
#0 432.8 json_out["return_val"] = hook(**hook_input["kwargs"])
#0 432.8 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#0 432.8 File "/root/.local/pipx/venvs/poetry/lib/python3.7/site-packages/pyproject_hooks/_in_process/_in_process.py", line 294, in build_editable
#0 432.8 return hook(wheel_directory, config_settings, metadata_directory)
#0 432.8 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#0 432.8 File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/api.py", line 82, in build_editable
#0 432.8 return WheelBuilder.make_in(
#0 432.8 ^^^^^^^^^^^^^^^^^^^^^
#0 432.8 File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/builders/wheel.py", line 88, in make_in
#0 432.8 wb.build(target_dir=directory)
#0 432.8 File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/builders/wheel.py", line 118, in build
#0 432.8 self._add_pth(zip_file)
#0 432.8 File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/builders/wheel.py", line 147, in _add_pth
#0 432.8 for include in self._module.includes:
#0 432.8 ^^^^^^^^^^^^
#0 432.8 File "/root/.pyenv/versions/3.11.4/lib/python3.11/functools.py", line 1001, in __get__
#0 432.8 val = self.func(instance)
#0 432.8 ^^^^^^^^^^^^^^^^^^^
#0 432.8 File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/builders/builder.py", line 97, in _module
#0 432.8 return Module(
#0 432.8 ^^^^^^^
#0 432.8 File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/utils/module.py", line 75, in __init__
#0 432.8 PackageInclude(
#0 432.8 File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/utils/package_include.py", line 31, in __init__
#0 432.8 self.check_elements()
#0 432.8 File "/tmp/tmpenhb8omp/.venv/lib/python3.11/site-packages/poetry/core/masonry/utils/package_include.py", line 72, in check_elements
#0 432.8 raise ValueError(
#0 432.8 ValueError: /home/app/packages/my-library/my-library does not contain any element
#0 432.8
#0 432.8
#0 432.8 at ~/.local/pipx/venvs/poetry/lib/python3.7/site-packages/poetry/installation/chef.py:147 in _prepare
#0 432.8 143│
#0 432.8 144│ error = ChefBuildError("\n\n".join(message_parts))
#0 432.8 145│
#0 432.8 146│ if error is not None:
#0 432.8 → 147│ raise error from None
#0 432.8 148│
#0 432.8 149│ return path
#0 432.8 150│
#0 432.8 151│ def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
#0 432.8
#0 432.8 Note: This error originates from the build backend, and is likely not a problem with poetry but with my-library (0.1.0 /home/app/packages/my-library) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 --editable "/home/app/packages/my-library"'.
If I copy the sources over before running poetry install, I don't have the issue - but I can't utilize Docker layering.
Thank you guys very much!
PS - I'm also trying to use Docker's cache mount, but Poetry seems to ignore it per the Poetry Config. but that's for another day.
ENV POETRY_CACHE_DIR='/home/.cache/pypoetry'
# ...
RUN --mount=type=cache,target=$POETRY_CACHE_DIR poetry install --no-root --no-directory --only my-service,main
Workarounds
COPY the source for my library and service prior to poetry install - but this is undesired and slow
Poetry Installation Method
pipx
Operating System
Debian
Poetry Version
Poetry 1.5.1
Poetry Configuration
In docker container:
# poetry config --list
cache-dir = "/root/ cache/pypoetry" # /home/ .cache/pypoetry
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir)/virtualenvs" # /home/.cache/pypoetry/virtualenvs/
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "[project_name)-py{python_version}"
### Python Sysconfig
_No response_
### Example pyproject.toml
_No response_
### Poetry Runtime Logs
```bash session
n/a
having something similar in latest aswell. originally the version was 1.5.1
@georgettica can you try without this line?
RUN poetry lock --no-update
Locking requires a metadata build for path dependencies, and this in turn requires the source of the package.
I think you meant to tag me but I'll try it out! if I want to install my deps per the lockfile, is there an alternative to doing that? @abn
Oops, yes. I am guessing you want some form of layer caching here. You can sort of achieve that, the issue you are hitting is that for installing an editable package, Poetry needs to build wheel metadata. This could change in the future.
Here is a made-up example of working around this issue.
/tmp/foo$ tree .
.
├── bar
│ ├── bar
│ │ └── __init__.py
│ ├── poetry.lock
│ ├── pyproject.toml
│ └── README.md
├── Containerfile
├── foo
│ └── __init__.py
├── poetry.lock
├── pyproject.toml
└── README.md
4 directories, 9 files
FROM docker.io/python:3.13
RUN python -m pip install --root-user-action=ignore -q poetry
RUN install -d foo
RUN install -d foo/bar
COPY pyproject.toml poetry.lock foo/.
COPY bar/pyproject.toml bar/poetry.lock bar/README.md foo/bar/.
COPY bar/bar/__init__.py foo/bar/bar/.
WORKDIR foo
RUN poetry install --no-root
Note the addition of README.md and single __init__.py file.
thanks @abn one more thing, would it be possible to have only a global/root-level poetry.lock? or do dependencies within the monorepo must have their own poetry.lock? thank you!
Anything you manage and distribute independently should have its own pyproject.toml file. If you do not care about distributing them or developing them independently, I would imaging you can just use something like this
packages = [
{ include = "package", from="src" },
{ include = "lib_a", from="services/lib_a/src"}
]
And define all metadata in your root pyproject.toml. But all of this is dependent on how you use your monorepo.
A while ago, I created #2270. No progress has been officially been made yet. You can check out the plugin a user mentioned there (not affiliated with the Poetry Team).