Poetry removes installed extras when adding a new dependency

[pronovic]

Description

I've been converting my projects to Poetry v2.0.0. I noticed that Poetry now removes extra dependencies ([project.optional-dependencies]) from the virtualenv when adding a new dependency. This is surprising behavior, and I don't ever remember seeing it with earlier versions of Poetry. If it's intentional, it seems like it should be documented.

I think what I'm demonstrating below should be easy to reproduce in pretty much any project. However, if you want a fully reproducible example, follow the instructions at pronovic/cookiecutter-pypi to create a sample repo — just use all of the defaults. That generates a project with the pyproject.toml I've attached. Initialize the git repository per the repo's README.md (needed due to the poetry-dynamic-versioning plugin), and then follow the steps below.

If you start with poetry sync, you get the plugin and dev dependencies installed as expected:

Git Status: [ master ✲ ✓ ]
mars:~/projects/repos/sample-project> poetry sync
Creating virtualenv sample-project in /home/pronovic/projects/repos/sample-project/.venv
Ensuring that the Poetry plugins required by the project are available...
The following Poetry plugins are required by the project but are not installed in Poetry's environment:
  - poetry-dynamic-versioning[plugin] (>=1.5.0,<2.0.0)
Installing Poetry plugins only for the current project...
Updating dependencies
Resolving dependencies... (1.2s)

Package operations: 4 installs, 0 updates, 0 removals

  - Installing markupsafe (3.0.2)
  - Installing dunamai (1.23.0): Installing...
  - Installing dunamai (1.23.0)
  - Installing jinja2 (3.1.5)
  - Installing poetry-dynamic-versioning (1.5.0): Installing...
  - Installing poetry-dynamic-versioning (1.5.0)

Writing lock file

Updating dependencies
Resolving dependencies... (1.1s)

Package operations: 28 installs, 0 updates, 0 removals

  - Installing distlib (0.3.9)
  - Installing filelock (3.16.1)
  - Installing iniconfig (2.0.0)
  - Installing packaging (24.2)
  - Installing platformdirs (4.3.6)
  - Installing pluggy (1.5.0)
  - Installing astroid (3.3.8)
  - Installing cfgv (3.4.0)
  - Installing click (8.1.8)
  - Installing dill (0.3.9)
  - Installing identify (2.6.5)
  - Installing isort (5.13.2)
  - Installing mccabe (0.7.0)
  - Installing mypy-extensions (1.0.0)
  - Installing nodeenv (1.9.1)
  - Installing pathspec (0.12.1)
  - Installing pytest (8.3.4)
  - Installing pyyaml (6.0.2)
  - Installing tomlkit (0.13.2)
  - Installing typing-extensions (4.12.2)
  - Installing virtualenv (20.28.1)
  - Installing black (24.10.0)
  - Installing colorama (0.4.6)
  - Installing coverage (7.6.10)
  - Installing mypy (1.14.1)
  - Installing pre-commit (4.0.1)
  - Installing pylint (3.3.3)
  - Installing pytest-testdox (3.1.0)

Writing lock file

Installing the current project: sample-project (0.0.0)

Ok, looks good. Then install extras:

Git Status: [ master ✲ …1 ]
mars:~/projects/repos/sample-project> poetry sync --all-extras
Installing dependencies from lock file

Package operations: 23 installs, 0 updates, 0 removals

  - Installing certifi (2024.12.14)
  - Installing charset-normalizer (3.4.1)
  - Installing idna (3.10)
  - Installing markupsafe (3.0.2)
  - Installing urllib3 (2.3.0)
  - Installing alabaster (1.0.0)
  - Installing babel (2.16.0)
  - Installing docutils (0.21.2)
  - Installing imagesize (1.4.1)
  - Installing jinja2 (3.1.5)
  - Installing pygments (2.19.1)
  - Installing requests (2.32.3)
  - Installing snowballstemmer (2.2.0)
  - Installing sphinxcontrib-applehelp (2.0.0)
  - Installing sphinxcontrib-devhelp (2.0.0)
  - Installing sphinxcontrib-htmlhelp (2.1.0)
  - Installing sphinxcontrib-jsmath (1.0.1)
  - Installing sphinxcontrib-qthelp (2.0.0)
  - Installing sphinxcontrib-serializinghtml (2.0.0)
  - Installing sphinx (8.1.3)
  - Installing zipp (3.21.0)
  - Installing importlib-metadata (8.5.0)
  - Installing sphinx-autoapi (3.4.0)

Installing the current project: sample-project (0.0.0+1.767d34b)

Ok, now add some dependency. I picked pydantic since I knew it had some transitive dependencies:

Git Status: [ master ✲ …1 ]
mars:~/projects/repos/sample-project> poetry add pydantic
Using version ^2.10.4 for pydantic

Updating dependencies
Resolving dependencies... (0.8s)

Package operations: 3 installs, 0 updates, 23 removals

  - Removing alabaster (1.0.0)
  - Removing babel (2.16.0)
  - Removing certifi (2024.12.14)
  - Removing charset-normalizer (3.4.1)
  - Removing docutils (0.21.2)
  - Removing idna (3.10)
  - Removing imagesize (1.4.1)
  - Removing importlib-metadata (8.5.0)
  - Removing jinja2 (3.1.5)
  - Removing markupsafe (3.0.2)
  - Removing pygments (2.19.1)
  - Removing requests (2.32.3)
  - Removing snowballstemmer (2.2.0)
  - Removing sphinx (8.1.3)
  - Removing sphinx-autoapi (3.4.0)
  - Removing sphinxcontrib-applehelp (2.0.0)
  - Removing sphinxcontrib-devhelp (2.0.0)
  - Removing sphinxcontrib-htmlhelp (2.1.0)
  - Removing sphinxcontrib-jsmath (1.0.1)
  - Removing sphinxcontrib-qthelp (2.0.0)
  - Removing sphinxcontrib-serializinghtml (2.0.0)
  - Removing urllib3 (2.3.0)
  - Removing zipp (3.21.0)
  - Installing annotated-types (0.7.0)
  - Installing pydantic-core (2.27.2)
  - Installing pydantic (2.10.4)

Writing lock file

So, adding a dependency on pydantic remove all of the extra dependencies that I just installed with poetry sync.

Workarounds

You can just reinstall the extras again later with poetry sync --all-extras or a similar command. It's just kind of odd that they get removed in the first place.

Poetry Installation Method

pipx

Operating System

debian stable

Poetry Version

Poetry (version 2.0.0)

Poetry Configuration

cache-dir = "/home/pronovic/.cache/pypoetry"
installer.max-workers = null
installer.no-binary = null
installer.only-binary = null
installer.parallel = true
installer.re-resolve = true
keyring.enabled = false
requests.max-retries = 0
solver.lazy-wheel = true
system-git-client = false
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/pronovic/.cache/pypoetry/virtualenvs
virtualenvs.prompt = "{project_name}-py{python_version}"
virtualenvs.use-poetry-python = false

Python Sysconfig

(Removed because GitHub says the issue is too big with it included.  I can provide separately if needed.)

Example pyproject.toml

---- poetry.toml

[virtualenvs]
create = true
in-project = true

[keyring]
enabled = false

---- pyproject.toml

[build-system]
requires = ["poetry-core (>=2.0.0)", "poetry-dynamic-versioning (>=1.5.0,<2.0.0)"]
build-backend = "poetry_dynamic_versioning.backend"

[tool.poetry]
requires-poetry = ">=2.0.0"
include = [
   { path = 'Changelog', format = 'sdist' },
   { path = 'NOTICE', format = 'sdist' },
   { path = 'LICENSE', format = 'sdist' },
   { path = 'README.md', format = 'sdist' },
   { path = 'docs', format = 'sdist' },
   { path = 'tests', format = 'sdist' },
]
packages = [ 
   { include = "sample", from = "src" },
]
classifiers=[
   "Development Status :: 4 - Beta",
]
version = "0.0.0" # published version is managed using Git tags (see below)

[tool.poetry.requires-plugins]
poetry-dynamic-versioning = { version = ">=1.5.0,<2.0.0", extras = ["plugin"] }

# Published version is managed using Git tags
# We get either the tag (like "0.24.1") or a snapshot-type version (like "0.24.1+3.e8319c4")
[tool.poetry-dynamic-versioning]
enable = true
pattern = '^[vV](?P<base>\d+\.\d+\.\d+)'  # this extracts the version from our vX.Y.Z tag format
format-jinja = "{% if distance == 0 and not dirty %}{{ base }}{% else %}{{ base }}+{{ distance }}.{{ commit }}{% endif %}"

[project]
name = "sample-project"
requires-python = ">=3.10,<4"
description = "Short description"
authors = [ { name="First Last", email="[email protected]" } ]
license = "Apache-2.0"
readme = "PyPI.md"
dynamic = [ "classifiers", "version" ]
dependencies = ["pydantic (>=2.10.4,<3.0.0)"]

[project.urls]
homepage = "https://pypi.org/project/sample-project/"
repository = "https://github.com/owner/samplerepo"

[project.optional-dependencies]
docs = [
   "importlib-metadata (>=8.5.0,<9.0.0)",
   "sphinx (>=8.1.3,<9.0.0)",
   "sphinx-autoapi (>=3.0.0,<4.0.0)",
]

[tool.poetry.group.dev.dependencies]
pytest = ">=8.0.2,<9.0.0"
pytest-testdox = ">=3.1.0,<4.0.0"
coverage = ">=7.4.4,<8.0.0"
pylint = ">=3.0.1,<4.0.0"
pre-commit = ">=4.0.1,<5.0.0"
black = ">=24.2.0,<25.0.0"
mypy = ">=1.6.0,<2.0.0"
isort = ">=5.12.0,<6.0.0"
colorama = ">=0.4.6,<1.0.0"

[tool.black]
line-length = 132
target-version = ['py310']
include = '(src\/scripts\/.*$|\.pyi?$)'
exclude = '''
/(
    \.git
  | __pycache__
  | \.tox
  | \.venv
  | \.poetry
  | build
  | dist
  | docs
  | notes
)/
'''

[tool.isort]
profile = "black"
line_length = 132
skip_glob = [ "docs", "notes", ".poetry" ]

[tool.coverage.paths]
source = [ "src" ]

[tool.coverage.run]
branch = true
source = [ "src/sample" ]

[tool.coverage.report]
show_missing = false
precision = 1

[tool.pytest.ini_options]
filterwarnings = [
    'error',  # turn all Python warnings into test failures, so they're hard to miss
]

[tool.mypy]
# Settings are mostly equivalent to strict=true as of v1.14.1
pretty = true
show_absolute_path = true
show_column_numbers = true
show_error_codes = true
files = [ "src/sample", "tests" ]
check_untyped_defs = true
disallow_any_generics = true
disallow_incomplete_defs = true
disallow_subclassing_any = true
disallow_untyped_calls = true
disallow_untyped_decorators = false
disallow_untyped_defs = true
no_implicit_optional = true
no_implicit_reexport = true
strict_equality = true
strict_optional = true
warn_redundant_casts = true
warn_return_any = true
warn_no_return = true
warn_unused_configs = true
warn_unused_ignores = true

# It's hard to make tests compliant using unittest.mock
[[tool.mypy.overrides]]
module = "tests.*"
check_untyped_defs = false
allow_untyped_defs = true

# There is no type hinting for pytest
[[tool.mypy.overrides]]
module = "pytest"
ignore_missing_imports = true

Poetry Runtime Logs

(Removed because GitHub says the issue is too big with it included.  I can provide separately if needed.)

[pronovic]

I've also noticed that something similar happens when I run poetry update. The extras are uninstalled before the command runs, and they're never listed in the output from poetry show --outdated.

I thought this might be related to my use of [project.optional-dependencies], so I fell back to using dynamic = ["dependencies"] and put all of the dependencies back into [tool.poetry.dependencies] and [tool.poetry.extras], like it was previously. However, I get the same behavior. So, it seems like extras are being treated differently in v2.0.0, regardless of how they're defined in pyproject.toml.

Note: I've tried this with both installer.re-resolve=false and installer.re-resolve=true, just in case that might somehow matter. However, I get the same behavior in either case.

[finswimmer]

Maybe related to or duplicate of #9950 ?

[radoering]

At least related. I can verify that it is caused by the same commit: https://github.com/python-poetry/poetry/pull/9345/commits/bd3500d07cd0ab0d10d26917fdda3abdd8e67704

I would not say it is a duplicate because groups and extras are different things - are handled differently - and it might be possible to fix one without the other when not paying attention.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.