Poetry relies on tomlkit `0.11.{2/3}` which has a memory leak

[TNonet]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: >>>cat /etc/os-release NAME="Ubuntu" VERSION="20.04.4 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.4 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal
• Poetry version: Multiple
• Link of a Gist with the contents of your pyproject.toml file:

Issue

TL;DR: poetry export now seems to use 5x+ more memory than it used to.

In a CircleCI workflow, we run poetry export. This has worked for months, but sometime between Friday (8/5/2022) and Monday (8/8/2022) morning, something changed and caused poetry export to use significantly more memory.

Using this cimg/python:3.9 docker image which has poetry 1.1.13 installed, we install poetry 1.1.14 using curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python3 - --version 1.1.14.

Then running poetry export produces the following allocations (using https://github.com/bloomberg/memray):

⠋ Calculating high watermark... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   Memray ERROR: Failed to process memory record

Allocation metadata
-------------------
Command line arguments: '/home/circleci/.local/share/pypoetry/venv/lib/python3.9/site-packages/memray/__main__.py run -o output.bin /home/circleci/.local/bin/poetry export'
Peak memory size: 1.878GB
Number of allocations: 55887

Biggest 10 allocations:
-----------------------
📂 1.078GB (100.00 %) <ROOT>
└── [[20 frames hidden in 14 file(s)]]
    └── 📂 1.078GB (100.00 %) <listcomp>  /home/circleci/.local/share/pypoetry/venv/lib/python3.9/site-packages/tomlkit/source.py:71
        ├── [[2 frames hidden in 2 file(s)]]
        │   └── 📄 86.750MB (7.86 %) update_wrapper  /home/circleci/.pyenv/versions/3.9.13/lib/python3.9/functools.py:56
        ├── 📂 219.125MB (19.86 %) __init__  /home/circleci/.local/share/pypoetry/venv/lib/python3.9/site-packages/tomlkit/toml_char.py:9
        │   ├── 📄 128.375MB (11.63 %) decorating_function  /home/circleci/.pyenv/versions/3.9.13/lib/python3.9/functools.py:519
        │   └── [[1 frames hidden in 1 file(s)]]
        │       └── 📄 90.750MB (8.22 %) update_wrapper  /home/circleci/.pyenv/versions/3.9.13/lib/python3.9/functools.py:56
        ├── 📂 215.750MB (19.55 %) __init__  /home/circleci/.local/share/pypoetry/venv/lib/python3.9/site-packages/tomlkit/toml_char.py:11
        │   ├── 📄 122.500MB (11.10 %) decorating_function  /home/circleci/.pyenv/versions/3.9.13/lib/python3.9/functools.py:519
        │   └── [[1 frames hidden in 1 file(s)]]
        │       └── 📄 93.250MB (8.45 %) update_wrapper  /home/circleci/.pyenv/versions/3.9.13/lib/python3.9/functools.py:56
        ├── 📂 212.250MB (19.24 %) __init__  /home/circleci/.local/share/pypoetry/venv/lib/python3.9/site-packages/tomlkit/toml_char.py:13
        │   ├── 📄 120.500MB (10.92 %) decorating_function  /home/circleci/.pyenv/versions/3.9.13/lib/python3.9/functools.py:519
        │   └── [[1 frames hidden in 1 file(s)]]
        │       └── 📄 91.750MB (8.32 %) update_wrapper  /home/circleci/.pyenv/versions/3.9.13/lib/python3.9/functools.py:56
        ├── [[2 frames hidden in 2 file(s)]]
        │   └── 📄 97.500MB (8.84 %) update_wrapper  /home/circleci/.pyenv/versions/3.9.13/lib/python3.9/functools.py:56
        └── [[2 frames hidden in 2 file(s)]]
            └── 📄 86.750MB (7.86 %) update_wrapper  /home/circleci/.pyenv/versions/3.9.13/lib/python3.9/functools.py:56

Workaround attempts:

1. Running the same workflow without installing poetry 1.1.13 runs fine but suffers from https://github.com/python-poetry/poetry/issues/5967.
2. Updating poetry using poetry update self results in the same OOM.
3. Uninstalling poetry using the info from this issue, https://github.com/python-poetry/poetry/issues/644, does not work. Therefore, I can't test uninstalling 1.1.13 and installing 1.1.14.

Running the same workflow on this cimg/python:3.10 docker image which has poetry 1.1.14 runs which much smaller allocation:

Allocation metadata
-------------------
Command line arguments: '/home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/memray/__main__.py run -o output.bin /home/circleci/.local/bin/poetry export'
Peak memory size: 209.743MB
Number of allocations: 488785

Biggest 10 allocations:
-----------------------
📂 170.559MB (100.00 %) <ROOT>
└── [[3 frames hidden in 2 file(s)]]
    └── 📂 170.559MB (100.00 %) _run_code  /home/circleci/.pyenv/versions/3.10.6/lib/python3.10/runpy.py:86
        ├── [[5 frames hidden in 4 file(s)]]
        │   └── 📂 169.304MB (99.26 %) wrap_handle  /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/cleo/commands/command.py:92
        │       ├── [[2 frames hidden in 2 file(s)]]
        │       │   └── 📂 141.291MB (82.84 %) read  /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/core/toml/file.py:32
        │       │       ├── [[2 frames hidden in 2 file(s)]]
        │       │       │   └── 📂 140.117MB (82.15 %) parse  /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/tomlkit/api.py:82
        │       │       │       ├── [[3 frames hidden in 2 file(s)]]
        │       │       │       │   └── 📄 36.000MB (21.11 %) __init__  /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/tomlkit/toml_char.py:8
        │       │       │       └── [[10 frames hidden in 2 file(s)]]
        │       │       │           └── 📄 14.087MB (8.26 %) _to_linecol  /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/tomlkit/source.py:179
        │       │       └── [[1 frames hidden in 1 file(s)]]
        │       │           └── 📄 1.173MB (0.69 %) decode  /home/circleci/.pyenv/versions/3.10.6/lib/python3.10/codecs.py:322
        │       ├── [[7 frames hidden in 5 file(s)]]
        │       │   └── 📂 25.014MB (14.67 %) parse  /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/tomlkit/api.py:82
        │       │       ├── 📂 16.000MB (9.38 %) parse  /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/tomlkit/parser.py:157
        │       │       │   ├── [[9 frames hidden in 2 file(s)]]
        │       │       │   │   └── 📄 14.000MB (8.21 %) _to_linecol  /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/tomlkit/source.py:179
        │       │       │   └── [[6 frames hidden in 3 file(s)]]
        │       │       │       └── 📄 2.000MB (1.17 %) <genexpr>  /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/tomlkit/items.py:412
        │       │       └── [[3 frames hidden in 2 file(s)]]
        │       │           └── 📄 5.000MB (2.93 %) __init__  /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/tomlkit/toml_char.py:8
        │       └── [[13 frames hidden in 11 file(s)]]
        │           └── 📄 3.000MB (1.76 %) <listcomp>  /home/circleci/.local/share/pypoetry/venv/lib/python3.10/site-packages/tomlkit/source.py:71
        └── [[75 frames hidden in 14 file(s)]]
            └── 📄 1.255MB (0.74 %) _compile_bytecode  <frozen importlib._bootstrap_external>:672

[radoering]

Can you check if it's related to the tomlkit version?

[TNonet]

Hi @radoering,

Thanks for the response! I will take a look at this soon.

[dsalisbury]

Hey hey!

Another data point: this is triggering in bitbucket pipelines with a step that has a 1GB memory limit for the docker service.

I can also recreate it reliably locally using plain docker commands. I made a repo here with a sample pyproject.toml, lockfile, and test harness to run things https://github.com/dsalisbury/poetry-6144

Watch along with docker stats and you'll see the container go up to liek 1023MiB then get killed

In my testing, using tomlkit==0.11.1 (the previous version) keeps the memory utilisation at a typical level

[dsalisbury]

I noticed that there's a new bugfix release of tomlkit (0.11.3) about 30 mins ago but it hasn't made any difference here

[radoering]

If it's a regression in tomlkit, you should probably open an issue at https://github.com/sdispater/tomlkit/issues.

[TNonet]

Downgrading tomlkit (0.11.3) to 0.11.1 seems to alleviate the issue! They should release a new version soon!

Can we exclude tomlkit 0.11.{2/3} from the valid tomlkit versions for poetry?

[Secrus]

Downgrading tomlkit (0.11.3) to 0.11.1 seems to alleviate the issue! They should release a new version soon!

Can we exclude tomlkit 0.11.{2/3} from the valid tomlkit versions for poetry?

Theoretically yes, but it would only work on the master branch version since I doubt this would be enough for another release to be made.

[TNonet]

This is now fixed with tomlkit https://github.com/sdispater/tomlkit/releases/tag/0.11.4.

Downgrading tomlkit (0.11.3) to 0.11.1 seems to alleviate the issue! They should release a new version soon! Can we exclude tomlkit 0.11.{2/3} from the valid tomlkit versions for poetry?

Theoretically yes, but it would only work on the master branch version since I doubt this would be enough for another release to be made.

While it may not be enough for another release, it could go out with the next.

[radoering]

I agree that we should exclude tomlkit 0.11.2 and 0.11.3. Just created a PR.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.