Error installing the tokenizers library with Python 3.13

[KEYTRON]

Expected behavior

The library should install successfully.

Current behavior

An error is raised: error: the configured Python interpreter version (3.13) is newer than PyO3's maximum supported version (3.12)

Additional Information

Issue with tokenizers installation

Versions:

• Tokenizers version: Not installed (failed to install)
• PyO3 version: Not installed (failed to install)
• Operating system: macOS 15.0.1
• Python version: 3.13.0
• pip version: 24.2

Description: I encountered an issue while trying to install the tokenizers package. The build process fails due to a compatibility issue with the version of PyO3 and the installed version of Python (3.13.0).

The error message indicates that the configured Python interpreter version (3.13) is newer than PyO3's maximum supported version (3.12).

Steps to reproduce:

1. Create a virtual environment.
2. Run the command pip install tokenizers.
3. Observe the error messages regarding the build process.

Error log: tokenizers_installation_issue.log.txt

Please provide guidance on how to resolve this issue.

[graipher]

The log on my machine also contains this line:

please check if an updated version of PyO3 is available. Current version: 0.21.2

The current version of PyO3 is 0.22.5, Python 3.13 support was initially added in 0.22.0.

I guess, the version needs to be bumped here?

[ArthurZucker]

#1646 is meant to adresse this, and there is a tracker #1639 !

[yz-al]

Blocked on this.