Expand section on packaging compiled extensions :)

[sneakers-the-rat]

Creating an issue to check interest/feasibility/etc. in expanding the section on packaging with compiled extensions!

There are a bunch of moving pieces here, and I don't think we can quite arrive at a single "best practices" because of the amount of variability inherent in the problem, but I think we can get a nice starting point and compile resources for further reading.

Sketching a few of the pieces that would need to be covered:

• CD: building for multiple architectures/versions, releasing from CI/CD
• Config: setting up pyproject.toml and accessory scripts for building
• Tooling: how a sample of package managers handle extension modules
• Frameworks: e.g. Cython, pyo3, etc.
• Integration with other build systems like cmake, meson, etc.
• Typing: type stub files and ux when exposing compiled modules
• Docs: documenting build process, the code itself, and making contribution accessible

[Midnighter]

Definitely an interesting topic. I have a candidate package that can use good guidance, too.

[sneakers-the-rat]

My nefarious plan to slowly convince someone to help me write "rust for python ppl" with pyo3 as a gateway.....

[Midnighter]

I assume you would want to switch to pdm for managing a package with extensions?

[lwasser]

I would love for someone to work on this. @ucodery created some examples iwth hatch of compiled C (i think) in last year's sprint at pycon and it worked. People have asked for this addition if anyone is interested in taking it on! (maybe it could be a team effort?)

[ucodery]

I think some of this is necessary and valuable to the packaging guide. However, I do want to caution how much we attempt to cover. Do we have a guid on the CAPI? General C code? Do we adopt best practices for compiled code layout, and linting, etc for all languages used in conjunction with Python?

I like our mantra of making opinionated decisions about Python packaging. But does that mean we choose build tools that work with all compiled languages, so we just pick one, or do we have a bit of TMO as there is often a different best choice depending on the extension? (e.g. meson will work with Rust code and PyO3 (I think), but for projects that are just py+rs maturin will probably be a better choice).

Hatch can definitely make package extensions. Its harder using Hatchling, and I didn't--I used hatch+meson.

[lwasser]

Good point! You know more about the complexity than I do. Could the guide possibly provide an overview as you begin to lay out above in terms of what tools might be better for which applications? And we could link to resources. Then have examples for more common (if that exists) applications?

[ucodery]

It seems inevitable that PyOS eventually gets at least a C-extension guide. But I wonder if this is another topic that is best to direct readers to Scientific Python? IIRC we link to them for guides to non-hatch projects and a big part of compiling extension modules comes down to switching your build-backend and following that tool's best practices. They already cover multiple compiled build-backends, including all we are discussing here, in addition to guides on several pure-python backends.

[lwasser]

We could do that. We could just link to their guides for compiled stuff . Can you help me find the pure Python backend guides? i know there are tabs, here. which is great. we intentionally kept our guide to a single thing based on user feedback that the options were overwhelming. I'm just trying to better understand where we should focus. and it's most likely that we should just close this issue for the time being and focus on updating other things like our tool overviews (and add more on environment managers and tools like pixi and UV)

[ucodery]

Yeah, that's their guide on pure Python. I was mostly referring to their guide on Python extension modules though. That guide is here and covers three more build-backends. So I count 7 backends already covered, which FWICT are the same ones this thread is discussing. I'm just not sure what we would be able to add for 2, 3, 6 of these backends that can't be found there.

[lwasser]

I think that there is some overlap between this issue and https://github.com/pyOpenSci/python-package-guide/issues/22

Here is my take on what people have told me or requested. The Scientific Python project's guide is, in my opinion, a fantastic resource.

I've heard from people that they want a living example (using one way to do things) of what it looks like to package code that is not pure Python.

One piece we could add here is a well-documented package example that guides someone through getting started in an opinionated manner (not 7 backends, just one). This guide could explain how it works and why we chose the tools we did. It could then cross-link to the other guide for more options (if people want that!)

My naivety also comes into play here, as I haven't done this myself. So, I'm not aware of the actual pain points or whether there's a reason for the numerous ways to do things here (ie are some tools better for some extension approaches? I can't answer that but I suspect you all could.

[sneakers-the-rat]

i could do this with pyo3 and rust, but i think we probably want to also have a C extension example since that's probably the more common case :)