pyo3 icon indicating copy to clipboard operation
pyo3 copied to clipboard

Fail before calling `Py_Initialize` on `PyPy`.

Open davidhewitt opened this issue 1 year ago • 2 comments

At the moment PyPy does export a Py_Initialize symbol, which in theory we allow linking against and calling. But this is just a stub and calling it segfaults. There's an upstream issue at https://github.com/pypy/pypy/issues/3836 to implement support (and I once offered, but since ran out of available time to invest in such a project at present).

Given that users can at the moment easily build PyO3 binaries with PyPy which immediately segfault, we should either:

  • Panic at runtime before calling Py_Initialize on PyPy, or
  • Fail to compile on PyPy e.g. by removing APIs such as prepare_freethreaded_python, and if the auto-initialize feature is enabled, maybe fail as part of the build.rs build script.

davidhewitt avatar Apr 03 '24 08:04 davidhewitt

Labelling as good first issue because I think this is a relatively localised change which will help avoid a nasty crash for unsuspecting users.

davidhewitt avatar Apr 03 '24 08:04 davidhewitt

I'd be happy to take a look at these, and have recently acquainted myself a few of the relevant areas of the code. Where I am unsure is how to test this. From a testcase perspective I would imagine needing a mix of trybuild tests to validate the compile failures and integration tests to validate runtime panics with versions tagged #[cfg(...)] for the PyPy and non-PyPy cases.

What kind of an overall environment would be needed for this? I assume that Pyo3 uses the Python version based on the link libraries available/specified not the interactive python binary selected. I also assume there is no difference between libraries and binaries using embedded python. Short of running the whole thing in a dedicated matrix pipeline, how would one go about installing a local setup that allows nox to run compilation and test against PyPy and cPython on demand? (I'm using a devcontainer, so it's easy enough to create a dedicated set up by just adjusting the underlying Dockerfile once I know what is needed)

MusicalNinjaDad avatar Apr 24 '24 17:04 MusicalNinjaDad

I am sorry for the slow reply here. I also got asked about this by someone at the PyCon sprints today and realised as we looked at the OP together that whatever I stated about adding cfg flags seems to already be true. As a consequence, I'm not actually certain what the TODO is here and think this issue should just be closed again. Sorry for any wasted time.

davidhewitt avatar May 20 '24 20:05 davidhewitt

Hey, don't worry, we're all doing this in our spare time, and thanks for the reply.

MusicalNinjaDad avatar May 26 '24 16:05 MusicalNinjaDad