python
[3.13] _PyObject_Init() assertion fails with the stable ABI on Python built with assertions (extension built with Python 3.11)
Crash report
What happened?
Consider the following extension.
C code:
#include <Python.h>
static PyObject *
foo_bar(PyObject *self, PyObject *args)
{
Py_INCREF(PyExc_TypeError);
PyErr_SetString(PyExc_TypeError, "foo");
return NULL;
}
static PyMethodDef foomethods[] = {
{"bar", foo_bar, METH_VARARGS, ""},
{NULL, NULL, 0, NULL},
};
static PyModuleDef foomodule = {
PyModuleDef_HEAD_INIT,
.m_name = "foo",
.m_doc = "foo test module",
.m_size = -1,
.m_methods = foomethods,
};
PyMODINIT_FUNC
PyInit_foo(void)
{
return PyModule_Create(&foomodule);
}
setup.py:
from setuptools import setup, Extension
setup(name='foo',
version='0',
ext_modules=[
Extension('foo', ['foo.c'], py_limited_api='cp38'),
])
If I compile the extension using Python older than 3.12, and then run the method, Python 3.13.0b3 (built --with-assertions) crashes:
$ python3.11 setup.py build_ext -i
running build_ext
building 'foo' extension
creating build
creating build/temp.linux-x86_64-cpython-311
x86_64-pc-linux-gnu-gcc -Wsign-compare -fPIC -I/usr/include/python3.11 -c foo.c -o build/temp.linux-x86_64-cpython-311/foo.o
creating build/lib.linux-x86_64-cpython-311
x86_64-pc-linux-gnu-gcc -shared build/temp.linux-x86_64-cpython-311/foo.o -L/usr/lib64 -o build/lib.linux-x86_64-cpython-311/foo.abi3.so
copying build/lib.linux-x86_64-cpython-311/foo.abi3.so ->
$ python3.13 -c 'import foo; foo.bar()'
python3.13: ./Include/internal/pycore_object.h:284: _PyObject_Init: Assertion `_PyType_HasFeature(typeobj, Py_TPFLAGS_HEAPTYPE) || _Py_IsImmortal(typeobj)' failed.
Aborted (core dumped)
I've been able to bisect it to c32dc47aca6e8fac152699bc613e015c44ccdba9 (CC @markshannon). Originally hit it in extensions using PyO3, and reported to https://github.com/PyO3/pyo3/issues/4311.
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
Python 3.13.0b3 (main, Jul 4 2024, 14:30:57) [GCC 14.1.1 20240622]
Linked PRs
- gh-121550
- gh-121725
The issue is confirmed, I have found the reason, but I'm not sure this is a bug, I prefer its undefined behavior.
It's caused by the Py_INCREF different behavior between 3.11 and 3.13(or master)
TL;DR;
For the 3.11 version of the Py_INCREF, we add the reference count directly without check FYI https://github.com/python/cpython/blob/3.11/Include/object.h#L491-L504
For the 3.13 version of the Py_INCREF, we add the reference count and check if it exceeds. If it exceeds, this means that this is an immortal object, we do noting except return the function. FYI https://github.com/python/cpython/blob/3.13/Include/object.h#L796-L846
And the function is an inline function, which is means we will expand the function into the binary during the compile. So in 3.11 version, the PyExc_TypeError's reference count is change from 4294967295(the immortal flag) to 4294967296. And the process will fail here https://github.com/python/cpython/pull/116115/files#diff-2a12f738a77b362d74a65949b58c37f2affcd15ba8b1c979b63bd00223b8a456R268
We can compare the assemble code to prove it
For 3.11
0000000000001120 <foo_bar>:
1120: 48 83 ec 08 sub $0x8,%rsp
1124: 48 8b 05 9d 2e 00 00 mov 0x2e9d(%rip),%rax # 3fc8 <PyExc_TypeError@Base>
112b: 48 8d 35 ce 0e 00 00 lea 0xece(%rip),%rsi # 2000 <_fini+0xe9c>
1132: 48 8b 38 mov (%rax),%rdi
1135: 48 83 07 01 addq $0x1,(%rdi)
1139: e8 f2 fe ff ff call 1030 <PyErr_SetString@plt>
113e: 31 c0 xor %eax,%eax
1140: 48 83 c4 08 add $0x8,%rsp
1144: c3 ret
1145: 66 66 2e 0f 1f 84 00 data16 cs nopw 0x0(%rax,%rax,1)
114c: 00 00 00 00
For 3.13
0000000000001120 <foo_bar>:
1120: 48 83 ec 08 sub $0x8,%rsp
1124: 48 8b 05 9d 2e 00 00 mov 0x2e9d(%rip),%rax # 3fc8 <PyExc_TypeError@Base>
112b: 48 8b 38 mov (%rax),%rdi
112e: 8b 07 mov (%rdi),%eax
1130: 83 c0 01 add $0x1,%eax
1133: 74 02 je 1137 <foo_bar+0x17>
1135: 89 07 mov %eax,(%rdi)
1137: 48 8d 35 c2 0e 00 00 lea 0xec2(%rip),%rsi # 2000 <_fini+0xe9c>
113e: e8 ed fe ff ff call 1030 <PyErr_SetString@plt>
1143: 31 c0 xor %eax,%eax
1145: 48 83 c4 08 add $0x8,%rsp
1149: c3 ret
114a: 66 0f 1f 44 00 00 nopw 0x0(%rax,%rax,1)
The patch to fix this issue is simple ,we just the code back
if (_PyType_HasFeature(typeobj, Py_TPFLAGS_HEAPTYPE) || _Py_IsImmortal(typeobj)) {
Py_INCREF(typeobj);
}
I'm not sure this patch has side effect or not. I may need @markshannon help
I setup a new disscussion here https://discuss.python.org/t/disable-the-inline-by-default-for-the-function-api-which-is-included-in-the-stable-abi/57832
After dive more deeper, I think maybe its the user code issue.
I think you use Py_INCREF instead of Py_IncRef, Py_INCREF is not part of the stable ABI.
If you use the Py_IncRef I think the code here would be not existed
The general impression I get from https://github.com/wjakob/nanobind/discussions/500 is that Py_INCREF is part of the limited API but is not explicitly listed due to tooling limitations. As a macro, it does not feature at the ABI level but its implementation does. Prior to 3.12, the implementation modified object reference counts directly. On 3.12 and later, the limited API implementation calls the private function _Py_IncRef.
The general impression I get from wjakob/nanobind#500 is that
Py_INCREFis part of the limited API but is not explicitly listed due to tooling limitations. As a macro, it does not feature at the ABI level but its implementation does. Prior to 3.12, the implementation modified object reference counts directly.
Yes, this is implemented as a Limit API/Stable ABI but actually it's not(can't guarantee the behavior and inlined). I'm not sure we should open a new discussion about this.
Without Py_LIMITED_API defined, some C API functions are inlined or replaced by macros. Defining Py_LIMITED_API disables this inlining, allowing stability as Python’s data structures are improved, but possibly reducing performance.
https://docs.python.org/3/c-api/stable.html#limited-api-scope-and-performance
I think we might need a core to decide this issue should be fixed or not
@vstinner @Fidget-Spinner @markshannon @brettcannon
It seems like this will break a large number of existing extensions if it's not resolved.
If the decision is not to fix this, then CPython needs to not load abi3 .sos that are impacted.
In the limited C API version 3.12 and newer, Py_INCREF() is now implemented as a function call. You can build your C extension with Python 3.12 and the binary should work on Python 3.11 unless you use features specific to Python 3.12. Moreover, the binary (.abi3.so shared library) should work on Python 3.12 and newer.
Example.
setup.py
from setuptools import setup, Extension
setup(name='foo',
version='0',
ext_modules=[
Extension('foo', ['foo.c'], py_limited_api='cp312'),
])
Build:
$ python3.12 setup.py build_ext -
(...)
$ ls *.so
foo.abi3.so*
It works on Python 3.11, 3.12, 3.13:
$ python3.11 -c 'import foo; foo.bar()'
TypeError: foo
$ python3.12 -c 'import foo; foo.bar()'
TypeError: foo
$ python3.13 -c 'import foo; foo.bar()'
TypeError: foo
It works even on Python 3.6:
$ python3.6 -c 'import foo; foo.bar()'
TypeError: foo
My concern is primarily for the existing abi3 wheels on PyPI. The whole point is that maintainers could publish them and they'd continue to work.
In the limited C API version 3.12 and newer, Py_INCREF() is now implemented as a function call.
- Issue: https://github.com/python/cpython/issues/105387
- Discussion: https://discuss.python.org/t/limited-c-api-implement-py-incref-and-py-decref-as-function-calls/27592
- Issue: https://github.com/capi-workgroup/problems/issues/45
In the limited C API version 3.12 and newer, Py_INCREF() is now implemented as a function call. You can build your C extension with Python 3.12 and the binary should work on Python 3.11 unless you use features specific to Python 3.12. Moreover, the binary (.abi3.so shared library) should work on Python 3.12 and newer.
Yes, the problem is the code which is using Py_INCREF and is compiled based on the Python < 3.12 for stable ABI can not be loaded in Python 3.13.
I'm not sure we should fix this or not.
PEP 683 promises to not break the backward compatibility for the stable ABI, but the implementation doesn't seem to be fully compatible.
The problem is that _Py_IsImmortal() checks exactly if ref count is equal to _Py_IMMORTAL_REFCNT (refcnt == _Py_IMMORTAL_REFCNT).
This problem remains me the commit 5f6ac2d88a49b8a7c764691365cd41ee6226a8d0 of the PR gh-112174.
The problem is that _Py_IsImmortal() checks exactly if ref count is equal to _Py_IMMORTAL_REFCNT (refcnt == _Py_IMMORTAL_REFCNT).
Yes, I have mentioned this at here https://github.com/python/cpython/issues/121528#issuecomment-2218444282
But I think here's another core problem we change the way that we treat the object which is not in the heap and is not immortal object(From just check it to refuse the object by crashing process. I'm not sure this is a correct way.
This problem remains me the commit 5f6ac2d of the PR gh-112174.
Sorry, I'm a little confused about the commit you mentioned because I'm not familiar with free threading. Would you mind give more detail here?
Another interesting change: commit 0bb0d88e2d4e300946e399e088e2ff60de2ccf8c of issue gh-109496 (not directly related).
Other interesting changes:
- commit bd1e9509a4475266b21ff432c7875efc289bc0ca of issue gh-118997 <= the issue is was looking for!
- commit 6199fe3b3236748033a7ce2559aeddb5a91bbbd9 of issue gh-105587
@mgorny:
$ python3.13 -c 'import foo; foo.bar()' python3.13: ./Include/internal/pycore_object.h:284: _PyObject_Init: Assertion `_PyType_HasFeature(typeobj, Py_TPFLAGS_HEAPTYPE) || _Py_IsImmortal(typeobj)' failed. Aborted (core dumped)
How did you install this python3.13? It seems to be a debug build of Python. Did you build it manually?
How did you install this
python3.13? It seems to be a debug build of Python. Did you build it manually?
it's not related debug build I guess. I can reproduce it in non-debug build Python 3.13(install by pyenv)
it's not related debug build I guess. I can reproduce it in non-debug build Python 3.13(install by pyenv)
To check if you're using a debug build, you can check:
$ python3 -c 'import sys; print("debug build?", hasattr(sys, "gettotalrefcount"))'
debug build? False
I cannot reproduce the issue with a release build.
The following code is a reference leak, you don't have to INCREF the TypeError exception:
Py_INCREF(PyExc_TypeError);
PyErr_SetString(PyExc_TypeError, "foo");
return NULL;
You should just:
PyErr_SetString(PyExc_TypeError, "foo");
return NULL;
it's not related debug build I guess. I can reproduce it in non-debug build Python 3.13(install by pyenv)
To check if you're using a debug build, you can check:
$ python3 -c 'import sys; print("debug build?", hasattr(sys, "gettotalrefcount"))' debug build? FalseI cannot reproduce the issue with a release build.
Weird, I can't reproduce this in non-debug build now. Sorry for the previous reply, maybe I mix up something
We're building --with-assertions on Gentoo, as noted in the first comment of this thread.
The problem right now is that all current versions of PyO3 produce "broken" extensions, and it will take some time before packages update to the very newest version that fixes this, and every single package published on PyPI before that happens will remain broken.
I wrote PR gh-121725 to fix the assertion. I'm not sure if it's correct. It's an heuristic.
Fixed by change https://github.com/python/cpython/commit/b826e459ca6b640f896c2a9551bb2c78d10f0e2b in the main branch. A backport will follow in the 3.13 branch.
Thanks for the bug report @mgorny. Thanks everyone who helped to analyze the root cause.