python
"import hashlib" crashes on aarch64 after Finalize and re-initialize
Crash report
What happened?
Compiling and linking below program results in a crash when run on aarch64 (macOS, Debian VM or Ubuntu 22 VM in Azure for example):
#include <Python.h>
int main(int argc, char *argv[])
{
Py_Initialize();
PyRun_SimpleString("import hashlib\nprint('test1')");
Py_Finalize();
Py_Initialize();
PyRun_SimpleString("import trace\ntrace.Trace().run('import hashlib')\nprint('test2')");
Py_Finalize();
return 0;
}
The trace in the second import is just for better context. It seems not reproducible on Intel. On a debug python build it asserts with: testcase1: ../Python/getargs.c:2052: parser_init: Assertion `parser->kwtuple != NULL' failed.
The backtrace looks like this:
#7 0x0000fffff7906fec [PAC] in _PyArg_UnpackKeywords (
args=args@entry=0xfffff7fec5d8, nargs=nargs@entry=0,
kwargs=kwargs@entry=0x0, kwnames=kwnames@entry=0xfffff6620be0,
parser=parser@entry=0xfffff6f70790 <_parser.16>, minpos=minpos@entry=0,
maxpos=maxpos@entry=1, minkw=minkw@entry=0, buf=buf@entry=0xffffffffe180)
at ../Python/getargs.c:2371
#8 0x0000fffff6f57500 [PAC] in _hashlib_openssl_md5 (module=0xfffff6553d70,
args=args@entry=0xfffff7fec5d8, nargs=nargs@entry=0,
kwnames=kwnames@entry=0xfffff6620be0)
at ../Modules/clinic/_hashopenssl.c.h:343
#9 0x0000fffff780c010 [PAC] in cfunction_vectorcall_FASTCALL_KEYWORDS (
func=0xfffff656c050, args=0xfffff7fec5d8, nargsf=
Looking at the hashlib_openssl_md5 function, the use of static non-const structures seems very suspicious at least.
CPython versions tested on:
3.12
Operating systems tested on:
Linux, macOS
Output from running 'python -VV' on the command line:
Python 3.12.4 (main, Jun 8 2024, 04:51:36) [Clang 15.0.0 (clang-1500.1.0.2.5)]
I have just run into this problem on x64 Windows PC, so I can confirm, that the issue is reproducible on Intel.
In my case I'm running this minimum example script:
import _hashlib
def test():
return _hashlib.openssl_md5(b"hello world!")
in a similar sequence from embedded CPython:
- Py_Initialize
- Load module with above example and call test function
- Py_Finalize
- Py_Initialize
- Load module with above example and call test function
- Py_Finalize
On the second attempt the same assertion fails on debug build, and NULL pointer crash happens on non-debug build.
The problem seems to be the initialization and cleanup of statically initialized _PyArg_Parser structures. The above example will call (through _PyArg_UnpackKeywords) the parser_init function. This function will check _PyArg_Parser.initialized and when that flag is not set it will eventually set _PyArg_Parser.initialized and _PyArg_Parser.kwtuple.
Later, as a result of Py_Finalize call the function parser_clear will be called on the statically initialized parser. This function will only clean the _PyArg_Parser.kwtuple but will leave the _PyArg_Parser.initialized set.
The next call to parser_init will see that the _PyArg_Parser.initialized is still set even thought the _PyArg_Parser.kwtuple is not initialized this time and left as NULL. This leads to the NULL pointer exception.
CPython version: 3.12.4 Operating system: Windows 10 64bit
I did not test on Windows. Only macOS Arm, Linux Arm, Linux x86_64 If you need to call the test function containing use of openssl function (vs. just importing/loading) then it might not be quite the same issue...
I'm pretty certain that this is the exact same issue just by the location of the assertion that failed:
../Python/getargs.c:2052: parser_init: Assertion 'parser->kwtuple != NULL' failed.
combined with the double initialization. There are more ways to trigger that same symptoms because the static parser initialization is used in many places in the same way.
EDIT:
Just to be sure, I tested on Windows with the script:
import hashlib
print('test1')
and have exactly the same result. The function _hashlib_openssl_md5 is being internally called as a result of just importing hashlib (it can even be seen in the stacktrace from the first post).
I have a fix that basically set to 0 the parser->once.v in the parser_clear. I ran all the tests but I'm not sure if I'm missing something else.
Unfortunately I'll be away for some time and I'm not able to fix it now.
This is actually been fixed in the meantime by @neonene with https://github.com/python/cpython/pull/122481
I tested the code in this issue and now it is working OK. I guess we can close this issue but I leave it to @rdoeffinger to verify. The fix went into 3.12.5.
Closing since the OP did not reply and the original issue is not reproducible anymore.
