Make the Python test suite pass with the GIL disabled

[colesbury]

Feature or enhancement

This issue is to track the remaining bugs that prevent the Python test suite from passing with the GIL disabled, as well as the PRs under review that fix known issues.

Note that this is not intended to cover all free-threading related issues: some thread-safety issues don't result in tests crashing or failing and are not tracked here. This is intended to cover the minimum set of changes needed to ensure that the test suite passes reliably with the GIL disabled.

PRs

• https://github.com/python/cpython/pull/114742
• https://github.com/python/cpython/pull/116775

Integration branch

• https://github.com/colesbury/cpython/tree/nogil-integration

Branches to merge

git merge disable-gil-default &&
git merge nogil_dict_pydictvalues &&
git merge nogil_settrace

This branch contains the merge of the above PRs and is used to find bugs that are not covered by outstanding PRs. Note that setting PYTHON_GIL=0 isn't sufficient because some tests do not propagate the environment variable to tests in subprocesses. Instead, the branch changes the default mode to disable the GIL for testing purposes.

Issues without PRs that fix them

• https://github.com/python/cpython/issues/111926

Bugs without issues

• test_importlib.test_concurrent_futures_circular_import sometimes segfaults (alternatively run Lib/test/test_importlib/partial/cfimport.py directly)
• test.test_import.ImportTests.test_concurrency segfaults (probably sys.settrace)
• test_regrtest hangs (maybe it's just slow?)
• test.test_smtpnet.SmtpSSLTest.test_connect_default_port connection closed (possibly related)
• test.test_importlib.test_threaded_import.ThreadedImportTests.test_parallel_module_init segfaults (likely weakref)
• test_threading.test_4_daemon_threads segfaults (crash using _List, which involves weakrefs)

[tonybaloney]

Partly related: https://github.com/python/cpython/issues/112677 which tests inside a sub interpreter (with each test having an isolated GIL).

[colesbury]

The relevant PRs are merged, so I'm closing this now.

We'll open new issues for bugs/crashes as they pop up.