PyDev.Debugger
PyDev.Debugger copied to clipboard
fabioz
Extensive multiprocessing causes pydevd to crash Python 3.12 (but not 3.11)
Issue
My current setup is:
- PyCharm 2023.3 (Professional Edition, #PY-233.11799.259, built on December 3, 2023)
- pydev debugger (build 233.11799.259)
- CPython 3.12.0, installed via
brew install [email protected] - macOS 14.1.2 on a Macbook Pro (2020, M1, 16GB RAM)
Using the following reproducing script:
import time
from tqdm.contrib.concurrent import process_map
def make_request(url: str) -> str:
time.sleep(60)
return f"Requesting {url}"
def main():
requests_to_make = ["https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost"]
responses = process_map(make_request, requests_to_make, chunksize=1)
print(responses)
if __name__ == "__main__":
main()
I create a new project in PyCharm, create a fresh venv with Python 3.12.0 and install tqdm.
When I now run the Script in Debug mode in PyCharm, then (most of the time) the underlying Python process will crash with EXC_BAD_ACCESS and a similar looking crash log (usually the address at KERN_INVALID_ACCESS varies).
If I reduce the number of items in the list requests_to_make (<30 entries), pydevd usually won't crash.
Crashes seem to start occurring between 30-40 entries, going higher makes the crash more reliable.
When I switch to a venv with Python 3.11.6, this issue does not occur and pydevd works correctly even with a large amount of items in requests_to_make.
Crash log (truncated, full log attached)
-------------------------------------
Translated Report (Full Report Below)
-------------------------------------
Process: Python [75372]
Path: /opt/homebrew/*/Python.framework/Versions/3.12/Resources/Python.app/Contents/MacOS/Python
Identifier: org.python.python
Version: 3.12.0 (3.12.0)
Code Type: ARM-64 (Native)
Parent Process: pycharm [71816]
Responsible: pycharm [71816]
User ID: 503
Date/Time: 2023-12-12 13:31:27.5514 +0100
OS Version: macOS 14.1.2 (23B92)
Report Version: 12
Anonymous UUID: 43948B48-2A90-9D43-EE02-BD74C35F2DA9
Sleep/Wake UUID: 391E73ED-5781-4072-B943-D273E7FB8613
Time Awake Since Boot: 47000 seconds
Time Since Wake: 2493 seconds
System Integrity Protection: enabled
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000808080808110 -> 0x0000008080808110 (possible pointer authentication failure)
Exception Codes: 0x0000000000000001, 0x0000808080808110
Termination Reason: Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process: exc handler [75372]
VM Region Info: 0x8080808110 is not in any region. Bytes after previous region: 70875382033 Bytes before following region: 105001204547312
REGION TYPE START - END [ VSIZE] PRT/MAX SHRMOD REGION DETAIL
commpage (reserved) 1000000000-7000000000 [384.0G] ---/--- SM=NUL ...(unallocated)
---> GAP OF 0x5f9000000000 BYTES
MALLOC_NANO 600000000000-600020000000 [512.0M] rw-/rwx SM=PRV
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 pydevd_pep_669_tracing_cython.cpython-312-darwin.so 0x105af9ec0 __pyx_pf_14_pydevd_bundle_29pydevd_pep_669_tracing_cython_15PyStartCallback_2__call__ + 4356
1 pydevd_pep_669_tracing_cython.cpython-312-darwin.so 0x105af8d3c __pyx_pw_14_pydevd_bundle_29pydevd_pep_669_tracing_cython_15PyStartCallback_3__call__ + 788
2 Python 0x104ea5a98 _PyObject_FastCallDictTstate + 96
3 Python 0x104eb187c _PyObject_Call_Prepend + 136
4 Python 0x104e42014 slot_tp_call + 144
5 Python 0x104d99720 _PyObject_MakeTpCall + 128
6 Python 0x104d4f8b4 call_one_instrument.llvm.7388746509351177983 + 240
7 Python 0x104d4d8a8 call_instrumentation_vector.llvm.7388746509351177983 + 288
8 Python 0x104dc21f8 _PyEval_EvalFrameDefault + 1352
9 Python 0x104ea5a98 _PyObject_FastCallDictTstate + 96
10 Python 0x104e44974 slot_tp_init + 212
11 Python 0x104d6576c type_call + 148
12 Python 0x104d99720 _PyObject_MakeTpCall + 128
13 Python 0x104dcc934 _PyEval_EvalFrameDefault + 44164
14 Python 0x104dae9f8 method_vectorcall.llvm.15811550574524407082 + 184
15 Python 0x104dce3a8 _PyEval_EvalFrameDefault + 50936
16 Python 0x104daea74 method_vectorcall.llvm.15811550574524407082 + 308
17 Python 0x104eac2cc _PyVectorcall_Call.llvm.10235175985831010521 + 152
18 Python 0x104dce3a8 _PyEval_EvalFrameDefault + 50936
19 Python 0x104ce7f08 PyEval_EvalCode + 184
20 Python 0x104d32ef8 builtin_exec + 416
21 Python 0x104df4050 cfunction_vectorcall_FASTCALL_KEYWORDS.llvm.2921711504878963487 + 92
22 Python 0x104dcc7e0 _PyEval_EvalFrameDefault + 43824
23 Python 0x104ce7f08 PyEval_EvalCode + 184
24 Python 0x104c96800 run_eval_code_obj.llvm.11164132066462609367 + 88
25 Python 0x104cb4770 run_mod.llvm.11164132066462609367 + 132
26 Python 0x104cf7b24 pyrun_file + 148
27 Python 0x104d57ae0 _PyRun_SimpleFileObject + 288
28 Python 0x104e1a730 _PyRun_AnyFileObject + 232
29 Python 0x104d6b9a8 pymain_run_file_obj + 220
30 Python 0x104d6b7e4 pymain_run_file + 72
31 Python 0x104e42a30 Py_RunMain + 728
32 Python 0x104e42ef4 Py_BytesMain + 40
33 dyld 0x187a8d0e0 start + 2360
Thread 1:
0 libsystem_kernel.dylib 0x187dcc0ac __psynch_cvwait + 8
1 libsystem_pthread.dylib 0x187e095fc _pthread_cond_wait + 1228
2 Python 0x104d62c00 PyThread_acquire_lock_timed + 284
3 Python 0x104d1e87c acquire_timed + 292
4 Python 0x104cde61c lock_PyThread_acquire_lock + 56
5 Python 0x104d45340 method_vectorcall_VARARGS_KEYWORDS.llvm.14063008687374067480 + 148
6 Python 0x104dcc7e0 _PyEval_EvalFrameDefault + 43824
7 Python 0x104daeab4 method_vectorcall.llvm.15811550574524407082 + 372
8 Python 0x104d99ed8 thread_run + 92
9 Python 0x104cff3dc pythread_wrapper.llvm.15843660109015549643 + 48
10 libsystem_pthread.dylib 0x187e09034 _pthread_start + 136
11 libsystem_pthread.dylib 0x187e03e3c thread_start + 8
Thread 2:
0 libsystem_kernel.dylib 0x187dcce40 __recvfrom + 8
1 _socket.cpython-312-darwin.so 0x1045dceb4 sock_recv_impl + 36
2 _socket.cpython-312-darwin.so 0x1045dc450 sock_call_ex + 192
3 _socket.cpython-312-darwin.so 0x1045dce6c sock_recv_guts + 56
4 _socket.cpython-312-darwin.so 0x1045db104 sock_recv + 108
5 Python 0x104deaf28 method_vectorcall_VARARGS.llvm.14063008687374067480 + 128
6 Python 0x104dcc7e0 _PyEval_EvalFrameDefault + 43824
7 Python 0x104daeab4 method_vectorcall.llvm.15811550574524407082 + 372
8 Python 0x104d99ed8 thread_run + 92
9 Python 0x104cff3dc pythread_wrapper.llvm.15843660109015549643 + 48
10 libsystem_pthread.dylib 0x187e09034 _pthread_start + 136
11 libsystem_pthread.dylib 0x187e03e3c thread_start + 8
Thread 3:
0 libsystem_kernel.dylib 0x187dcbea8 __semwait_signal + 8
1 libsystem_c.dylib 0x187caf2f8 nanosleep + 220
2 Python 0x104d1ee90 time_sleep + 124
3 Python 0x104df4534 cfunction_vectorcall_O.llvm.2921711504878963487 + 108
4 Python 0x104dcc7e0 _PyEval_EvalFrameDefault + 43824
5 Python 0x104daeab4 method_vectorcall.llvm.15811550574524407082 + 372
6 Python 0x104d99ed8 thread_run + 92
7 Python 0x104cff3dc pythread_wrapper.llvm.15843660109015549643 + 48
8 libsystem_pthread.dylib 0x187e09034 _pthread_start + 136
9 libsystem_pthread.dylib 0x187e03e3c thread_start + 8
Thread 4:
0 libsystem_kernel.dylib 0x187dcc0ac __psynch_cvwait + 8
1 libsystem_pthread.dylib 0x187e095fc _pthread_cond_wait + 1228
2 Python 0x104d62bdc PyThread_acquire_lock_timed + 248
3 Python 0x104d1e87c acquire_timed + 292
4 Python 0x104cde61c lock_PyThread_acquire_lock + 56
5 Python 0x104df52d0 cfunction_call + 72
6 Python 0x104d99720 _PyObject_MakeTpCall + 128
7 Python 0x104dcc194 _PyEval_EvalFrameDefault + 42212
8 Python 0x104daeab4 method_vectorcall.llvm.15811550574524407082 + 372
9 Python 0x104dce3a8 _PyEval_EvalFrameDefault + 50936
10 Python 0x104ea5a98 _PyObject_FastCallDictTstate + 96
11 Python 0x104eb187c _PyObject_Call_Prepend + 136
12 Python 0x104e42014 slot_tp_call + 144
13 Python 0x104ead92c _PyObject_Call + 124
14 Python 0x104d99ed8 thread_run + 92
15 Python 0x104cff3dc pythread_wrapper.llvm.15843660109015549643 + 48
16 libsystem_pthread.dylib 0x187e09034 _pthread_start + 136
17 libsystem_pthread.dylib 0x187e03e3c thread_start + 8
Thread 5:
0 libsystem_kernel.dylib 0x187dcc0ac __psynch_cvwait + 8
1 libsystem_pthread.dylib 0x187e095fc _pthread_cond_wait + 1228
2 Python 0x104d62bdc PyThread_acquire_lock_timed + 248
3 Python 0x104d1e87c acquire_timed + 292
4 Python 0x104cde61c lock_PyThread_acquire_lock + 56
5 Python 0x104d45340 method_vectorcall_VARARGS_KEYWORDS.llvm.14063008687374067480 + 148
6 Python 0x104dcc7e0 _PyEval_EvalFrameDefault + 43824
7 Python 0x104daeab4 method_vectorcall.llvm.15811550574524407082 + 372
8 Python 0x104dce3a8 _PyEval_EvalFrameDefault + 50936
9 Python 0x104ea5a98 _PyObject_FastCallDictTstate + 96
10 Python 0x104eb187c _PyObject_Call_Prepend + 136
11 Python 0x104e42014 slot_tp_call + 144
12 Python 0x104ead92c _PyObject_Call + 124
13 Python 0x104d99ed8 thread_run + 92
14 Python 0x104cff3dc pythread_wrapper.llvm.15843660109015549643 + 48
15 libsystem_pthread.dylib 0x187e09034 _pthread_start + 136
16 libsystem_pthread.dylib 0x187e03e3c thread_start + 8
