InterpreterPoolExecutor Hangs on Certain Functions

[paultiq]

Bug report

Bug description:

** I first noticed this when testing the backports https://github.com/ericsnowcurrently/interpreters/issues/17 on 3.13 on windows and WSL. Issue reproduced on 3.14.0a1+ 34653bb

The following code hangs:

from concurrent.futures.interpreter import InterpreterPoolExecutor

def my_func():
    return 1+2

with InterpreterPoolExecutor() as executor:
    future = executor.submit(my_func)
    
result = future.result()  # Get the result of the function
print(result)

Yet, if my_func is defined in another module (or imported, or a builtin), it works fine.

from concurrent.futures.interpreter import InterpreterPoolExecutor

with InterpreterPoolExecutor() as executor:
    future = executor.submit(print, "foo")
    
result = future.result()  # Get the result of the function
print(result)

EDIT: Also hangs if my_func is in another module that imports modules that dont support sub-ints.

Importing a module with the following will also hang:

import numpy 

def my_func(x):
    print(f"{x=}")

Note

** test_interpreter_pool imports all the used functions, such as mul, which won't reproduce this error.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Windows

Linked PRs

• gh-125898

[zware]

I believe this is expected: a subinterpreter is effectively another interpreter process within the same actual process. IIUC, InterpreterPoolExecutor is subject to most of the same restrictions as ProcessPoolExecutor.

[paultiq]

The issue here, I'd argue, is the hang... not whether or not it works. Any error would be preferable.

I do see this comment (now), so perhaps some error should be expected? "Functions defined in the main module cannot be pickled and thus cannot be used." - https://github.com/python/cpython/blob/main/Doc/library/concurrent.futures.rst

---

But to the point about ProcessPoolExecutor, the follow code works... but hangs with InterpreterPoolExecutor (w or w/o freeze_support)

ProcessPoolExecutor Works

from concurrent.futures import ProcessPoolExecutor
from multiprocessing import freeze_support
def myprint(x):
    print(f"{x=}")

if __name__ == '__main__':
    freeze_support()
    with ProcessPoolExecutor() as executor:
        future = executor.submit(myprint, "Hi")
        result = future.result()
        print(result)

InterpreterPoolExecutor Hangs

from concurrent.futures.interpreter import InterpreterPoolExecutor
def myprint(x):
    print(f"{x=}")

if __name__ == '__main__':
    with InterpreterPoolExecutor() as executor:
        future = executor.submit(myprint, "Hi")
        result = future.result()
        print(result)

[paultiq]

Updated the original text to add that importing a module that imports numpy will also hang

[ZeroIntensity]

I'll investigate.

[ZeroIntensity]

Ah, I found the problem. The issue is this loop: https://github.com/python/cpython/blob/main/Lib/concurrent/futures/interpreter.py#L188

_exec executes _call_pickled in the subinterpreter, but then the unpickling fails before the exception is in the catching context, so the error isn't sent down to the queue, then that loop runs infinitely waiting for the queue to send something. I'll submit a patch, thank you for the report @paultiq!

[ZeroIntensity]

This should be all fixed now :)