miguelgrinberg
Test failures with Python 3.11 pre-release (3.11b3)
Describe the bug Several tests fail when testing Python 3.11 beta 3 for Fedora Linux 37. The development version of Fedora will start to use a pre-release of Python 3.11 soon, and Fedora 37 will be released with Python 3.11 this fall.
To Reproduce Steps to reproduce the behavior:
- Check out current
main. - Edit
tox.inito add a Python 3.11 environment:
diff --git a/tox.ini b/tox.ini
index 146ccad..7a0766d 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,5 +1,5 @@
[tox]
-envlist=flake8,py37,py38,py39,py310,pypy3,docs
+envlist=flake8,py37,py38,py39,py310,py311,pypy3,docs
skip_missing_interpreters=True
[gh-actions]
- Make sure the appropriate interpreter is available. (On Fedora Linux 36:
sudo dnf install python3.11 python3.11-devel) - Run
tox. (Ortox -e py311.)
Unfortunately, this isn’t quite enough to reproduce what we’re seeing, because you’ll be blocked by problems with dependencies that we’ve worked around downstream. Hopefully the fixes will soon be available in the upstream releases of those dependencies.
Expected behavior
All tests pass in all environments. (In my local environment, something breaks on pypy3 that I haven’t looked into, but that’s not relevant here.)
Logs
=================================== FAILURES ===================================
____________________ TestAsyncServer.test_connect_bad_poll _____________________
self = <tests.asyncio.test_asyncio_server.TestAsyncServer testMethod=test_connect_bad_poll>
import_module = <MagicMock name='import_module' id='140656792811152'>
AsyncSocket = <MagicMock name='AsyncSocket' id='140656792810384'>
@mock.patch('engineio.asyncio_socket.AsyncSocket')
@mock.patch('importlib.import_module')
def test_connect_bad_poll(self, import_module, AsyncSocket):
a = self.get_async_mock()
import_module.side_effect = [a]
AsyncSocket.return_value = self._get_mock_socket()
AsyncSocket.return_value.poll.side_effect = [exceptions.QueueEmpty]
s = asyncio_server.AsyncServer()
_run(s.handle_request('request'))
assert a._async['make_response'].call_count == 1
> assert a._async['make_response'].call_args[0][0] == '400 BAD REQUEST'
E AssertionError: assert '200 OK' == '400 BAD REQUEST'
E - 400 BAD REQUEST
E + 200 OK
tests/asyncio/test_asyncio_server.py:291: AssertionError
_______________ TestAsyncServer.test_connect_transport_websocket _______________
self = <tests.asyncio.test_asyncio_server.TestAsyncServer testMethod=test_connect_transport_websocket>
import_module = <MagicMock name='import_module' id='140656788334672'>
AsyncSocket = <MagicMock name='AsyncSocket' id='140656788328912'>
@mock.patch('engineio.asyncio_socket.AsyncSocket')
@mock.patch('importlib.import_module')
def test_connect_transport_websocket(self, import_module, AsyncSocket):
a = self.get_async_mock(
{
'REQUEST_METHOD': 'GET',
'QUERY_STRING': 'transport=websocket',
'HTTP_UPGRADE': 'websocket',
}
)
import_module.side_effect = [a]
AsyncSocket.return_value = self._get_mock_socket()
s = asyncio_server.AsyncServer()
s.generate_id = mock.MagicMock(return_value='123')
# force socket to stay open, so that we can check it later
AsyncSocket().closed = False
_run(s.handle_request('request'))
assert (
s.sockets['123'].send.mock.call_args[0][0].packet_type
== packet.OPEN
> )
E AttributeError: 'function' object has no attribute 'mock'
tests/asyncio/test_asyncio_server.py:313: AttributeError
___________ TestAsyncServer.test_connect_transport_websocket_closed ____________
self = <tests.asyncio.test_asyncio_server.TestAsyncServer testMethod=test_connect_transport_websocket_closed>
import_module = <MagicMock name='import_module' id='140656791298256'>
AsyncSocket = <MagicMock name='AsyncSocket' id='140656791305040'>
@mock.patch('engineio.asyncio_socket.AsyncSocket')
@mock.patch('importlib.import_module')
def test_connect_transport_websocket_closed(
self, import_module, AsyncSocket
):
a = self.get_async_mock(
{
'REQUEST_METHOD': 'GET',
'QUERY_STRING': 'transport=websocket',
'HTTP_UPGRADE': 'websocket'
}
)
import_module.side_effect = [a]
AsyncSocket.return_value = self._get_mock_socket()
s = asyncio_server.AsyncServer()
s.generate_id = mock.MagicMock(return_value='123')
# this mock handler just closes the socket, as it would happen on a
# real websocket exchange
async def mock_handle(environ):
s.sockets['123'].closed = True
AsyncSocket().handle_get_request = mock_handle
_run(s.handle_request('request'))
> assert '123' not in s.sockets # socket should close on its own
E AssertionError: assert '123' not in {'123': <engineio.asyncio_socket.AsyncSocket object at 0x7fed364c6810>}
E + where {'123': <engineio.asyncio_socket.AsyncSocket object at 0x7fed364c6810>} = <engineio.asyncio_server.AsyncServer object at 0x7fed35ff0c90>.sockets
tests/asyncio/test_asyncio_server.py:361: AssertionError
_____________________ TestAsyncServer.test_disconnect_all ______________________
self = <tests.asyncio.test_asyncio_server.TestAsyncServer testMethod=test_disconnect_all>
def test_disconnect_all(self):
s = asyncio_server.AsyncServer()
s.sockets['foo'] = mock_foo = self._get_mock_socket()
s.sockets['bar'] = mock_bar = self._get_mock_socket()
> _run(s.disconnect())
tests/asyncio/test_asyncio_server.py:157:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/asyncio/test_asyncio_server.py:33: in _run
return asyncio.get_event_loop().run_until_complete(coro)
/usr/lib64/python3.11/asyncio/base_events.py:650: in run_until_complete
return future.result()
../../BUILDROOT/python-engineio-4.3.2-1.fc37.x86_64/usr/lib/python3.11/site-packages/engineio/asyncio_server.py:178: in disconnect
await asyncio.wait([client.close()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
fs = {<coroutine object AsyncMock.<locals>.mock_coro at 0x7fed361f0e50>, <coroutine object AsyncMock.<locals>.mock_coro at 0x7fed361f05e0>}
async def wait(fs, *, timeout=None, return_when=ALL_COMPLETED):
"""Wait for the Futures or Tasks given by fs to complete.
The fs iterable must not be empty.
Coroutines will be wrapped in Tasks.
Returns two sets of Future: (done, pending).
Usage:
done, pending = await asyncio.wait(fs)
Note: This does not raise TimeoutError! Futures that aren't done
when the timeout occurs are returned in the second set.
"""
if futures.isfuture(fs) or coroutines.iscoroutine(fs):
raise TypeError(f"expect a list of futures, not {type(fs).__name__}")
if not fs:
raise ValueError('Set of Tasks/Futures is empty.')
if return_when not in (FIRST_COMPLETED, FIRST_EXCEPTION, ALL_COMPLETED):
raise ValueError(f'Invalid return_when value: {return_when}')
fs = set(fs)
if any(coroutines.iscoroutine(f) for f in fs):
> raise TypeError("Passing coroutines is forbidden, use tasks explicitly.")
E TypeError: Passing coroutines is forbidden, use tasks explicitly.
/usr/lib64/python3.11/asyncio/tasks.py:424: TypeError
______________ TestAsyncServer.test_http_upgrade_case_insensitive ______________
self = <tests.asyncio.test_asyncio_server.TestAsyncServer testMethod=test_http_upgrade_case_insensitive>
import_module = <MagicMock name='import_module' id='140656789877840'>
AsyncSocket = <MagicMock name='AsyncSocket' id='140656789870800'>
@mock.patch('engineio.asyncio_socket.AsyncSocket')
@mock.patch('importlib.import_module')
def test_http_upgrade_case_insensitive(self, import_module, AsyncSocket):
a = self.get_async_mock(
{
'REQUEST_METHOD': 'GET',
'QUERY_STRING': 'transport=websocket',
'HTTP_UPGRADE': 'WebSocket',
}
)
import_module.side_effect = [a]
AsyncSocket.return_value = self._get_mock_socket()
s = asyncio_server.AsyncServer()
s.generate_id = mock.MagicMock(return_value='123')
# force socket to stay open, so that we can check it later
AsyncSocket().closed = False
_run(s.handle_request('request'))
assert (
s.sockets['123'].send.mock.call_args[0][0].packet_type
== packet.OPEN
> )
E AttributeError: 'function' object has no attribute 'mock'
tests/asyncio/test_asyncio_server.py:335: AttributeError
=============================== warnings summary ===============================
../../../../usr/lib/python3.11/site-packages/eventlet/support/greenlets.py:6
/usr/lib/python3.11/site-packages/eventlet/support/greenlets.py:6: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
preserves_excinfo = (distutils.version.LooseVersion(greenlet.__version__)
../../../../usr/lib/python3.11/site-packages/eventlet/support/greenlets.py:7
/usr/lib/python3.11/site-packages/eventlet/support/greenlets.py:7: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
>= distutils.version.LooseVersion('0.3.2'))
tests/asyncio/test_async_asgi.py: 46 warnings
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_async_asgi.py:23: DeprecationWarning: There is no current event loop
return asyncio.get_event_loop().run_until_complete(coro)
tests/asyncio/test_async_tornado.py::TornadoTests::test_translate_request
tests/asyncio/test_async_tornado.py::TornadoTests::test_translate_request
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_async_tornado.py:16: DeprecationWarning: There is no current event loop
return asyncio.get_event_loop().run_until_complete(coro)
tests/asyncio/test_asyncio_client.py: 83 warnings
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:33: DeprecationWarning: There is no current event loop
return asyncio.get_event_loop().run_until_complete(coro)
tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_background_tasks
/builddir/build/BUILDROOT/python-engineio-4.3.2-1.fc37.x86_64/usr/lib/python3.11/site-packages/engineio/asyncio_client.py:177: DeprecationWarning: There is no current event loop
return asyncio.ensure_future(target(*args, **kwargs))
tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_background_tasks
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:238: DeprecationWarning: There is no current event loop
pending = asyncio.all_tasks(loop=asyncio.get_event_loop()) \
tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_background_tasks
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:240: DeprecationWarning: There is no current event loop
asyncio.get_event_loop().run_until_complete(asyncio.wait(pending))
tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_disconnect_polling_abort
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:212: RuntimeWarning: coroutine 'AsyncMock.<locals>.mock_coro' was never awaited
assert c not in client.connected_clients
Enable tracemalloc to get traceback where the object was allocated.
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_disconnect_websocket_abort
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:228: RuntimeWarning: coroutine 'AsyncMock.<locals>.mock_coro' was never awaited
assert c not in client.connected_clients
Enable tracemalloc to get traceback where the object was allocated.
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_signal_handler
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:1456: DeprecationWarning: There is no current event loop
asyncio.get_event_loop().run_until_complete(test())
tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_trigger_event_coroutine_async
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:943: DeprecationWarning: There is no current event loop
asyncio.get_event_loop().run_until_complete(fut)
tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_trigger_event_coroutine_async_error
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:971: DeprecationWarning: There is no current event loop
asyncio.get_event_loop().run_until_complete(fut)
tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_trigger_event_function_async
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:930: DeprecationWarning: There is no current event loop
asyncio.get_event_loop().run_until_complete(fut)
tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_trigger_event_function_async_error
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_client.py:957: DeprecationWarning: There is no current event loop
asyncio.get_event_loop().run_until_complete(fut)
tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_websocket_connection_no_open_packet
tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_websocket_connection_with_cookies
tests/asyncio/test_asyncio_client.py::TestAsyncClient::test_websocket_upgrade_no_pong
tests/common/test_client.py::TestClient::test_signal_handler
tests/common/test_client.py::TestClient::test_signal_handler
tests/common/test_client.py::TestClient::test_signal_handler
/builddir/build/BUILDROOT/python-engineio-4.3.2-1.fc37.x86_64/usr/lib/python3.11/site-packages/engineio/asyncio_client.py:200: DeprecationWarning: There is no current event loop
loop = asyncio.get_event_loop()
tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_background_tasks
/builddir/build/BUILDROOT/python-engineio-4.3.2-1.fc37.x86_64/usr/lib/python3.11/site-packages/engineio/asyncio_server.py:332: DeprecationWarning: There is no current event loop
return asyncio.ensure_future(target(*args, **kwargs))
tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_background_tasks
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_server.py:1012: DeprecationWarning: There is no current event loop
pending = asyncio.all_tasks(loop=asyncio.get_event_loop()) \
tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_background_tasks
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_server.py:1014: DeprecationWarning: There is no current event loop
asyncio.get_event_loop().run_until_complete(asyncio.wait(pending))
tests/asyncio/test_asyncio_server.py: 68 warnings
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_server.py:33: DeprecationWarning: There is no current event loop
return asyncio.get_event_loop().run_until_complete(coro)
tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_trigger_event_coroutine_async
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_server.py:1094: DeprecationWarning: There is no current event loop
asyncio.get_event_loop().run_until_complete(fut)
tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_trigger_event_coroutine_async_error
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_server.py:1122: DeprecationWarning: There is no current event loop
asyncio.get_event_loop().run_until_complete(fut)
tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_trigger_event_function_async
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_server.py:1081: DeprecationWarning: There is no current event loop
asyncio.get_event_loop().run_until_complete(fut)
tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_trigger_event_function_async_error
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_server.py:1108: DeprecationWarning: There is no current event loop
asyncio.get_event_loop().run_until_complete(fut)
tests/asyncio/test_asyncio_socket.py: 50 warnings
/builddir/build/BUILD/python-engineio-4.3.2/tests/asyncio/test_asyncio_socket.py:28: DeprecationWarning: There is no current event loop
return asyncio.get_event_loop().run_until_complete(coro)
tests/common/test_client.py::TestClient::test_read_loop_websocket_disconnected
/usr/lib64/python3.11/inspect.py:3081: RuntimeWarning: coroutine 'AsyncMock.<locals>.mock_coro' was never awaited
arg_val = next(arg_vals)
Enable tracemalloc to get traceback where the object was allocated.
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_connect_bad_poll
FAILED tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_connect_transport_websocket
FAILED tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_connect_transport_websocket_closed
FAILED tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_disconnect_all
FAILED tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_http_upgrade_case_insensitive
================= 5 failed, 475 passed, 275 warnings in 5.66s ==================
Additional context
I’m not sure it will be trivial for me to prepare a PR to fix these, but I will be happy to do any experiments that might be helpful, and to test any candidate fixes.
I will look into these failures if they continue to exist when 3.11 is officially out.
I will look into these failures if they continue to exist when 3.11 is officially out.
I appreciate that. I hope it will be possible to investigate these a little earlier, though. By the time Python 3.11 final is released (2022-10-03), Fedora Linux 37 will be entering final freeze, and (if nothing is fixed before then) the python-engineio package and its dependent packages will be broken in the final release.
Would you consider revisiting this, say, once Python 3.11 is at the Release Candidate stage (scheduled 2022-08-01)?
@musicinmybrain Would you be interested in debugging these problems on your side? I'm open to a PR if you figure out what's wrong in time.
@musicinmybrain Would you be interested in debugging these problems on your side? I'm open to a PR if you figure out what's wrong in time.
Yes, I will definitely spend some time studying it in the interim.
I can now reproduce this in a virtualenv, at least.
$ python3.11 -m venv _env
$ . _env/bin/activate
(_env) $ pip install --upgrade pip setuptools wheel
(_env) $ AIOHTTP_NO_EXTENSIONS=1 FROZENLIST_NO_EXTENSIONS=1 YARL_NO_EXTENSIONS=1 pip install -e .[client,asyncio_client] pytest tornado
(_env) $ pytest --ignore=tests/common/test_async_eventlet.py -k 'not test_async_mode_eventlet'
I get two WebSocket-related failures not reported above, and the process hangs and does not respond to ^C after printing the test summary, but I am at least able to reproduce the four remaining failures described in this report without relying on anything specific to Fedora Linux.
This is just a status update. I haven’t spent any more time trying to understand the remaining errors.
Now
AIOHTTP_NO_EXTENSIONS=1 FROZENLIST_NO_EXTENSIONS=1 YARL_NO_EXTENSIONS=1 pip install -e .[client,asyncio_client] pytest tornado
can simply be
pip install -e .[client,asyncio_client] pytest tornado
The result of
pytest --ignore=tests/common/test_async_eventlet.py -k 'not test_async_mode_eventlet'
has not changed since my previous comment.
Fedora Rawhide is now using Python 3.11b3, but Python 3.11 in Fedora 37 is at risk due to possible CPython upstream schedule slip.
No change with Python 3.11b5 and python-engineio 4.3.4.
Incidentally, it looks like Python 3.11 in Fedora 37 is likely going to stay on track.
We are just skipping these tests in Fedora Rawhide/37 for the time being in order to unblock dependent packages. We are guessing that these failures likely reflect a problem with mocking rather than a problem with the library’s operation, but we don’t understand what’s happening well enough to fix it.
I can confirm that this is an issue with the mocking that I'm doing, which causes some tests to fail. There are no indications that the actual package is failing under 3.11.
I haven't been able to determine what the problem is yet. Something within asyncio and/or the unittest.mock package must have changed in a way that only in four tests in this package a mock that is inserted on the AsyncSocket class is not applied, causing the tests to run with the real object instead of the mocked one, thus failing.
FAILED tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_connect_bad_poll - AssertionError: assert '200 OK' == '400 BAD REQUEST'
FAILED tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_connect_transport_websocket - AttributeError: 'function' object has no attribute 'mock'
FAILED tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_connect_transport_websocket_closed - AssertionError: assert '123' not in {'123': <engineio.asyncio_socket.AsyncSocket object...
FAILED tests/asyncio/test_asyncio_server.py::TestAsyncServer::test_http_upgrade_case_insensitive - AttributeError: 'function' object has no attribute 'mock'
This patch to unittest.mock caused the failures. The mock patching did not import anything in 3.10 and older. In 3.11 it does import something, so mocking fails when the import_module() function has also been mocked and cannot be used.
For now I'm going to reverse the order of my mocks, so that import_module() is done last. Longer term I should probably figure out a way to mock without messing with such a core function of Python. :)
Further investigation points to https://github.com/python/cpython/commit/ab7fcc8fbdc11091370deeb000a787fb02f9b13d as the change after which mocking broke. Filed https://github.com/python/cpython/issues/98771 with CPython.
Thank you for taking the time to bisect this and submit a useful report upstream.
I can confirm that https://github.com/miguelgrinberg/python-engineio/commit/ac3911356fbe933afa7c11d56141f0e228c01528 fixes the test failures on Python 3.11 when applied as a patch in Fedora Linux.